SSL, Tomcat, Android, and keeping my sanity

Recently, at work, I was tasked with getting some SSL certs installed and working on a tomcat installation. This was a bit outside my normal duties, as work is rather segregated, and tomcat falls under an application admin's responsibilities, not a server admin's. However, there isn't an app admin available who knows tomcat, so I was given the job by virtue of competence, and having built the server. For reference, the OS was RHEL 5.7, running Tomcat 6.0.33, and trying to use JSSE for SSL.

Our normal setup for web servers is apache, sometimes using the Cool Web Stack or something like it, so tomcat isn't something that my team has any familiarity with. Additionally, I built this server, but that was just the OS (RHEL). A third party installed the tomcat application server with grails on top of it, for the purpose of hosting some mobile apps (Android and iPhone) created using their toolkit. They set this up without SSL, and in looking through their available documentation, the only reference I could find to SSL was a single footnote on a document about the security of the system which essentially said, since you asked, of course this should all be done over SSL. Just ignore the fact that none of our documentation or reference implementations bother to do so.

So, I set about trying to get the SSL cert working, armed with a set of instructions (team standard procedures) for doing so with apache, and a single page from our wiki on setting up SSL in Tomcat. This page was proof that someone had done so in the past, but it consisted of a couple command lines to run, some java source code to be compiled and then invoked, and no reference to the implementation details of telling tomcat to use the SSL cert itself. The command line invocations converted the standard x509 cert we received from our CA and the key we generated when making the CSR into another format, DER.The java source code formed a program which would read in the der formatted certificate and key, and convert them into a Java keystore (JKS) formatted file. The instructions were for Solaris, our main OS, and not RHEL which these servers were running. The java program wouldn't compile, because the JDK that the third party installed for use with tomcat seemed to have a broken compiler! I installed a new jdk from the RHEL repos, and found that the source code didn't have any include statements, which also caused problems. A bunch of wildcard based includes later, I had a compiled program and a freshly created keystore. I installed it into tomcat, using some helpful online instructions, made a mental note that I wanted to come back at some point and find a way to convert the SSL cert without using a custom compiled program, because that seems like overkill for a problem where standard tools should exist, and continued on my way, after verifying that the site was accessible over SSL now.

I did end up finding a way to convert from the openSSL cert to a java keystore without using a custom compiled java program. After much, much searching, I found a tough to navigate site that was stuffed with useful information! This page shows how to use the openssl tool to combine a key and a cert into one PKCS12 file. Then, this page shows how to use the java (or JDK maybe) command keytool to import the PKCS12 file into a Java KeyStore file. Thus, we are now able to use two commands where before we used 2 different commands & a custom compiled java program. (In theory, we don't have to perform the conversion to a Java KeyStore format, as Tomcat can be told to use the PKCS12 file directly as a keystore. However, this involves more poorly documented tomcat configuration, and I didn't want to keep pressing my luck once I got everything working. If someone else wants to try for efficiency later on, then they are more than welcome to it.)

Those in charge of this project then decided I should turn off all non-SSL traffic to the servers. After doing this, they discovered that they could not download the new APK files to their Android phones from the server over SSL. Android was throwing an untrusted certificate error (the kind you expect with a self-signed cert, not a CA issued one) and will silently fail to download files from a server over SSL in this scenario.

I suspected that the intermediate certs were not being handed out correctly, and was eventually able to prove this with the help of these two sites. Our internal instructions said to import the intermediate cert from our CA into the JKS file with an alias of intermediateca. The official instructions said to import it with an alias of root. Somewhere else online said to use an alias of intermediate. I tried all of these, as well as combining them all, with no luck. I looked through the documentation, and could find no mention of a specific alias name to use for tomcat to magically pick it up and serve it out.

I went searching again, and finally stumbled upon this question on stack overflow. This was the same problem I was having, so I tried the solution, but ran into problems. They placed the intermediate cert into /etc/ssl/certs, then ran the command to create the PKCS12 file with an additional flag -chain. RHEL doesn't have an /etc/ssl/certs, so I searched, and found the equivalent at /etc/pki/tls/certs. I tried placing the intermediate cert there, and running the command with -chain added, and got errors because the cert wasn't found. I then went looking to see what options could be passed to the openssl command, and found the -CAfile and -caname flags. Using these, I was able to use the -chain flag and eventually get a Java KeyStore that caused tomcat to serve out the intermediate cert correctly.


After some experimenting, I finally isolated what creates a working keystore. The -chain flag with the openssl command is the critical key. Combine this with -CAfile to create the PKCS12 file with the intermediate cert included. The -caname flag ends up to not be needed at all. Importing the intermediate certs into the JKS (Java KeyStore) file with an alias, doesn't matter at all. (It doesn't break anything to have them there, but it also isn't needed for it to work.) Counterintuitively, the working JKS file will only show to contain one cert when viewed with keytool -list.

[root@fido sslcerts]# keytool -list -keystore fido.jks 
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

tomcat, Mar 22, 2012, PrivateKeyEntry, 
Certificate fingerprint (MD5): 83:F5:A4:7D:2A:39:35:FB:8B:41:B7:34:B5:97:45:92

I was able to verify with both of the SSL checking sites above that this file will serve out the intermediate certs correctly. On Android, it no longer throws the untrusted cert error, and now silently validates. So, now, we have a new working procedure that only requires the same files we were downloading from the CA before, and uses two commands to turn them into a working Java KeyStore that will serve out intermediate certs correctly

openssl pkcs12 -export -inkey fido-2012-03-15.key -in fido-2012-03-15-cert.cer \
-out fido_key_cert_chain.p12 -chain -name tomcat -CAfile fido-2012-03-15-interm.cer 

keytool -importkeystore -srckeystore fido_key_cert_chain.p12 -srcstoretype pkcs12 \
-srcstorepass changeme -destkeystore fido.jks -deststoretype jks -deststorepass changeme

Citrix Receiver on 64-bit Arch Linux

I recently made the switch from Ubuntu to Arch Linux (with E17) on my workstation at work, and I am now in the process of getting all my apps setup and working again. Pretty high up on that list was the Citrix receiver. Work is a largely Windows based setup, with Exchange for e-mail. I actually prefer a few things about the newer versions of Outlook, when compared with most of the Linux clients I have tried in the past.
This means my choices boil down to:

1. Get Outlook running under Wine (yeah, that'll work out real well) 
2. Run Outlook in a Windows VM (I have this setup, but prefer not to run the VM at all times, it is a real memory hog)
3. Run a separate computer with Windows on it, just for Outlook (not going to happen)
4. Get the Citrix receiver working, and use the Citrix version of Outlook that we make available

I chose #4. The problem here is not the linux part, install packages exist for the receiver on linux. The problem is the 64-bit part. Out Citrix server only hands out a 32-bit installer. I went looking, and while Citrix has started offering 64-bit installers now, they only come in .deb or .rpm. There isn't a .tar.gz package like there is for 32-bit. Arch linux, of course, does not use .deb or .rpm packages.

I had the receiver installed on Ubuntu, and it worked mostly. It worked fine, but I could not run the manager app to change the settings, which meant that I could never map my local hard drive to show up as a drive in the software run on Citrix. For e-mail, this meant I could not save or send attachments, unless I used a USB stick for them, because the default settings would auto-mount USB devices.

I found these instructions on the Arch wiki, and followed the manual install instructions. Probably because I had already downloaded the package from Citrix, and sunk some time into getting the installer to run, so I didn't want to take the easy route out and use a pre-built package now.

Just to get the installer to run, I had to do some research, and finally figure out that the "no such file or directory" errors being thrown by echo_cmd were because I only had the 64-bit glibc libraries, and I needed to install the lib32-glibc from the multilib repo as well.

I followed the instructions on the wiki, making modifications as I went because my install of the receiver was in a different directory, and got a working install, except that I had problems getting firefox to see the Citrix plugin for some reason. I also was not able to get the manager app (wfcmgr) to run, despite the wiki article explicitly saying it should. I was getting this error:

/opt/Citrix/ICAClient/wfcmgr: error while loading shared libraries: libXm.so.4: cannot open shared object file: No such file or directory

I did some more digging, and found the 32-bit library package from the AUR containing libXm.so.4, aur-lib32-openmotif, installed into /opt/lib32/usr/lib directory, instead of into the /usr/lib32 directory where the wfcmgr program was trying to find it.

One way to fix this is with a simple

sudo ln -s /opt/lib32/usr/lib/libXm.so.4 /usr/lib32/libXm.so.4

however, I chose to modify the PKGBUILD to put the libraries in /usr/lib32 with all the others, in case a program went looking for one of the other openmotif libraries in the future.

I also figured out that my problem getting firefox to see the plugin was that I checked whether the plugin was setup with

sudo nspluginwrapper -l

using sudo here, because the wiki article showed the install command, nspluginwrapper -i, being run as root. This showed the plugin to be in place already:


/root/.mozilla/plugins/npwrapper.npica.so
  Original plugin: /opt/Citrix/ICAClient/npica.so
  Plugin viewer: /usr/lib/nspluginwrapper/i386/linux/npviewer
  Wrapper version string: 1.4.4-1

It took me too long to realize that this plugin was not system-wide, but was root-specific, and that I needed to do this as my user instead. I checked and it was not setup for my user, so I used the -i command to install it, and restarted firefox. It is now detected, which means I don't have to skip past the install prompt from the server when the silent-detection routine fails to find the plugin on my system.

In the end, I have a newer version of Citrix Receiver installed and I was able to setup my home directory to be mapped as a drive to be seen inside the Citrixed apps. I do not have the USB device support, because the installer can't figure out Arch's system for managing services, but I don't think I'll miss that.