Four days ago the news about the Heartbleed got every sysadmin's attention. Renowned security expert Bruce Schneier writes:

This means that anything in memory -- SSL private keys, user keys, anything -- is vulnerable. And you have to assume that it is all compromised. All of it.

"Catastrophic" is the right word. On the scale of 1 to 10, this is an 11.

Using a webtool to test for Heartbleed it became clear that my encoding startup Transloadit was also hit. Luckily since we are running stunnel for SSL the bleed is limited, but our certificates could still be stolen. In this post I want to show how I examined and stopped the Heartbleed in Ubuntu/stunnel, but the same approach will work for Apache, Nginx, HAProxy and other programs using OpenSSL to handle secure traffic.

If you pull all your software from a package manager like APT, you may choose to just upgrade all-the-things. But if your setup or requirements are more specific here's how to go through the verification/upgrading process step by step.

The bug was in OpenSSL which stunnel uses. Your stunnel may either reference it as a shared library, or if statically compiled, contain it (in that case you'll have to recompile the whole stack).

To find out, I first want to know where our SSL handler is installed.

$ which stunnel
  /srv/current/stack/bin/stunnel

Looks like we run a version in our stack directory. Let's see what shared libraries it uses by running ldd on it. If we spot libssl, we can just upgrade that shared library and restart stunnel to address the Heartbleed.

$ ldd /srv/current/stack/bin/stunnel
  linux-vdso.so.1 =>  (0x00007fff8f6e7000)
  libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fc73f4ab000)
  libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f73f9c31000)
  libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f73f9a2d000)
  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f73f9810000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f73f9451000)
  libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f73f924c000)
  libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f73f9035000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f73fa272000)

It uses quite a few shared libraries. But notice libssl.so on the second line, pointing to the file /lib/x86_64-linux-gnu/libssl.so.1.0.0.

Let's find out if there is an APT package that is providing this affected file:

$ dpkg -S /lib/x86_64-linux-gnu/libssl.so.1.0.0
  libssl1.0.0: /lib/x86_64-linux-gnu/libssl.so.1.0.0

So libssl1.0.0 is the package name providing the SSL library. Doing a quick apt-get update and apt-get install libssl1.0.0 shows there are indeed updates available, and that libssl-dev is coupled.

I verify that doing the upgrade solves Heartbleed on one server, and decide to roll this out in a forceful way using this script:

$ export DEBIAN_FRONTEND=noninteractive && \
  sudo -E apt-get -y update && \
  sudo -E apt-get -y install libssl-dev && \
  sudo -E service stunnel restart

And that fixed our Heartbleed, although we should still re-issue our certificate to make sure people who might have stolen it while we were exposed cannot impersonate us online.

If you were running an SSL handler that had access to more than just the certificates, e.g. Apache being able to access user sessions, you might want to ask your users to change their passwords.

As of late there's been quite some critique on the code quality of OpenSSL. As efforts are being made to improve, it might make sense to move out SSL handling to an isolated component such as stunnel.

Thanks for reading. I hope this post helps people better diagnose and fix Heartbleed in their setups. Let me know if you have suggestions, I'm always open to learning.