Feb 012016

Those who care about their communications (individuals or enterprise), may at one point decide to look into encrypting email. Email is unfortunately a product of its past, designed for sending communications from one mailbox and delivering to another across the internet, yet during a time where encrypting that communication wasn’t even an afterthought. There have been some bolt on patches to secure email but really a nice new protocol is needed. Being stuck with what we have, you may have decided for one reason or another that S/MIME certificates are what you’d like to use to secure your email. Definitely a lot of people are concerned with the privacy of their email if you look at the section, “How important is it that your online information remain private?” in this article. I recently needed to ensure such certificates were also FIPS compliant. I had a hard time using the normal openssl binaries and ensuring I was using FIPS compliant commands to generate the certificates. So first we will compile an openssl binary in FIPS mode. This binary will error as soon as we run a command that is not FIPS compliant, ensuring our resultant certs are good. Then I’ll show how to generate the certs to be either self-signed by your own CA if you will use and trust among friends/family or to have signed by an Enterprise CA if you are a company with a trusted Enterprise CA and clients that trust it. Regardless, at the end you’ll have S/MIME certs you can use in your mail clients for secure communications.

h4. Building OpenSSL with FIPS Mode

I’m using Ubuntu 14.04.03 LTS, instructions may vary slightly on a different target system.
First download and extract the openssl source tarballs we will need (the below are the latest at time of this writing but always grab the latest stable releases)

wget http://www.openssl.org/source/openssl-1.0.2e.tar.gz
wget http://www.openssl.org/source/openssl-fips-2.0.11.tar.gz
tar xvzf openssl-fips-2.0.11.tar.gz
tar xvzf openssl-1.0.2e.tar.gz

You’ll probably need build-essential package which I had already installed so go ahead and `aptitude install build-essential`.
Next lets build the FIPS module our openssl will need:

cd openssl-fips-2.0.11/
make install

Near the bottom of that output you should see something like installing to /usr/local/ssl/fips-2.0, we will need that directory in a bit to reference.
Now lets compile our own openssl, cd into the openssl-1.0.2e/ dir you extracted above

./config fips shared
make depend
make install

What we did here was tell our compiled openssl that we have a shared fips module to use. The output of the above should tell you “OpenSSL shared libraries have been installed in:

So what you’ve done is you have your normal system openssl completely intact but now in /usr/local/ssl you have the one compiled with FIPS support.
You can check the version
openssl version will come from your system and output something like `OpenSSL 1.0.1f 6 Jan 2014`
whereas if you cd /usr/local/ssl/bin and run ./openssl version you’ll see our fips one `OpenSSL 1.0.2e-fips 3 Dec 2015`.
Great now lets export a couple variables so that our compiled openssl can get to the shared fips module:
export LD_LIBRARY_PATH=/usr/local/ssl/fips-2.0 && export OPENSSL_FIPS=1
One final test to prove this openssl will error on anything that is not FIPS compliant is we can try to get an MD5 hash of a file.
./openssl md5 /home/user/somefile
and you’ll get some error output like:
Error setting digest md5
140006545020576:error:060A80A3:digital envelope routines:FIPS_DIGESTINIT:disabled for fips:fips_md.c:180:

since md5 hash is not FIPS compliant.

h4. Creating the FIPS S/MIME Certs

Now that we have an openssl that will only allow us to run things that are FIPS compliant we can generate some S/MIME certs.
I’m going to number the steps to take here to create your certs with a comment after each numbered step describing what the step is doing. Where you see multiple of the same number, that means you chose the step you want based on your desired outcome (what options you want, what CA will be used, etc.).
1. ./openssl genrsa -out newkey.key 4096 – where newkey.key is the key and can be named anything you want, we are just generating a 4096 bit key.
2. ./openssl pkcs8 -v1 PBE-SHA1-3DES -topk8 -in newkey.key -out enc_newkey.key – this takes our normal key above and encodes it in pkcs8 format. This is a common format used but you have options, like my next command uses v2 which isn’t as widely accepted and there is also pkcs12. If you think you need to use some variant of a command I specify here you can get more information with running ./openssl pkcs12 /? and it’ll output your options.
2. ./openssl pkcs8 -v2 des3 -topk8 -in newkey.key -out enc_newkey.key Here is another variant of the above command if you wanted to use version2 of pkcs8, again chose one of these commands to run.
*With any of the step 2 commands, you will be asked for a password, please enter something from 4 to 1023 characters long and then provide it when asked in step 3 below.
3. ./openssl req -new -key enc_newkey.key -out new_request.csr – Now we take our new pkcs8 encoded key and generate a CSR (Certificate Signing Request) with it.
(see CSR Creation Info below for examples of fields)

Depending on how you want to have your certificate signed, use ONE of the Step 4’s below:
4. ./openssl x509 -req -days 3650 -in new_request.csr -signkey enc_newkey.key -out email.crt – This is the self-signed option
4. ./openssl x509 -req -days 3650 -in new_request.csr -signkey enc_newkey.key -CA enterprise_ca.cer -CAkey enterpriseprivatekeynopass.pem -set_serial 13 -out email.crt – This shows using a CA you have the cert and key for.

Using a Microsoft Enterprise CA Web UI:

Click on Request a Certificate Link
Click on the advanced certificate request link
Paste contents of CSR into top box (DO NOT include the Beginning and Ending lines!!!)
Click submit
Download the Base 64 Encoded Certificate and Chain, name them yournameB64.cer and yournameB64.p7b

If you used the Web UI then this is your Step 5:
5. openssl pkcs12 -export -out yourname.pfx -inkey yournamekey.p7b -in yournameB64.cer -certfile enterprise_ca.cer
5. ./openssl pkcs12 -export -descert -in email.crt -inkey enc_email.key -out email.pfx
This is exporting your new cert and key into a pfx file that is generally used to import into mail clients to support S/MIME.

h5. CSR Creation Info
Example answers:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:South Dakota
Locality Name (eg, city) []:Sioux Falls
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Company Inc
Organizational Unit Name (eg, section) []:Development
Common Name (e.g. server FQDN or YOUR name) []:John Doe
Email Address []:john.doe@company.com

Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Nov 302011
Mastering SSH Keys

Welcome to the wonderful world of SSH keys! If you don’t yet share my enthusiasm you soon will! SSH keys are a perfect way for you to control access to your machine, whether that be a very secure way for only you to have access, locking down other authorized users and preventing their passwords from getting distributed or stolen, or even allowing access to scripts for very specific purposes. SSH keys accomplish all of it and more. First, let me just lay out how simple SSH keys are in case you aren’t clear and maybe get discouraged by the length of this post. The basics of it are, you have a host server create a private and public key pair. You can optionally provide a password for the keys. You copy the public key to servers you wish to log on to, while keeping private key secured and on your system. Then when you login to the system the private key is used to decrypt the public (also with optional password) and you get in. It is that simple at its core, there are just a lot of various configuration options you can use to accomplish certain features or functionality that may be more desirable for you. The keys are very secure as using this method alone prevents against man-in-middle attacks and eliminates a few other possible break-ins from password only authentication.
The most common way I see SSH keys being used is as a password-less easy login for an admin to use for his remote linux systems. Although this is better than using a password only system you still are vulnerable to someone obtaining your private key and using it for instant access to any system with the corresponding public key. I prefer to use them as a type of two factor authentication for the ssh logins where I need to provide the password to the key plus (of course) have the actual key. Regardless, let’s get a basic configuration going and start using ssh-keys. First on the server side you can follow a config I’ve used in a previous post but general settings to ensure:

  • Port 2992 – (I always recommend changing the default port)
  • Make sure both “RSAAuthentication” and “PubkeyAuthentication” are both set to yes
  • PermitRootLogin no
  • AuthorizedKeysFile %h/.ssh/authorized_keys
  • I’d also only allow users in a certain group to use ssh for further protection so “AllowGroups sshlogin” for example and add users who will ssh into this system into the sshlogin group.

Now on your machine, the computer you will be ssh’ing from, you need to create the key pair and copy the public key to the host(s). Generate the key pair as follows:
Create the .ssh directory in your linux home directory if it doesn’t exist already. Ensure correct permissions with chmod 0700 .ssh. Then create the actual keys and when prompted specify a pass-phrase to use (This pass-phrase is the optional part if you just want to authenticate with the keys alone but as I mentioned above I prefer supplying one so you have a two factor authentication going on). Keep in mind this can be a passphrase not just a password.
ssh-keygen -b 2048 -t rsa -f ~/.ssh/MainKey
The -b specifies the length of the key and the -t the type (which is rsa or dsa).
I’d use RSA 2048 or if you are the more paranoid type RSA 4096. I won’t delve into the RSA/DSA debate except to say that I’ve done a LOT of reading on the topic and I’m choosing RSA keys and with sensitive servers I’d go with RSA at 4096.
[Update, Dec 2015] :: DSA is being deprecated in OpenSSH so I now strongly recommend RSA over it. If you are running version 6.5 or newer of openssh I’d actually recommend using ed25519 keys over RSA and you can read this blog post for more details. Typically you can check your version with ssh -V command. Your command to generate this key would simply be
ssh-keygen -t ed25519 -f ~/.ssh/MainKey
Note for Mac OSX users, the version of ssh is too old for ed25519 unless you are on OSX 10.11, El Capitan. If you are on an older Mac OSX version simply google for homebrew install of openssh to upgrade your version and use that.
[End Update]
Once the keys are created scp the public key to the server but copy the public key into the .ssh directory named as “authorized_keys” (or wherever you specified in sshd_config the AuthorizedKeysFile). You can append multiple public keys to the authorized keys file just cat mykey.pub >> authorized_keys for each public key.

Now you may have noticed above when generating the key pair we passed the -f option. This way we can name our private key something other than the default. If you do not specify the -f the default will be id_rsa with the corresponding id_rsa.pub. So now to ssh into our system we will want to specify the identity file to use as:
ssh -i /home/user/.ssh/MainKey user@myserver.com -p2992
The reason I want to show the more complicated use case of specifying a particular identity file is that you may want to have different key pairs for servers that are yours versus your employers’ or any other separation that would be important to you. You can simplify the management of which keys are used for which hosts by specifying in the ~/.ssh/config file.
vi ~/.ssh/config

Add both host names and their identity file as follows:
Host server1.myserver.com
IdentityFile ~/home/user/.ssh/MainKey
Host server1.workserver.com
IdentityFile ~/home/employee/.ssh/id_rsa

That’s all there is to your basic key based login! Now lets briefly go over many of the popular use cases for these ssh-keys so you can see how powerful and helpful they can be for you.

First, as I mentioned earlier, I frequently see administrators using these without a password so they can quickly login to the servers they manage. The problem is you are now solely relying on the security of that private key for complete access to those systems. I will strongly recommend (yet again) putting a pass-phrase on your keys and then if you want a quick password-less login then simply configure ssh-agent to cache your pass-phrase on your system for that session. For details just search for “ssh-agent cache password” and you’ll find a few examples. In this way if your key is somehow stolen that person still has no access to the servers as they don’t know the password, yet you have the efficiency of not typing a pass-phrase once you set up and configure ssh-agent to cache it on your system. Win-win here.

Next, we can limit by IP or IP subnet where users with ssh-keys are able to login FROM. So say you are the server admin and you have other users you manage on the server. Maybe you (especially after reading this) have a requirement that they only login with ssh-keys. Being security conscious you might have a few concerns, not the least are:

  • Did they create a passphrase or leave blank?
  • Will they give their key and pass to someone else to use?
  • What if their key is stolen or compromised?

One thing you can do if you just control the server with the public key, is limit by IP or subnet where they can be coming from. In this way if the users key is stolen it can’t be used anyway if that person isn’t trying to login from the same IP (or subnet) as where the original user is from. You can either wait till they login the first time and see what IP they are coming from or ask them to send you the subnet or possible IP’s they might come from. Once you have this IP or list of IP’s just open the authorized_keys file on your server and add a “from=” line to the beginning of their public key as shown below:

[root@mainserver .ssh]# cat authorized_keys
from="10.20.30.*,172.16.31.*,192.168.1.*" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEArAkcHTOXZiDxcJEHNmrJRoM2HJE9Rq1uoiVHuTSjgl0THp0UFDNepnmCvk5bX22KzjAUa8PWnq1ZDw5Uf9/i6N/SiXittnxT0dFnZGj6RRep5Ae3AmOJbg0i69XL9o2zBJnYo2JVPXJkCDhSvVWokZUn5QjaJiGNigP9plA1He94Slkhn2jTxx1iehx9Vy/ojnxsJDqpTa8hX1GQK/b1jzvWdN3Qg+EhlxBDfKSA8u4uGsPP+6hXGgFLRluG/6yizj8LDF1LRWIKYaBvaLNJ+720sAI9O4miHyxY4n3ghBDbULPoLGz5a6bYRJ9pY9i0ySQEmXNSD+0u31+fAaGGpQ== user@sauronseye

So if you have this public key and it is supposed to be used by a user at Company A and that user gives it to someone else at Company B or it is compromised by evil haxor, then that key whether it had a password or not will not allow them access. Then with your normal audits you do (right?) you see failed logins from a foreign IP and realize the key was compromised. Server is still secure your world is safe from harm :).

The last and yet another very common way of using ssh-keys is for specialized scripts use. You have a script on one server and at a certain point in the script you’d like it to kick off a command or script on another server. How do you do this? Well you can create an ssh-key specific for this purpose. Create your keys just as above but name its identity for the job at hand so you don’t get confused.
ssh-keygen -t rsa -f ~/.ssh/login_audit
For an example lets say you have primary internal server and you want it to go out and gather a report of all the last logins from your remote servers. You could put a script like this on the remote server:

# Script: last_logins.sh
last -a > /home/user/last_logins.barney
scp /home/user/last_logins.barney fred@mainserver:.
# End Script

Then in the authorized_keys file on the remote server edit the public key just as we added the “from=” line but instead use “command=”:

command="/home/user/scripts/last_logins.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding [rest of pub key...]

Now whenever you ssh in using that identity file the only thing it will do is run that command and exit. There are countless uses for this alone and being aware that you can use ssh-keys like this will help you realize WHEN you should use them like this.
There is a bunch more you can do with ssh and ssh-keys so use this as a starting point and take advantage of the tools at your disposal!

We discussed above using ssh-keys as two-factor authentication to login to a server via ssh. Some of you may be aware that Google offers a two factor authentication to login to your Google services (gmail, apps, etc.). They accomplish this with an application you can install called Google Authenticator.
If you don’t use google authenticator and 2-factor authenication on your google account you should strongly consider it. Gmail accounts are prime targets for being compromised and a lot of people use that account for a wide array of things that would make the compromise that much more damaging. Read some of the links below for starters:

Once you have google authenticator installed cause now you fear for the safety of your gmail… what else can you do with this thing? Well we can use it for two-factor authentication to login to our machines, much in the same way we used the ssh-keys.
I’ve found a great post that details how to do this and don’t forget to read the comments there as they include a lot of valuable information as well:

Linked also in that post is how to use a YubiKey to login with two factor authentication. These are all great and really secure methods to login and authenticate to your systems.

Hopefully you can realize the importance of two factor authentication to secure the things that really matter to you. There are countless hacks and exploits and more found every day. Trust me it is much easier to take the extra 10 minutes to set this up than the hours or days of recovery from a security breach. One needs to be proactive about security because if you are reactive then you are too late.

Jun 232011
Using AppArmor

Application Armor.  I’m sure you can imagine what this little utility does but allow me to elaborate a bit and explain why configuring and using AppArmor has become Part III of our little series.  First let’s quickly look at what we have so far if you’ve gone through Quick Secure Setup Part I and Quick Secure Setup Part II.  From the security standpoint your server has every non-essential port turned off.  Only applications you explicitly care about have their ports open through your firewall.  You are getting security updates within 24 hours and are reviewing the logs for anomalies.  For the ports that are open, fail2ban is checking the logs for errors and banning the offending IPs.  So what’s left?  Well a whole lot actually but we won’t concern ourselves with every little aspect of security.  We are securing the main attack vectors to a standard server sitting out on the internet and there is one attack left that will help you sleep better at night if you have a defense against it.

The zero day exploit.  You’ve done all this to your server and it’s locked down nice and tight.  Along comes a new exploit to apache, ntp, postfix, ssh, vsftpd, etcetera, ports you need open to run your services and before any security updates can be released to address the issue, the black hats out there have a tool ready to take advantage.  It might be an exploit that allows someone to crash the exploited service so, for example your web site would go down but no real compromise has taken place, all the way to something allowing them to escalate to superuser privileges.  You’ve gone to bed confident and secure and woke up with your server in some not happy place and you following right behind.  How do we alleviate this?  It’s the next most common thing that could happen that we’ve yet to address.  Oh, wait, you’ll skip this Part?  Exploits aren’t that common?  You might want to look at the US-CERT Cyber Security Bulletins and subscribe to their RSS feed.  Keep in mind despite all the efforts we take here you won’t be left with an impenetrable fortress.  There’d be much more to do for that.  What you will be left with however, is a server that would require someone to target it explicitly, with motivation, to find a way to compromise it.  Keep in mind, YOU will potentially be one of the weak links for your server’s security.  On a sufficiently secured server it may be much easier for an attacker to leverage social engineering tactics to gain some sort of access so be careful about the type of information you divulge on those social applications, chats, forums, etcetera.

Now lets get on with business, we have zero day exploits and application bugs to defend against.  Welcome AppArmor.

Useful AppArmor Links

Peruse the links above so you can start familiarizing yourself with what AppArmor does but let me explain in completely non-technical terms.  The applications on your system serve some function for you, the user.  Just as say a user will use a car to go from point A to point B.  Imagine the user has many options for getting from point A to point B just as an application may be able to do its job in different ways.  The best way might be for the user to take the highway arriving in great time and this might be the way programmed into the application.  But the user could drive the car on local roads or even off-road.  When the user does this they could be affecting other things not intended within the “system”.  For example the user could take the car onto private roads not intended for its traffic, or crash into a mailbox on its way.  All this would be bad and an exploited application could do the same type of things, going places it was never intended or affecting another process on your system it otherwise wouldn’t.  Now if we create an effective AppArmor profile for this application it’s like we’ve built a railroad directly from point A to point B.  The application, now a locomotive instead of a car, cannot leave the rails.  It has one allowed path it is now restricted to and its the path we defined by either laying down the rails or configuring the AppArmor profile.  So even if the application is exploited and wants to leave the rails to drive elsewhere, it simply can’t, it is contained.  This won’t be a step by step guide at building AppArmor profiles. I’ll give you a good overview and pointers but the best thing to do is read and most importantly in this case experiment for yourself. One of the better guides on AppArmor profiles I’ve found is Introduction to AppArmor by bodhi.zazen

First step, you should identify the applications or processes you would like to define an AppArmor profile for.  You could run a command like netstat -tulpn and take a look at the output. Anything that lists the Local Address as isn’t listening from the outside world but anything else is a candidate to AppArmor. Or you could look at ufw status verbose and see the ports you have open to the outside world. Any application listed there is a great candidate to AppArmor.  The first thing to run is apparmor_status. It will print out a nice listing of the default profiles on your server and tell you the profiles defined that are in enforce mode. On one of my servers just mysqld and ntpd are in enforce mode by default. There are two other main processes running that have external ports open, apache2 and sshd. So lets work on the steps required to create a profile for apache2, then you can follow that same framework for any other processes you want to AppArmor on your server.  First we run aa-genprof /usr/sbin/apache2 to start creating/editing the profile for apache.  Now with the later ubuntu versions there is a generic, very permissive profile already in place so it would build off of that. Finish the genprof and run aa-complain /etc/apparmor.d/usr.lib.apache2.mpm-prefork.apache2. This puts apache2 in complain mode. Now run your site for a while and do everything you normally do with it. You can also just leave apache in complain mode for a day or two as your site goes through its normal usage. Once you’ve done enough run aa-logprof and any “complaints” apache had regarding its profile you can now step through the wizard and add to the profile. To enforce the apache profile its simply aa-enforce /etc/apparmor.d/usr.lib.apache2.mpm-prefork.apache2. Apache is a special case with apparmor as there is an apparmor module you can enable a2enmod apparmor which then will enable you to lock down your virtual hosts correctly via AAHats. So for this site I went into the <Directory /var/www/mysite> area and added AAHatName blog.vigilcode.com, restarted apache and ran aa-logprof again and it stepped through new things to add to the profile and more specifically the new Hat.  You can find more detail on this process here.

Apache is more challenging to AppArmor because of the child processes is spawns and with virtualhosts each being another “server” in essence there is more to do and follow. When learning and getting familiar with AppArmor I think the best first step is to use your desktop and create an AppArmor profile for your web browser. It’s something you use all the time so can leave in complain mode for a while, then throw it in enforce and get accustomed to debugging issues by putting back in complain mode and either manually looking over the log or running aa-logprof to see what else came up.

This is definitely the least “quick” part of the Quick Secure Setup series but if you can take your time and master a few good profiles for your servers with the other steps taken so far then you won’t have many security worries.

Future: I’m currently keeping my eye on Tomoyo Linux as an AppArmor competitor so you might want to take a gander over there if for nothing else but for the knowledge.