Invariant Properties

In my last article, Setting Up SSH Identity Forwarding on Jump Hosts, I discussed how to improve security by eliminating the need to have SSH keys on your jump host by passing through SSH identity information. There’s no reason for your servers to have private SSH keys on them^[1] – and that means one less thing to worry about falling into the hands of an attacker with access to either your system or your backup media.

Using SSH identify forwarding is also much more convenient than having to reenter a password every time you hop from your jump host to a server but that was a secondary concern.

SSH identify forwarding has one serious drawback – ssh-agent still has a copy of your key on the jump host while you’re logged in. Your risk exposure is much lower than if you kept your private key on the jump host but it’s not eliminated.

There’s a solution to this: multi-factor authentication. An attacker with just your SSH identify information will still be unable to connect to the other systems.

The setup process is straightforward.

1. Install libpam-google-authenticator on the server.
2. Create the MFA key using google-authenticator -l ‘name@system’ where ‘name@system’ is meaningful to the user. This will display a scannable image on the console, the recovery codes, and the BASE-32 encoded key. It will also create a .google-authenticator file containing this information.
   • Scan the image with one or more smart devices with the Google Authenticator app (or something equivalent).
   • Add the BASE-32 encoded key to a “one time password (OTP)” field in 1Password (or something equivalent).
   • Copy the .google-authenticator file to a safe location.

   The PAM module uses the presence of this file as a marker of which users require MFA. You do not have to require MFA for all users although it would be a good idea.

3. Configure libpam-google-authenticator in /etc/pam.d. You want to add the following line to the appropriate files (e.g., login). During initial deployment add
   1.    auth required pam_google_authenticator.so nullok`

      auth required pam_google_authenticator.so nullok`

   Once all users have been set up change the entry to

   1.    auth required pam_google_authenticator.so no_increment_hotp

      auth required pam_google_authenticator.so no_increment_hotp

   This module has several other options, see the documentation for details.

4. Edit /etc/ssh/sshd_config and make the following changes:
   • ChallengeResponseAuthentication yes
   • AuthenticationMethods publickkey,keyboard-interactive
5. Test the new configuration. Do not log out of the recovery SSH session until you have verified that you can access the system with the new configuration. See note below if you use encrypted home directories.

You can deploy the ~/.google-authenticator file to multiple systems. The benefit is that this is easier to automate and only requires the user to track a single number – a major consideration if you’re using hardware fobs – but the drawback is that an attacker with access to the .google_authenticator file on one system will be able to get the MFA value for any system using the same file. A reasonable balance may be using the same file for servers in the same role (e.g., all database servers, all appservers, etc.) but different files for each category. That means someone with full access to the appservers will still be unable to access the database servers. This isn’t an unreasonable burden for sysadmins using an authenticator app or password manager but still require multiple hardware fobs.

Cautions

This is not a panacea. An attacker with shell access or backup media can see your .google_authenticator file. You might want to move the file from the default location in order to prevent attackers from finding the file in its default location using scripts. See the libpam-google-authenticator documentation for details.

Encrypted home directories can leave you unable to access your system. This won’t be obvious at the time if your recovery session uses the same account as the account you’re editing. The solution is to move the location of the .google_authenticator file outside of your home directory. See the libpam-google-authenticator documentation for details.

A misconfiguration can leave you unable to access your system. Always have an established recovery session in a root shell before attempting this. Don’t count on being able to run ‘sudo’ – it’s possible for a misconfiguration to leave you unable to run sudo in your recovery session. (Ask me how I know.) Ideally deployment should be automated using a well-tested puppet or ansible script.

Clock skew can leave you unable to access your system. Make sure you are running NTP. In highly secure environments the NTP port may be blocked – in this case you must ensure that the system has a reliable time via a different mechanism.

The .google_authenticator file should never be backed up. If you are on an ext2-based filesystem change the extended attributes with “sudo chattr -dis .google-authenticator”. That marks the file as “no dump”, “immutable”, and “secure delete”. Some backup software honors this extended attribute, other backup software requires you to explicitly add this file to a “don’t archive” list.

Alternatives

The libpam-google-authenticator module requires the user to have a hardware fob or authenticator app. Many people now prefer an approach where the system sends a text message containing a one-time code to a preconfigured number. This is not difficult in the AWS ecosystem since it’s easy to send a text message using the SNS service.

Doing this has the potential benefit of eliminating the need for files containing sensitive information. It depends on whether you’re willing to give up flexibility – using the standard TOPT code allows the user to continue using a hardware fob or authenticator app but requires you keep the sensitive files. Using a random code eliminates the need for a sensitive file but prevents the use of alternate devices.

The potential drawbacks are that you’re replacing one piece of sensitive information with another – the phone number to receive the text message, that the user may not be able to receive the text message, e.g., limited phone service, and a really sophisticated attacker could spoof the phone service and intercept the code.

Another concern is that this approach requires time for the message to be sent, received, and entered. If using a TOPT-based code and a standard ‘tick’ there may only be 31 seconds for this to complete.

I do not know if a PAM module already exists that implements this approach but it would not be difficult to implement if you have access to AWS SNS.

Another alternative is to extend this module so it stores the TOPT secrets in either a database (e.g., sqlite or sleepycat) or retrieves it via LDAP. The former makes it harder for an attacker to discover the secret, the latter removes the secret from the server entirely albeit at the cost of requiring an additional server.

The bottom line

Multi-factor authentication is a valuable security tool. It can make life much more difficult for attackers – the mere presence of MFA may scare off a potential attacker. It allows you to check off boxes in security audits.

However it requires an unencrypted secret so it is worthless against anyone with filesystem or backup media access. At least system passwords are hashed with salts and SSH keys have encryption passwords. TOPT secrets have neither. This means that unauthorized disclosure is immediately fatal since there’s no need to crack the hash or passphrase.

Footnotes

[1] There is one exception to the “no SSH keys on the servers” rule. You might need keys with restricted access for routine operations, e.g., if a cron task needs to execute a command on a different server. In these cases you’ll need a SSH key for the cron task but the destination server should restrict the commands that can be executed using that key.