Invariant Properties

In my last article, Setting Up SSH Identity Forwarding on Jump Hosts, I discussed how to improve security by eliminating the need to have SSH keys on your jump host by passing through SSH identity information. There’s no reason for your servers to have private SSH keys on them^[1] – and that means one less thing to worry about falling into the hands of an attacker with access to either your system or your backup media.

Using SSH identify forwarding is also much more convenient than having to reenter a password every time you hop from your jump host to a server but that was a secondary concern.

SSH identify forwarding has one serious drawback – ssh-agent still has a copy of your key on the jump host while you’re logged in. Your risk exposure is much lower than if you kept your private key on the jump host but it’s not eliminated.

There’s a solution to this: multi-factor authentication. An attacker with just your SSH identify information will still be unable to connect to the other systems.

The setup process is straightforward.

1. Install libpam-google-authenticator on the server.
2. Create the MFA key using google-authenticator -l ‘name@system’ where ‘name@system’ is meaningful to the user. This will display a scannable image on the console, the recovery codes, and the BASE-32 encoded key. It will also create a .google-authenticator file containing this information.
   • Scan the image with one or more smart devices with the Google Authenticator app (or something equivalent).
   • Add the BASE-32 encoded key to a “one time password (OTP)” field in 1Password (or something equivalent).
   • Copy the .google-authenticator file to a safe location.

   The PAM module uses the presence of this file as a marker of which users require MFA. You do not have to require MFA for all users although it would be a good idea.

3. Configure libpam-google-authenticator in /etc/pam.d. You want to add the following line to the appropriate files (e.g., login). During initial deployment add
   1.    auth required pam_google_authenticator.so nullok`

      auth required pam_google_authenticator.so nullok`

   Once all users have been set up change the entry to

   1.    auth required pam_google_authenticator.so no_increment_hotp

      auth required pam_google_authenticator.so no_increment_hotp

   This module has several other options, see the documentation for details.

4. Edit /etc/ssh/sshd_config and make the following changes:
   • ChallengeResponseAuthentication yes
   • AuthenticationMethods publickkey,keyboard-interactive
5. Test the new configuration. Do not log out of the recovery SSH session until you have verified that you can access the system with the new configuration. See note below if you use encrypted home directories.

You can deploy the ~/.google-authenticator file to multiple systems. The benefit is that this is easier to automate and only requires the user to track a single number – a major consideration if you’re using hardware fobs – but the drawback is that an attacker with access to the .google_authenticator file on one system will be able to get the MFA value for any system using the same file. A reasonable balance may be using the same file for servers in the same role (e.g., all database servers, all appservers, etc.) but different files for each category. That means someone with full access to the appservers will still be unable to access the database servers. This isn’t an unreasonable burden for sysadmins using an authenticator app or password manager but still require multiple hardware fobs.

Cautions

This is not a panacea. An attacker with shell access or backup media can see your .google_authenticator file. You might want to move the file from the default location in order to prevent attackers from finding the file in its default location using scripts. See the libpam-google-authenticator documentation for details.

Encrypted home directories can leave you unable to access your system. This won’t be obvious at the time if your recovery session uses the same account as the account you’re editing. The solution is to move the location of the .google_authenticator file outside of your home directory. See the libpam-google-authenticator documentation for details.

A misconfiguration can leave you unable to access your system. Always have an established recovery session in a root shell before attempting this. Don’t count on being able to run ‘sudo’ – it’s possible for a misconfiguration to leave you unable to run sudo in your recovery session. (Ask me how I know.) Ideally deployment should be automated using a well-tested puppet or ansible script.

Clock skew can leave you unable to access your system. Make sure you are running NTP. In highly secure environments the NTP port may be blocked – in this case you must ensure that the system has a reliable time via a different mechanism.

The .google_authenticator file should never be backed up. If you are on an ext2-based filesystem change the extended attributes with “sudo chattr -dis .google-authenticator”. That marks the file as “no dump”, “immutable”, and “secure delete”. Some backup software honors this extended attribute, other backup software requires you to explicitly add this file to a “don’t archive” list.

Alternatives

The libpam-google-authenticator module requires the user to have a hardware fob or authenticator app. Many people now prefer an approach where the system sends a text message containing a one-time code to a preconfigured number. This is not difficult in the AWS ecosystem since it’s easy to send a text message using the SNS service.

Doing this has the potential benefit of eliminating the need for files containing sensitive information. It depends on whether you’re willing to give up flexibility – using the standard TOPT code allows the user to continue using a hardware fob or authenticator app but requires you keep the sensitive files. Using a random code eliminates the need for a sensitive file but prevents the use of alternate devices.

The potential drawbacks are that you’re replacing one piece of sensitive information with another – the phone number to receive the text message, that the user may not be able to receive the text message, e.g., limited phone service, and a really sophisticated attacker could spoof the phone service and intercept the code.

Another concern is that this approach requires time for the message to be sent, received, and entered. If using a TOPT-based code and a standard ‘tick’ there may only be 31 seconds for this to complete.

I do not know if a PAM module already exists that implements this approach but it would not be difficult to implement if you have access to AWS SNS.

Another alternative is to extend this module so it stores the TOPT secrets in either a database (e.g., sqlite or sleepycat) or retrieves it via LDAP. The former makes it harder for an attacker to discover the secret, the latter removes the secret from the server entirely albeit at the cost of requiring an additional server.

The bottom line

Multi-factor authentication is a valuable security tool. It can make life much more difficult for attackers – the mere presence of MFA may scare off a potential attacker. It allows you to check off boxes in security audits.

However it requires an unencrypted secret so it is worthless against anyone with filesystem or backup media access. At least system passwords are hashed with salts and SSH keys have encryption passwords. TOPT secrets have neither. This means that unauthorized disclosure is immediately fatal since there’s no need to crack the hash or passphrase.

Footnotes

[1] There is one exception to the “no SSH keys on the servers” rule. You might need keys with restricted access for routine operations, e.g., if a cron task needs to execute a command on a different server. In these cases you’ll need a SSH key for the cron task but the destination server should restrict the commands that can be executed using that key.

One of the standard security checklist items for AWS EC2 instances is that they should never permit direct SSH access. Instead you should create a single “jump host” that runs nothing but an SSH daemon and is only visible from trusted fixed IP addresses or your VPN. The latter allows access from anywhere but does not introduce a security weakness if you use your own OpenVPN server. That only costs $5/month with a ‘droplet’ at Digital Ocean. (I haven’t been able to get properly VPNs working on EC2 instances.)

The servers all run SSH daemons but you either configure them to only listen to their private IP address and/or configure the VPC rules to only allow access via their private IP address. Their SSH daemon is never visible to the outside world.

To access your servers you first log into the jump host, then log into the final destination. You’re limited to the command line but that shouldn’t be an issue. Some people will enable port forwarding through the jump host but I personally consider SSH port forwarding a temporary solution at best – if you need anything other than the command line you should commit to a VPN solution.

The cost of this approach is trivial. It only requires an AWS EC2 ‘nano’ instance (about $5/month), and even that can be avoided if you only launch this server as needed.

Putting your SSH keys on the jump host would significantly reduce the security of your system. We want to carry our SSH identifies with us as we log into a series of servers. Fortunately that’s easy to do. It only requires changes to two files.

First, we must tell ssh to forward our SSH credentials. This can be done globally or on a per-host basis.

/etc/ssh/ssh_config or ~/.ssh/config

ForwardAgent yes

Second, we must tell the ssh daemon to allow us to forward our SSH credentials.

/etc/ssh/sshd_config

AllowAgentForwarding yes

Alternative: NetCat

Local Linux god Kevin Fenzi expressed concern with the ssh-agent approach since anyone with root access on the jump host can reuse your SSH credentials while you’re logged in and recommends using a local ssh configuration that will transparently trigger nc (netcat) on the jump host so you’ll immediately jump to the final destination.

I was unable to get it to work but here’s his example:

Host your-internal-hostname-or-ip
   HostName %h
   ProxyCommand ssh -q yourusename@your-jump-host /usr/bin/nc %h 22

One of the biggest headaches when working with Kerberos is that you need to set up external files in order to use it. That should be a simple one-time change but it can introduce subtle issues such as forcing developers to be on the corporate VPN when doing a build on their laptop.

The Hadoop developers already have a solution – the “MiniKDC” embedded KDC server. This class can be used to create a temporary KDC in the build environment that eliminates any need for external files or network resources. This approach comes at a cost – on my system it takes about 15 seconds to stand up the embedded server. You don’t want to run these tests on every build but a brief delay during integration tests is better than introducing a dependency on a VPN and a running server.

Update regarding ticket caches on 11/22/2017

Important update on ticket caches (and TGT) on 11/22/2017. I need to emphasize that the standard implementation of the Krb5LoginModule does not create ticket cache files. That may not be clear below. I will post a followup article that discusses using the external kinit program to create Kerberos ticket caches.

Embedded Servers and JUnit 4 Rules

Modern test frameworks have a way to stand up test resources before test and tear them down afterwards. With JUnit 4 this is done with Rules. A rule is an annotation that the test runner recognizes and knows how to use. The details are different in other test frameworks (or JUnit 5) but the underlying concepts are the same.

An embedded KDC is a class-level external resource.

public class EmbeddedKdcResource extends ExternalResource {
    private final File baseDir;
    private MiniKdc kdc;

    public EmbeddedKdcResource() {
        try {
            baseDir = Files.createTempDirectory("mini-kdc_").toFile();
        } catch (IOException e) {
            // throw AssertionError so we don't have to deal with handling declared
            // exceptions when creating a @ClassRule object.
            throw new AssertionError("unable to create temporary directory: " + e.getMessage());
        }
    }

    /***
     * Start KDC.
     */
    @Override
    public void before() throws Exception {

        final Properties kdcConf = MiniKdc.createConf();
        kdcConf.setProperty(MiniKdc.INSTANCE, "DefaultKrbServer");
        kdcConf.setProperty(MiniKdc.ORG_NAME, "EMBEDDED");
        kdcConf.setProperty(MiniKdc.ORG_DOMAIN, "INVARIANTPROPERTIES.COM");

        // several sources say to use extremely short lifetimes in test environment.
        // however setting these values results in errors.
        //kdcConf.setProperty(MiniKdc.MAX_TICKET_LIFETIME, "15_000");
        //kdcConf.setProperty(MiniKdc.MAX_RENEWABLE_LIFETIME, "30_000");

        kdc = new MiniKdc(kdcConf, baseDir);
        kdc.start();

        // this is the standard way to set the default location of the JAAS config file.
        // we don't need to do this since we handle it programmatically.
        //System.setProperty("java.security.krb5.conf", kdc.getKrb5conf().getAbsolutePath());
    }

    /**
     * Shut down KDC, delete temporary directory.
     */
    @Override
    public void after() {
        if (kdc != null) {
            kdc.stop();
        }
    }

    /**
     * Get realm.
     */
    public String getRealm() {
        return kdc.getRealm();
    }

    /**
     * Create a keytab file with entries for specified user(s).
     *
     * @param keytabFile
     * @param names
     * @throws Exception
     */
    public void createKeytabFile(File keytabFile, String... names) throws Exception {
        kdc.createPrincipal(keytabFile, names);
    }
}

Functional Tests

Once we have an embedded KDC we can quickly write tests that attempt to get a JAAS LoginContext using Kerberos authentication. We call it a success if LoginContext#login() succeeds.

public class BasicKdcTest {

    @ClassRule
    public static final TemporaryFolder tmpDir = new TemporaryFolder();

    @ClassRule
    public static final EmbeddedKdcResource kdc = new EmbeddedKdcResource();

    private static KerberosPrincipal alice;
    private static KerberosPrincipal bob;
    private static File keytabFile;
    private static File ticketCacheFile;

    private KerberosUtilities utils = new KerberosUtilities();

    @BeforeClass
    public static void createKeytabs() throws Exception {
        // create Kerberos principal and keytab filename.
        alice = new KerberosPrincipal("alice@" + kdc.getRealm());
        bob = new KerberosPrincipal("bob@" + kdc.getRealm());
        keytabFile = tmpDir.newFile("users.keytab");
        ticketCacheFile = tmpDir.newFile("krb5cc_alice");

        // create keytab file containing key for Alice but not Bob.
        kdc.createKeytabFile(keytabFile, "alice");

        assertThat("ticket cache does not exist", ticketCacheFile.exists(), equalTo(true));
    }

    /**
     * Test LoginContext login without TGT ticket (success).
     *
     * @throws LoginException
     */
    @Test
    public void testLoginWithoutTgtSuccess() throws LoginException {
        final LoginContext lc = utils.getKerberosLoginContext(alice, keytabFile);
        lc.login();
        assertThat("subject does not contain expected principal", lc.getSubject().getPrincipals(),
                contains(alice));
        lc.logout();
    }

    /**
     * Test LoginContext login without TGT ticket (unknown user). This only
     * tests for missing keytab entry, not a valid keytab file with an unknown user.
     *
     * @throws LoginException
     */
    @Test(expected = LoginException.class)
    public void testLoginWithoutTgtUnknownUser() throws LoginException {
        @SuppressWarnings("unused")
        final LoginContext lc = utils.getKerberosLoginContext(bob, keytabFile);
    }

    /**
     * Test getKeyTab() method (success)
     */
    @Test
    public void testGetKeyTabSuccess() throws LoginException {
        assertThat("failed to see key", utils.getKeyTab(alice, keytabFile), notNullValue());
    }

    /**
     * Test getKeyTab() method (unknown user)
     */
    @Test(expected = LoginException.class)
    public void testGetKeyTabUnknownUser() throws LoginException {
        assertThat("failed to see key", utils.getKeyTab(bob, keytabFile), notNullValue());
    }
}

Next Steps

The next article will discuss the Apache Hadoop UserGroupInformation class and how it connects to JAAS authentication.

Source

You can download the source for this article here: JAAS with Kerberos; Unit Test using Apache Hadoop Mini-KDC.

Java’s JAAS abstraction is a powerful tool to handle authentication but it has two major weaknesses in practice. First, nearly all of the discussions on how to use it assume that the developer can write the JAAS configuration file to a secure location. That may not be easy in a hosted environment. Second, JAAS has some unexpected behavior that made sense at the time but which can bite developers today. Neither is difficult to overcome once you know the solution.

This article will discuss the solution to these problems and give a concrete example using Kerberos authentication. Kerberos is widely used in the Hadoop ecosystem and a future article will discuss how to use the Hadoop-specific UserGroupInformation class.

Limitations

Unfortunately this code does not completely eliminate the need for external files. First, we must still provide an explicit Kerberos keytab file. I will update this article if I find an approach that eliminates this limitation.

Second, we must still provide an external krb5.conf configuration file. This is required by the Krb5LoginModule class and I think we’re limited to changing the location of this file.

Update regarding ticket caches on 11/22/2017

Important update on ticket caches (and TGT) on 11/22/2017. I need to emphasize that the standard implementation of the Krb5LoginModule does not create ticket cache files. That may not be clear below. I will post a followup article that discusses using the external kinit program to create Kerberos ticket caches.

The JAAS Configuration Class

The first issue to discuss is eliminating the need for a JAAS configuration file. We want to be able to configure JAAS programmatically, perhaps using information provided via a traditional database or a cloud discovery service such as Spring Cloud Config. JAAS provides an oft-overlooked class that can replace the external configuration: javax.security.auth.login.Configuration. The most general solution is to create a class that accepts a Map in its constructor and uses it to create an array of AppConfigurationEntry values.

class CustomLoginConfiguration extends javax.security.auth.login.Configuration {
    private static final String SECURITY_AUTH_MODULE_KRB5_LOGIN_MODULE =
            "com.sun.security.auth.module.Krb5LoginModule";

    private final Map<String, String> entries = new HashMap<>();

    /**
     * Constructor taking a Map of parameters
     */
    public CustomLoginConfiguration(Map<String, Map<String, String>> params) {
        for (Map.Entry<String, Map<String, String>gt; entry : params.entrySet()) {
            entries.put(entry.getKey(),
                    new AppConfigurationEntry(SECURITY_AUTH_MODULE_KRB5_LOGIN_MODULE,
                AppConfigurationEntry.LoginModuleControlFlag.REQUIRED, entry.getValue()));
        }
    }

    /**
     * Get entry.
     */
    @Override
    public AppConfigurationEntry[] getAppConfigurationEntry(String name) {
        if (entries.containsKey(name)) {
            return new AppConfigurationEntry[] { entries.get(name) };
        }
        return new AppConfigurationEntry[0];
    }
}

In practice we’ll only need one or two JAAS configurations in our application and it may be more maintainable to write a convenience class. It is very easy to use an external configuration file to identify the required properties and then convert the final file into a static method that populates the Map.

import static java.lang.Boolean.TRUE;

class Krb5WithKeytabLoginConfiguration extends CustomLoginConfiguration {

    /**
     * Constructor taking basic Kerberos properties.
     *
     * @param serviceName JAAS service name
     * @param principal Kerberos principal
     * @param keytabFile keytab file containing key for this principal
     */
    public Krb5WithKeytabLoginConfiguration(String serviceName, KerberosPrincipal principal, File keytabFile);
       super(serviceName, makeMap(principal, keytabFile);
    }

    /**
     * Static method that creates the Map required by the parent class.
     *
     * @param principal Kerberos principal
     * @param keytabFile keytab file containing key for this principal
     */
    private static Map<String, String> makeMap(KerberosPrincipal principal, File keytabFile) {
        final Map<String, String> map = new HashMap<gt;();

        // this is the basic Kerberos information
        map.put("principal", principal.getName());
        map.put("useKeyTab", TRUE.toString());
        map.put("keyTab", keytabFile.getAbsolutePath());

        // 'fail fast'
        map.put("refreshKrb5Config", TRUE.toString());

        // we're doing everything programmatically so we never want to prompt the user.
        map.put("doNotPrompt", TRUE.toString());
        return map;
    }
}

The JAAS CallbackHandler and LoginContext Classes

A nasty surprise for many developers is that the JAAS implementation will fall back to a non-trivial default implementation if a custom CallbackHandler is not provided. At best this will result in confusing error messages, at worst an attacker can override the default implementation with one that is much more open than the developer intended.

Fortunately this is easy to handle when creating the JAAS LoginContext. We could use an empty handler method but it doesn’t hurt to log any messages in case there’s a problem.

This example uses the LoginContext method that takes a suggested Subject. It may be possible to provide the keytab information via the Subject’s private credentials instead of passing in an explicit file location via the ‘keyTab’ property but I haven’t found it yet. I’ve left the code in place in case it will help others.

class KerberosUtilities {
    private static final Logger LOG = LoggerFactory.getLogger(KerberosUtilities.class);

    /**
     * Get JAAS LoginContext for specified Kerberos parameters
     *
     * @param principal Kerberos principal
     * @param keytabFile keytab file containing key for this principal
     */
    public LoginContext getKerberosLoginContext(KerberosPrincipal principal, File keytabFile)
            throws LoginException, ConfigurationException {

        final KeyTab keytab = getKeyTab(keytabFile, principal);

        // create Subject containing basic Kerberos parameters.
        final Set<Principal> principals = Collections.<Principal> singleton(principal);
        final Set<?> pubCredentials = Collections.emptySet();
        final Set<?> privCredentials = Collections.<Object> singleton(keytab);
        final Subject subject = new Subject(false, principals, pubCredentials, privCredentials);

        // create LoginContext using this subject.
        final String serviceName = "krb5";
        final LoginContext lc = new LoginContext(serviceName, subject,
                new CallbackHandler() {
                    public void handle(Callback[] callbacks) {
                        for (Callback callback : callbacks) {
                            if (callback instanceof TextOutputCallback) {
                                LOG.info(((TextOutputCallback) callback).getMessage());
                            }
                        }
                    }
                }, new Krb5KeytabLoginConfiguration(serviceName, principal, keytabFile);

        return lc;
    }

    /**
     * Convenience method that verifies keytab file exists, is readable, and contains appropriate entry.
     */
    public KeyTab getKeyTab(File keytabFile, KerberosPrincipal principal)
            throws LoginException {

        if (!keytabFile.exists() || !keytabFile.canRead()) {
            throw new LoginException("specified file does not exist or cannot be read");
        }

        // verify keytab file exists
        KeyTab keytab = KeyTab.getInstance(principal, keytabFile);
        if (!keytab.exists()) {
            throw new LoginException("specified file is not a keytab file");
        }

        // verify keytab file actually contains at least one key for this principal.
        KerberosKey[] keys = keytab.getKeys(principal);
        if (keys.length == 0) {
            throw new LoginException("keytab file does not contain required entry");
        }

        // destroy keys since we don't need them, we just need to make sure they exist.
        for (KerberosKey key : keys) {
            try {
                key.destroy();
            } catch (DestroyFailedException e) {
                LOG.debug("unable to destroy key");
            }
        }

        return keytab;
    }
}

The Final Bits for Kerberos

There is one final problem. The Krb5LoginModule expects the Kerberos configuration file to be located at a standard location, typically /etc/krb5.conf on Linux systems. This can be overridden with the java.security.krb5.kdc system property.

The default realm is usually set in the Kerberos configuration file. You can override it with the java.security.krb5.realm system property.

Cloudera (Hadoop Cluster)

You must set one additional system property, at least when using Cloudera clients with Hive:

• javax.security.auth.useSubjectCredsOnly=false

For more information on this see Hive JDBC client error when connecting to Kerberos Cloudera cluster .

Debugging

Finally there are several useful system properties if you are stuck:

• sun.security.krb5.debug=true
• java.security.debug=gssloginconfig,configfile,configparser,logincontext

Next Steps

The next article will discuss writing unit tests using an embedded KDC server.

Source

You can download the source for this article here: JAAS with Kerberos; Unit Test using Apache Hadoop Mini-KDC.