Invariant Properties

I’m sure I’ve discussed this a number of years ago but a question came up after the recent Boulder Linux User Group meeting and I decided this would be a good time to revisit it.

The question is how do you protected sensitive information from illicit insertion or modification when the attacker has full SQL access as the website user?

Important: I am focused on approaches we can use in the database itself, not our application, since the former will protect our data even if an attacker has full access to the database. These approaches are invisible to our database frameworks, e.g., JPA, once we have created the tables.

An Approach Without Triggers

At a minimum we can ensure that the database was properly configured with multiple users:

app_owner – owns the schema and tables. Often does not have INSERT/UPDATE/DELETE (or even SELECT) privileges on the tables.

app_user – owns the data but cannot modify the schema, tables, etc.

We can make this much more secure by splitting app_user into two users, app_reader and app_writer. The former user only has SELECT privileges on the tables. This is the only account used by user-facing code. The app_writer user adds INSERT/UPDATE/DELETE privileges and is only used by the methods that actually need to modify the data. Data is typically read so much more often that it is written that it often makes sense to view an application as actually two (or more) separate but related applications. In fact they may be – you can improve security by handling any data manipulation via microservices only visible to the application.

There is a big downside to this – modern database frameworks, e.g., JPA or Hibernate, make heavy use of caching to improve performance. You need to ensure that the the cache is properly updated in the app_reader cache whenever the corresponding record(s) are updated in the app_writer account.

Security Defense

This is highly database specific – does the database maintain logs that show when a user attempts to perform a non-permitted action? If so you can watch the logs on the app_reader account. Any attempt to insert or update data is a strong indication of an attacker.

Triggers Based On Related Information

A 3NF (or higher) database requires that each column be independent. In practice we often perform partial denormalization for performance reasons, e.g., adding a column for the day of the week in addition to the full date. We can easily compute the former from the latter but it takes time and can’t be indexed.

There’s a risk that a bug or intruder will introduce inconsistencies. One common solution is to use an INSERT OR UPDATE trigger that calculates the value at the time the data is inserted into the database. E.g.,

CREATE FUNCTION calculate_day_of_week() ....

CREATE TABLE date_with_dow (
    date text,
    dow  text
);

CREATE FUNCTION set_day_of_week() RETURNS trigger AS $$
    BEGIN
        NEW.date = OLD.date;
        NEW.dow = calculate_day_of_week(OLD.date);
        RETURN NEW;
    END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER set_day_of_week BEFORE INSERT OR UPDATE ON date_with_dow
   FOR EACH ROW EXECUTE PROCEDURE set_day_of_week();

This ensures that the day of week is properly set. A software bug, or attacker, can try specifying an invalid value but they’ll fail.

Of course we don’t really care (much) if the day of the week is incorrect. However there are other times when we care a great deal, e.g., cached attributes from digital certificates. If someone can insert a certificate with mismatched cached values, esp. if it they can replace an existing table entry, then they can do a lot of damage if the code doesn’t assume that the database could be corrupted and thus perform its own validation checks on everything it gets back. (First rule of security: never trust anything.) Even with tests we’ll only know that the data has been corrupted, not when and not how broadly.

Security Defense

Developers are information packrats. Can we learn anything from the provided day of week value?

Yes. It’s a huge red flag if the provided value doesn’t match the calculated value (modulo planned exceptions, e.g., passing null or a sentinel value to indicate that the application is deferring to the database). It’s easy to add a quick test:

CREATE FUNCTION calculate_day_of_week() ....

-- user-defined function that can do anything from adding an entry into
-- a table to sending out an email, SMS, etc., alert
CREATE FUNCTION security_alert() ....

CREATE TABLE date_with_dow (
    date text,
    dow  text
);

CREATE FUNCTION set_day_of_week() RETURNS rigger AS $$
    DECLARE
        calculated_dow text;
    BEGIN
        NEW.date = OLD.date;
        NEW.dow = calculate_day_of_week(OLD.date);
        IF (NEW.dow  OLD.date) THEN
            security_alert("bad dow value!");
            RETURN null;
        END IF;
        RETURN NEW;
    END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER set_day_of_week BEFORE INSERT OR UPDATE ON date_with_dow
    FOR EACH ROW EXECUTE PROCEDURE set_day_of_week();

Sidenote: check out your database documentation for more ideas. For instance many applications use @PrePersist annotations to autofill creationDate and lastUpdateDate. It’s easy to do this via a trigger – and using a trigger ensures that the data updated even if an attacker does it via SQL injection or direct access. More impressively you can write audit information to a separate table, perhaps even in a separate schema that the app_user only has INSERT privileges for in order to prevent an attacker from learning what the system has learned about them, much less altering or deleting that information.

I’ve written triggers that generate XML representations of the OLD and NEW values and write them to an audit table together with date, etc. On INSERT the OLD data is null, on DELETE the NEW data is null. Using XML allows us to use a common audit table (table name is just a field) and potentially allows you to add transaction id, etc.

It is then easy to use a bit of simple XML diff code to see exactly what changed when by reviewing the audit table.

Resources:

• PostgreSQL
• MySQL
• Oracle

Triggers Based On Secrets

What about tables where there’s no “related” columns? Can we use a trigger to detect an illicit attempt to INSERT or UPDATE a record?

Yes!

In this case we want to add an extra column to the table. It can be anything – the sole purpose is to create a way to pass a validation token to the trigger.

What are validation tokens?

A validation token can be anything you want. A few examples are:

A constant – this is the easiest but will be powerful as long as you can keep it secret. An example is ’42’. An obvious variant is the sum of several of the other columns of the table. This value should not be written to the database or it will be exposed to anyone with SELECT privileges.

A time-based value – your webserver and database will have closely synced clocks so you can use a time-based protocol such as Time-based One-time Password (TOTP) Algorithm. If both the database and application servers use NTP you can keep the window as small as a few seconds. Just remember to include one tick on either side when validating the token – NTP keeps the clocks synchronized but there can still be a very small skew plus network latency to consider.

Note: TOTP requires a shared secret and is independent of the contents of the INSERT or UPDATE statement.

You can save a time-based value but it is meaningless without a timestamp – and some algorithms can be cracked if you have a series of values and the starting time.

An HMAC value – most people will be familiar with standard cryptographic hashes such as MD-5 or SHA-1 (both considered cracked) or SHA-256. They’re powerful tools – but in part because everyone will compute the same values given the same input.

In our case we want an HMAC – it is a cryptographically strong message digest that also requires an encryption key. An attacker cannot generate their own HMAC but anyone with the corresponding digital certificate can verify one. An HMAC value requires something in the message to be processed and it needs to be intrinsic to the value of the record. For instance a digital certificate, a PDF document, even a hashed password. Don’t use it to hash the primary key or any value that can be readily reused.

You can freely save an HMAC value.

Subsequent validation

We would like to know that values haven’t been corrupted, e.g., by an attacker knowledgeable enough to disable the trigger, insert bad values, and then restore the trigger. The last step is important since we can / should run periodic scans to ensure all security-related features like these database triggers are still in place. Can we use these techniques to validate the records after the fact?

Constant value: no.

Time-base value: only if we record a timestamp as well, and if we do then we have to assume that the secret has been compromised. So… no.

HMAC value: yes.

Backups and Restorations

Backups and restorations have the same problems as subsequent validations. You can’t allow any magic values to be backed up (or an attacker could learn it by stealing the backup media) and you can’t allow the time-based values plus timestamps to be backed up (or an attacker could learn the shared secret by stealing the backup media). That means you would need to disable the trigger when restoring data to the database and you can’t verify that it’s properly validated afterwards. Remember: you can’t trust backup media!

The exception is HMAC tokens. They can be safely backed up and restored even if the triggers are in place.

Security Defense

You can add a token column to any table. As always it’s a balance between security and convenience and the less powerful techniques may be Good Enough for your needs. But for highly sensitive records, esp. those that are inserted or updated relatively infrequently, an HMAC token may be a good investment.

Implementation-wise: on the application side you can write a @PrePersist method that handles the creation of the TOTP or HMAC token. It’s a standard calculation and the biggest issue, as always, is key management. On the database side you’ll need to have a crypto library that supports whatever token method you choose.

Shadow Tables

Finally, there are two concerns with the last approach. First, it requires you to have crypto libraries available in your database. That may not be the case. Second if a value is inserted it’s impossible know for sure that it was your application that inserted it.

There’s a solution to this which is entirely app-side. It might not give you immediate notification of a problem but it still gives you some strong protection when you read the data.

You start as above – add a @PrePersist method that calculates an HMAC code. Only now you edit the domain bean so that the HMAC column uses a @SecondaryTable instead of the main table. (I think you can even specify a different schema if you want even higher security.) From the data perspective this is just two tables with a 1:1 relationship but from the source code perspective it’s still a single object.

Putting this into a separate table, if not a separate schema as well, means that a casual attacker will not know that it is there. They might succeed in inserting or modifying data but not realize that the changes will be detected even if audit triggers are disabled.

The final step is adding a @PostLoad method that verifies the HMAC code. If it’s good you can have confidence the data hasn’t been corrupted. If it’s incorrect or missing you know there’s a problem and you shouldn’t trust it.

For advanced users the developer won’t even know that the extra data is present – you can do a lot with AOP and some teams are organized so that the developers write unsecured code and a security team – which is focused entirely on security, not features – is responsible for adding security entirely through AOP code interwoven into the existing code. But that’s a topic for a future blog….

The possibility of a massive twitter password breach brings up a basic requirement in our world. I discussed encrypting personally identifiable information (PII) last August but didn’t mention login information specifically.

Encrypt your usernames and email addresses.

Use a unique, random salt/IV for each user.

This won’t stop a dedicated attacker with resources but it will slow them down and that buys time for you to discover the breach and for your users to change their passwords. Maybe not much time, maybe a week instead of a few days until a majority of passwords is cracked, but that could make a big difference for a lot of users.

How do I find a user if their username and email is encrypted?

This is easy to answer – also store a hash of each user’s username and email. Think HashMap – not unique identifier – it’s okay if there are a few users in each ‘bin’. In fact this is desirable since it makes the job of the attacker a little bit harder. You can use the last few bytes of a SHA hash.

You can efficiently determine the correct user – compute the password hash for each potential match using their unique hash salt. If there’s no matches you know it’s an invalid username and/or password. If there is you can encrypt the provided username or password, again using the unique salt (possibly different from the password hash salt) and see if there’s a match.

(Note: if you compare ciphertext you must make sure it does not contain the random salt/IV. Otherwise you’ll need to decrypt the values and compare the plaintext.)

Can I store a hash of the username or email address instead?

Good idea! The answer is maybe. It depends on your data model. That brings us to the next point…

Any other ideas to improve my security?

Think microservices. Even if you have a monolithic application you probably have it broken apart into separate components internally – e.g., an e-commerce site probably has user management and authentication, inventory, shopping cart, fulfillment, payment, etc. There’s no reason these components need to share the same database – they can use separate schemas and database accounts.

You can take that a step further and use different physical databases for critical information. E.g., the user authentication information can (and arguably should) be stored on a different server than anything user-facing.

If you do this then you can store a hash of the username and email address in your authentication tables and database. Again this will only slow down a dedicated attacker (they’ll have likely usernames and email addresses from prior attacks) but it buys you, and your users, time.

Another good idea is to periodically reencrypt everything with a new key. An attacker might not be able to obtain a copy of the database and the encryption key at the same time so this gives an extra measure of security. You might be tempted to use multiple encryption keys, e.g., one identified by the month they registered, but I doubt the improved security is worth the increased complexity. IMHO it’s much better to use a single key that you rotate on a regular basis.

I’m small fry – who would target me? Why should I bother with this?

You might be “small fry” but many of your users will use the same password on other sites. Attackers know this and that “small fry” often have lax security out of ignorance or the misguided belief that they’ll never be the target of an attack.

The the first part of this introduction I discussed setting up the Kerberos infrastructure and creating users. In this part I will demonstrate how to use it to connect to a PostgreSQL server.

Note: GSSAPI is a generic security interface that can use different protocols on the backend. Kerberos is one of many protocols that can be used with GSSAPI. Using it allows developers to write flexible code while still allowing the system administrator to have the final word on which authentication method(s) will be accepted. When given a choice you should enable GSSAPI instead of just Kerberos.

Configuring the Server

The default servicename for PostgreSQL is ‘postgres’. There is one notable exception – working with Active Directory requires a servicename of ‘POSTGRES’ and that requires recompiling the server. The full principal for a PostgreSQL server is ‘postgres/fqdn@realm’ although in a some situations you’ll need to specify postgres@fqdn.

We start by creating a keytab file for the server:

$ kadmin
kadmin% ank -randkey postgres/db.invariantproperties.com
kadmin% ktadd -k krb5.keytab postgres/db.invariantproperties.com

Note that the value of the fully-qualified domain name (fqdn) is critical. The server and client must both recognize it as the name of the server – the server uses it to identify which keytab entry to use as it comes up and the client uses it when it constructs its query to the KDC. Ideally the name will be fully supported by DNS – including reverse lookups – but it’s often enough to put entries into the /etc/hosts files if you’re only working with a few systems.

Once you have your keytab file you should copy it to /etc/postgresql/9.4/main/krb5.keytab and change the file’s ownership. You should also make sure that the file can only be read by the server.

$ sudo chown postgres:postgres krb5.keytabe
$ sudo chmod 0600 krb5.keytab

We now need to tell the server the location of the keytab file and to listen to all network interfaces. Kerberos can only be used on TCP connections.

/etc/postgresql/9.4/main/postgresql.conf

...
krb_server_keyfile = '/etc/postgresql/9.4/main/krb5.keytab'
listen_addresses = '*'

We must also tell the server which databases can be accessed via Kerberos. Security note: in production we never want to use a wildcard (‘all’) for both database and user on a single line.

/etc/postgresql/9.4/main/pg_hba.conf

# TYPE  DATABASE        USER            ADDRESS                 METHOD       OPTIONS
host    all             all             52.34.69.195/32         gss          include_realm=1 map=gss krb_realm=INVARIANTPROPERTIES.COM

We will always want to retain the Kerberos realm so we don’t confuse ‘bob@FOO.COM’ with ‘bob@BAZ.COM’ but this requires us to use the pg_ident mapping file. The name of the mapping is arbitrary – I’m using ‘gss’ for convenience.

In this case we’re being even stricter and requiring that the Kerberos domain match ‘INVARIANTPROPERTIES.COM’ specifically. This isn’t always possible but you can repeat the line if you only need to support a few realms.

We must add entries to the server’s identity lookup file. The easiest approach is to add authorized users directly:

/etc/postgresql/9.4/main/pg_ident.conf

# MAPNAME    SYSTEM-USERNAME                    PG-USERNAME
gss          bgiles@INVARIANTPROPERTIES.COM     bgiles

This is not a good long-term solution since you must manually restart the database, or at least reload the configuration files, every time you need to add or remove an authorized user. It’s much better to qualify the username, e.g., with ‘/postgres’, and then use regular expression matching in the pg_ident file.

/etc/postgresql/9.4/main/pg_ident.conf

# MAPNAME    SYSTEM-USERNAME                                    PG-USERNAME
gss           /^([^/]+)\/postgres@INVARIANTPROPERTIES\.COM$     \1

Important: do not share Kerberos credentials. Either map multiple Kerberos users to the same PostgreSQL identity or map them to different PostgreSQL identities and use the standard grants and roles to control access within the database.

We’re now ready to restart the PostgreSQL server.

$ sudo service postgresql restart

Example

The following is an example dialogue as I attempt to log into the server.

# try to log in without any Kerberos tickets
bgiles@snowflake:~$ psql -h kpg
psql: GSSAPI continuation error: Unspecified GSS failure.  Minor code may provide more information
GSSAPI continuation error: No Kerberos credentials available

# log in as regular user, try to connect. The error is a little confusing but we do not get access.
bgiles@snowflake:~$ kinit bgiles
Password for bgiles@INVARIANTPROPERTIES.COM: 

bgiles@snowflake:~$ psql -h kpg
psql: duplicate GSS authentication request

# log in as database user, try to connect.
bgiles@snowflake:~$ kinit bgiles/postgres
Password for bgiles/postgres@INVARIANTPROPERTIES.COM: 

bgiles@snowflake:~$ psql -h kpg
psql (9.4.7, server 9.4.6)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

bgiles=# 

Establishing a JDBC Connection

Connecting to the database via the CLI demonstrates that we have properly set up Kerberos authentication but in practice we will want to access the database programmatically. That means JDBC connections.

Is it possible?

The answer is yes and no. I’ll cover it in further detail in the next part but it’s easy to establish the connection but there’s an unexplained problem when mapping the GSS/Kerberos identity to the PostgreSQL identity. Investigation continues…

PostgreSQL currently supports Kerberos authentication if you use a simple Kerberos principal (user) and password as your connection properties. It does not support a compound Kerberos principal as I discussed above, nor does it support the use of keytab files.

I plan to submit a patch to support keytabs in the next few weeks. I don’t know how long review will take and of course it’s unlikely to be applied retroactively. Write me if you want a copy.

What About Other Databases?

Other databases also support Kerberos.

Oracle: Oracle has supported Kerberos for a long time with an optional security extension. In 2013 Oracle made it free to use for all versions of its database. I haven’t had a chance to see if I can use it with Oracle 11 XE for Linux. It was cut at about the same time but it can be tricky to set up Oracle XE on Linux. (I had it working on Ubuntu 14.04 LTS but then a routine package update broke it.)

Microsoft SQLServer: SQLServer has also supported Kerberos for a long time, at least in its Active Directory wolf’s clothing. I would expect MSSQL XE to also support AD/Kerberos.

MySQL/MariaDB: MySQL does not support Kerberos. MariaDB does via a plugin but I haven’t been able to bring my MariaDB server up to verify this.

Cassandra: Cassandra supports Kerberos. See Datastax.

MongoDB: MongoDB supports Kerberos.

Neo4j: No information.

Apache Hive: Hive supports Kerberos.

H2/Derby/embedded databases: They do not appear to support Kerberos but I can’t say this with any certainty.

(Note: if you’re not familiar with Oracle XE and MSSQL XE they’re the ‘first hit is free’ versions of the respective databases. You can use them on your website or application as long as the server is limited to a single CPU and you do not have more than 10 GB (iirc) of data. They’re usually one rev back from the commercial product. A Personal Use license lets you use the latest versions of the software but there are extremely tight restrictions – it’s basically only useful when learning how to use the respective database.)

Next Time

Next time I will discuss establishing low-level connections using Kerberos authentication.

Old protocols never die.

Kerberos is a three party protocol that provides strong mutual authentication between clients and servers. It is designed for academic and enterprise organizations where there is a single source of truth regarding identify, authentication and authorization. Think universities with faculty, students, and staff, or businesses with employees with different responsibilities. Changes must be propagated immediately – it is not acceptable for there to be a lag such as we see with X.509 digital certificates using periodically updated CRL lists. (It is possible, but expensive, to use OCSP to verify a certificate on each use.)

Strong mutual authentication is when both parties to a transaction have high confidence in the identity of the other party. A good example is when you log onto your bank’s website. The bank authenticates the user with username, password, and second factor like the user accepting the “personal image”. The user authenticates the bank by checking the HTTPS icon in the corner and the recognized “personal image”. Many of us consider the latter somewhat broken since the system will accept a certificate from ANY certificate authority preloaded into the browser. I have nothing against the Moldovian postal service but I don’t want to have to trust it in order to check my bank balance. I have nothing against it but I have no reason to trust it either. With Kerberos there’s a personal relationship – it is our employer, our university, etc., so there’s a much higher level of trust.

Kerberos came out of the Athena project at MIT in the 1980s and was “embraced and extended” by the Borg, I mean by Microsoft, in Windows 2000 as the basis of Active Directory. The “client” was traditionally a person but it is increasingly software as well.

There are three open source Kerberos implementations widely available to Linux users: MIT, Heimdal, and Shishi. MIT Kerberos 5 is the reference implementation and somewhat more tuned to enterprise users, Heimdal is an implementation made outside of the US, and I know nothing about Shishi. The fact that Heimdal was developed outside of the US was important while the US banned export under ITAR regulations (since it includes strong encryption) and may become important again and we refight the crypto wars.

I will be discussing MIT Kerberos since I’m a little more familiar with it.

The Three-Headed Hellhound

Kerberos is named after the three-headed dog that guards the gate of Hell. This reflects the three parties in a Kerberos session:

The Client

The client may be a person (student, employee) or enterprise software. The client does not log into a remote system. Instead the client logs into a local agent and that agent performs all necessary negotiation with the KDC and servers. In the protocol this is the “ticket granting ticket (TGT)”.

As mentioned earlier the client may also be enterprise software.

The Server

The server can be any software requiring authentication. Traditional servers include telnet (ktelnet) allowing remote access to a shell and NFS (allowing remote access to a filesystem). In the enterprise environment we often see kerberized web services.

The Key Distribution Center (KDC)

The key distribution center (KDC) is the trusted third party. In early implementations the KDC was literally responsible for generating symmetric keys to be used by the client and server, hence the name. I don’t know if that’s still true. The client and server are free to renegotiate their session keys at any time.

All of this occurs within a Kerberos realm. This is an arbitrary string but by convention is the domain name in all caps, e.g., INVARIANTPROPERTIES.COM. This ensures that realms will be unique (modulo bad players). Users within a realm are identified by principals. An individual may have more than one principal but no principal should be used by more than one individual. Principals have the format username@REALM or username/role@REALM, e.g., bgiles/admin@INVARIANTPROPERTIES.COM. Servers have a similar format: servicename/fqdn@REALM.

KDC installation on Debian and Ubuntu

Today we will discuss the installation and configuration of an MIT Kerberos5 KDC on an AWS EC2 instance. MIT can use either an internal database or LDAP for its backing store. The latter will be a better choice if you already use an LDAP database for user management.

I will only discuss the former in this blog entry but might revisit the latter in the future.

Hardware Requirements

The primary hardware requirement for a KDC is security. A compromised KDC will put the entire enterprise at risk. It should be physically secured and run no other services. In production we want at least two servers (one primary and at least one secondary) and they should be in physically separate locations.

It is extremely important that the clocks are synchronized across the all systems within a realm. Historically tickets would be valid for 5 to 15 minutes but with NTP running on all systems this window can be reduced to seconds. Note: this is the window when a ticket can be used for authentication and has no bearing on how long the user can use the service.

A KDC has modest computational requirements. I believe we can start with a micro instance and only upgrade if the need arises. This means the financial cost is extremely modest even if we use multiple KDCs.

We need to open three ports in our firewall. Ports 88 and 750 are required by Kerberos, port 22 for SSH is only required for initial installation and can can be blocked at the AWS Security Group level unless explicitly needed. Subsequent access can eat its own dog food with ktelnet (kerberized telnet), krsh (kerberized rsh), or SSH with Kerberos extensions.

The KDC admin server must also open port 749. Access to this port should be limited.

Debian/Ubuntu Packages

We need to install two packages: krb5-kdc and krb5-admin-server. If you are using LDAP you will want to install krb5-ldap as well. Finally we want to install krb5-doc for documentation.

$ sudo apt-get install krb5-kdc krb5-admin-server krb5-doc

During installation we will be asked three things:

• Name of our realm – this is traditionally our domain name in all-caps.
• KDC servers – this system
• KDC admin servers – this system

If you wish you can leave the servers blank at this time and fix the /etc/krb5.conf file below.

Database Initialization

We initialize the internal database with a call to krb5_util. We need to provide our realm name.

$ sudo /usr/sbin/kdb5_util create -r INVARIANTPROPERTIES.COM -s

This will take a minute or so to complete.

Defining user rights

We define our users’ rights via an ACL file – /etc/krb5kdc/kadm5.acl. See the man pages for details. One thing to keep in mind is that the user will remain logged in for 7 days by default. Roles with more authority should require reauthentication more frequently.

/etc/krb5dkc/kadm5.acl

*/admin@INVARIANTPROPERTIES.COM x  * -maxlife 4h
*/root@INVARIANTPROPERTIES.COM  ci *1@INVARIANTPROPERTIE.COM
*/root@INVARIANTPROPERTIES.COM  l  *

See the man page for kadm5.acl for a description of these entries. The gist is that any principal with ‘admin’ has full administrative rights but must reauthenticate every 4 hours. The superuser on any system can list and update user credentials.

Creating our first administrative user(s)

We must bootstrap our administrative users on the KDC itself. Subsequent user management can be handled remotely. Note that we do NOT need the master password for this – this is why it is critical that access to the KDC be limited.

This shows how to add a user with both regular and administrative rights.

$ sudo kadmin.local
kadmin.local: add_principal bgiles
Enter password for principal "bgiles@INVARIANTPROPERTIES.COM": *******
Re-enter password for principal "bgiles@INVARIANTPROPERTIES.COM": *******
kadmin.local: add_principal bgiles/admin
Enter password for principal "bgiles/admin@INVARIANTPROPERTIES.COM": *******
Re-nter password for principal "bgiles/admin@INVARIANTPROPERTIES.COM": *******
kadmin.local: quit.

Registering our servers

If you did not specify the KDC server and KDC admin servers when we installed the packages you can fix this now. Edit the appropriate entry in the /etc/krb5.conf file:

,,,
[realms]
        INVARIANTPROPERTIES.COM = {
                kdc = kdc1.invariantproperties.com:88
                kdc = kdc2.invariantproperties.com:88
                admin_server = kdc1.invariantproperties.com
                default_domain = invariantproperties.com
        }
...
[domain_realm]
        .invariantproperties.com = INVARIANTPROPERTIES.COM
        invariantproperties.com = INVARIANTPROPERTIES.COM
...

This snippet demonstrates how to specify multiple KDC servers. I have not discussed how to perform database replication among multiple KDC servers. This is one task where an LDAP backing store would make much more convenient.

Preparing the Client

We are now ready to set up the client systems so the user can log into Kerberos (that is, get a ticket-granting-ticket (TGT)). I am not going to discuss setting up and accessing kerberized services at this time.

Debian/Ubuntu Packages

We need to install one package: krb5-user. We may also want to install krb5-doc for the rare user who reads the documentation before calling the help desk.

$ sudo apt-get install krb5-user krb5-doc

You will be asked to specify the Kerberos realm, KDC and KDC admin servers.

Logging in and out

The user can now log into the system using kinit. This command has many options – see the man page for details.

$ kinit
Password for bgiles@INVARIANTPROPERTIES.COM: ******
$ kinit bgiles/admin
Password for bgiles/admin@INVARIANTPROPERTIES.COM ******

We can determine our current credentials with klist

$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: bgiles/admin@INVARIANTPROPERTIES.COM

Valid starting       Expires              Service principal
04/17/2016 09:40:51  04/17/2016 19:40:51  krbtgt/INVARIANTPROPERTIES.COM@INVARIANTPROPERTIES.COM
	renew until 04/18/2016 09:40:46

(note that the credentials expire at 10 hours, not 4 hours. I need to look into that…)

Finally we can log out with kdestroy

$ kdestroy
$ klist
klist: Credentials cache file '/tmp/krb5cc_1000' not found

You can change your password with kpasswd and change your active principal with kswitch.

Remote administration

Administrators can run kadmin remotely. It is no longer necessary for them to physically log into the KDC and run kadmin.local. They will be prompted to reenter their credentials since this is a sensitive operation.

Keytab files

Finally it can be inconvenient to re-enter your password every time you access a remote service. Keytab files allow you to store a Kerberos principal and encryption key for us in place of a password. You must regenerate the keytab files every time you change your Kerberos password.

Keytabs are maintained with the ktutil command.

For more information see What is a keytab, and how do I use one? from Indiana University.

Next Time

Next time I will discuss setting up kerberized services.

Finally A Cautionary Tale

I almost forgot something important – a server should never ask for a user’s Kerberos credentials. The user always performs local authentication and everything else is negotiated behind the scenes. I mention this because Apache had (and still has?) a Kerberos authentication module that worked by prompting the user for his Kerberos credentials and then logging into the system as that that user on the server in order to obatain resources from other systems. This is entirely contrary to the design goals of Kerberos and it is a huge red flag when it happens – it is possible to get “transferable” tickets that can be used by a server on your behalf.

Something that quickly stands out when you review the various security certifications is that nearly all require documented work experience. Documented primary work experience – as in your job description – and not just experience as part of a non-infosec job. (As a counterexample you might be a devop implementing a PCI-DSS-compliant system. You have to know a lot about security but your job description will be developer/architect/etc. and it will not count towards these certifications.) CompTIA is a notable exception and it catches flak for it from some people.

Does this make sense?

I think it does in the NOC. I keep coming back to an analogy to EMTs – you want someone who’s well-trained and experienced who can respond quickly and accurately on the front lines. A certificate that requires experience can greatly simplify the life of the hiring manager.

However I think that it does NOT make sense outside of the NOC. Experience is rarely a bad thing but if you’re designing and building software you don’t just need to know what the latest attacks are – you need to be able to integrate security fundamentals with development experience in order to anticipate where today’s bright idea could lead to attacks and how to avoid them. It’s not a one-legged stool where you can get by with just infosec experience or development experience. You need to have a balance.

Why would a developer want to go to the cost and effort to get a formal certification?

The job requires it

Some jobs, esp. in the defense industry, require formal certification for developer positions (IAT level II or III under DoD 8570.01-M). This is a complex situation since these jobs often require a security clearance as well and in a strong job market many developers may decide it isn’t worth the hassles.

This is not a theoretical concern to me since there’s a good match in town but it requires a TS/SCI clearance. Is it worth the hassles since there’s at least a half-dozen other positions within a few miles from home that won’t ask (and leak!) highly sensitive information in another OPM leak? Once is enough.

The job requires ongoing training

Many jobs require ongoing security training. That’s a few days to a week dedicated to learning the latest developments (e.g., drop any remaining use of SHA-1 immediately) with background training throughout the rest of the year by senior developers. You don’t have to have formal certification to perform the latter role but it makes life easier for everyone involved since it establishes a clear floor for the annual updates.

The agile methodology requires an advocate for security

The agile methodology is widely used since work is prioritized by the “product owner” on the basis of business needs. This is obviously a Good Thing since it provides the best value to the business but product owners are not security experts and might not understand the importance of security user stories. A team advocate for security can help educate the project manager and product owner. The best way to ensure the product owner will listen is to have a disinterested third party attest to his or her skills.

It’s cheaper to design it right than patch

This is true of all design methodologies – it is always cheaper to make a change early. A design with an eye towards security – esp. noticing where seemingly small changes will dramatically strengthen or weaken security – will always be cheaper to implement than a security-blind design that has to be fixed later. Once again the knowledge does not require certification but convincing non-technical people of the importance of the changes might.

It makes you a better developer and tester

Finally security training makes you a better developer and tester because it gives you a better awareness of how things can break – if we’re in a hurry we’ll often write test cases on the basis of the happy path and something that breaks every conditional branch and overlook the fact that we’re not checking everything it should. This is why test coverage is a risky metric. Security awareness means we can take a step back and ask how we would attack a piece of code if we were an attacker and that’s probably going to expose unwarranted assumptions.

The bottom line

The bottom line is that developers will care about training and experience, not pieces of paper, and the fastest way to flip the bozo bit is to come into the room demanding that people should pay attention to you just because you have of a slip of paper. The business side has different rules. They must rely on internal and external evaluations and formal certification by respected organizations is an important tool to get your message out.

If your job requires it, get a cert. If your boss’s boss cares about it, get a cert. If you need to get past the Guardians of HR with their dreaded keyword search, get a cert. Otherwise read a study guide and/or watch training videos so you keep current but only spend the money if you, yourself, would like it for your own reasons. For instance I like taking the actual exam since it forces me to study everything covered and not just the bits I find interesting or easy.

Scratch one more item off my long-term to-do list. I’ve wanted to turn on HTTPS for a long time but I didn’t want to mess up anything else on my virtual machine. That’s no longer an issue so the site is now HTTPS-only.

In case anyone else is interested I got my certificate at Namecheap. It’s a straightforward process – buy the certificate, activate it by providing basic information and a CSR, and prove that you control the domain by adding a CNAME entry. No money is involved so I picked a low-cost item.

Many critics have pointed out that Ashley Madison should have encrypted all personally identifiable information (PII). The database contained sensitive information that would cause harm to users if it was released.

We are probably not involved in dating websites based on infidelity, at least not as a developer. But the nature of our business doesn’t matter after a breach – if there’s financial information involved then there’s the need to notify users, offer credit monitoring, face the exposure to lawsuits, increased costs for PCI-DSS compliance inspections, etc. You need to consult a lawyer but I think many of these costs can be mitigated if all PII is encrypted.

Encryption may also be required by law or industry standards. I have worked at several sites, including a federal contractor, where all UII must be encrypted. At least one site required all PII to be encrypted as well. It’s not the norm, yet, but it’s not unheard of either.

Bottom line: there’s a very real impact on the bottom line if information is leaked, this isn’t entirely in your business’s control (e.g., the Target breach was due to a subcontractor being sloppy with its access credentials), and relatively inexpensive countermeasures can reduce this exposure. It will be very uncomfortable if something goes wrong and a former user’s attorney asked you why you didn’t take all reasonable steps to protect her client.

What is personally identifiable information (PII)?

There are two closely related concepts, personally identifiable information (PII) and uniquely identifiable information (UII). Personally identifiable information (PII) is enough to give someone a good idea who the individual is but not with 100% certainty. Uniquely identifiable information (UII) gives you nearly 100% certainty.

It’s important to note that a UII is like being pregnant. You can’t be a little pregnant – it’s all or nothing. Your UII might be nothing but an burner email address. It doesn’t matter if the user used the same burner email address at a different site where additional information is available, or perhaps the user self-disclosed that information in the site’s content. Once the person has been identified on the second site the attacker can come back to your site and use that burner email address to tie information back to the individual. You can’t prevent that so you should assume the UII is already compromised elsewhere.

Some things are intrinsically UII:

• social security number
• email address
• license number (e.g., driver’s license, professional license)
• account number

Other things are considered safe in isolation but with two or more pieces of information they become PII and then UII.

• first name
• last name
• address (risky)
• zip code
• phone number area code
• full phone number (risky)
• birth year (or age)
• date of birth
• IP address (with timestamp)
• name of employer
• car make and model
• and so on

A good conceptual model for the difference between PII and UII as that you can fit all of the PII matches onto a piece of paper the size of a postcard. A UII allows you to actually address a postcard. Another model could be that PII allows you to approach individuals and ask “why should I believe you are this person?” and expect them to make a solid case. UII allows you to approach an individual and ask “why should I believe you are not this person?” and expect them to be unable to make a solid case.

Obviously for some people (e.g., “John Smith”) you will need many pieces of information. For other people a last name and zip code may be sufficient, or full date of birth and zip code.

The usual warnings about false matches still apply. Your “unique match” could still be two men with the same full name, the same address, and same birth date (month and day) if they are father and son who coincidentally share the same birthday. The son could be a minor, a college student using his parent’s address to receive mail, or a boomerang kid.

A more unusual case happened in Australia a few years back. It seemed to be a clear case of fraud – a woman was getting student aid at two schools while working in a third city. It was actually three unrelated women who, by pure chance, had the same full name and date of birth.

These are still considered UII matches since this happens so rarely – literally less than one-in-a-million occurances.

Sidenote: given someone’s full birthdate and the city where they were born you can often make a good guess at their social security if they’re under 35 or so. The IRS now requires a SSN for anyone claimed as a dependent so most people get them for their children immediately after birth. SSNs aren’t issued sequentially at the national level but they do (or did) follow a predictable pattern if you know the state where they are issued and the person’s last name. The average person won’t know a good starting SSN for a search but an attacker might. This should drive home the fact that seemingly safe information can be combined to reveal much more sensitive information.

Objection: I use the information constantly

A common objection to encrypting PII information is that it’s constantly used so it’s an unreasonable burden to be repeatedly decrypting it.

This is an objection that rarely stands up to careful examination. Take Amazon as a theoretical example. It’s heavily used by many tens or even hundreds of millions of users. Surely PII encryption would be too costly.

Except… when do they actually need my PII? They use my first name for personalization but they only use my full PII in two places:

1. When I pay for an order (billing address) and

2. When I ship an order (mailing address).

And that’s it. I’m sure it comes up elsewhere but the vast bulk of usage is related to either billing or mailing. Everything else can use an internal ID that reveals nothing about me.

Objection: my application still needs unencrypted PII data

That might have been true in the ’00s. But as businesses “move into the cloud” they’ll often switch to a microservices architecture where monolithic applications are broken down into tiny pieces that do just one thing. Even traditionally hosted applications may break down a monolithic application into separate components in order to spread the work across multiple servers.

What are two obvious candidates for a microservice? Billing (finances) and shipping (fulfillment).

Those two services need to have access to the unencrypted PII but they don’t have to share it with the overall application. They don’t even need to share their database with the overall application – this is a good place to put the service on a dedicated system behind your firewall.

Objection: my application identifies users with their email address and you said that is UII

This is valid concern but easily handled. Do not use the email address for authentication – use the hash of the email address. If you’re worried about hash collisions you can use the hash (once verified) to pull up the PII and verify it.

Objection: my application needs to be able to search for users

Again we can handle this with careful hashing. Store the hash of the search criteria, e.g., last name, city, state, etc. You can now hash the search criteria, perform the search using the hashed values, and then decrypt the PII for final comparisons.

We can make this more flexible by using the soundex index of our search criteria. This will create larger buckets but we’ll be more likely to catch ‘near misses’.

Objection: won’t all of these hashes be subject to a salami attack?

Short answer: yes. You might not know whether the most frequent hash on last name is “Smith” or “Johnson” but you know it’s more likely to be that than “Roberts”. Likewise the most frequent hashed states are more likely to be California or Texas than Wyoming or Idaho.

There’s an easy workaround – salted hashes. We can start with a small number of salts that are partitioned by the data. (E.g., names from A-F use salt 12). Statistics are periodically collected and if a single salt is notably more frequently used than others then either the bin can be split (names A-C use salt 20, D-F use salt 21) or additional salts can be used on the existing bin. There isn’t a large performance hit as long as the number of possible salts is reasonable.

Objection: I don’t know how to properly use encryption

Or worse, you think you know how to properly use encryption but are wrong.

This is a valid concern but beyond the scope of this discussion.

We often need to store structured binary data in our database – images, pdf documents, etc., but also have a need to search by, or index on, attributes of that data. E.g., we might store the height and width of an image, or the OCR text from a PDF document (for full text searches).

The normal solution is to store the object as a BLOB, programmatically extract the required information, and store it as additional columns. In nearly all cases that’s easy to do and good enough to solve our problem.

Release the kraken!

That’s “nearly” all cases. There are times when we need to exercise more care. Times when kraken, I mean lawyers, might be involved. Times when you might be asked if you took all reasonable steps to ensure that the data was not illicitly modified without you realizing it.

In these cases it’s not enough to just cache a value. We need to enlist the database to help us.

Threat model

There are two threats. First, the contents of the BLOB could be modified while all extracted values are left unchanged. This would mean that extra matches are found during searches. On the bright side we can verify the results actually matched and be alerted there’s a problem.

The second threat is more subtle. Extra results for some queries will be matched by fewer results in other queries. This means we can make records “disappear” even if there are measures in place to detect or prevent the deletion of records.

As a concrete example the first threat could be that an arrest record summary is changed from a felony to a misdemeanor. The second threat is that the search for prior arrests comes up empty.

X.509 digital certificates

For the rest of this article I’ll use X.509 digital certificates as a specific example. There are three reasons for this. First, they’re non-trivial and require a C-language extension to implement the user-defined functions and types – these are the types of things that are usually handled entirely at the application level. Second, they’re a real world example of the need to search and index values by cached values since it’s too expensive to recalculate them for each use. Finally, and not least, I’m familiar with them.

This article uses my PostgreSQL cert extension. This is a highly unstable extension (hence the 0.1.0 version as I write this) but it contains the minimum functions to store a digital certificate as a ‘cert’ type and to extract useful information from it. The details are not important – we just need an example of extracting “interesting” values from a BLOB. If this is distracting just replace any “cert” with a “blob” in the examples below.

Hardening the database

The database almost certainly does not have the functions we need to extract information from our BLOB. If it did it would probably be a first-class type, not a BLOB. This means we have to define our own functions. We might even define our own user-defined type (UDT) so we have some measure of type safety since we can put object validation into the methods that convert the object between internal and external formats.

Constraints

We can start by adding constraints on our table that ensure the cached values match the computed values.

--
-- Create a table containing digital certificates and cached values.
--
create table certs (
    cert cert,
    serial_number bignum,
    not_before timestamp,
    not_after timestamp,
    issuer text,
    subject text,

    -- define constraints
    CONSTRAINT cert_serial CHECK (serial_number = get_serial_number(cert)),
    CONSTRAINT cert_not_before CHECK (not_before = get_not_before(cert)),
    CONSTRAINT cert_not_after CHECK (not_after = get_not_after(cert)),
    CONSTRAINT cert_issuer CHECK (issuer = get_issuer(cert)),
    CONSTRAINT cert_subject CHECK (subject = get_subject(cert))
);

The constraints make it impossible to insert or update invalid cached values. This gives us false confidence though – if an attacker has sufficient privileges it is possible to drop constraints, insert bad data, and then restore the constraints. This works since existing rows are not checked when constraints are added.

A good countermeasure to this is to periodically validate the constraints.

ALTER TABLE certs VALIDATE CONSTRAINT check_serial;
ALTER TABLE certs VALIDATE CONSTRAINT check_not_before;
ALTER TABLE certs VALIDATE CONSTRAINT check_not_after;
ALTER TABLE certs VALIDATE CONSTRAINT check_subject;
ALTER TABLE certs VALIDATE CONSTRAINT check_issuer;

Triggers

Another drawback to using constraints is that you must determine the correct values to insert. This either requires maintaining code in two places – the database and the application or writing much more complex INSERT and UPDATE statements. Fortunately it is easy write a trigger that sets the fields to the correct value upon any INSERT or UPDATE.

--
-- Create a stored procedure that sets several columns by calling the
-- appropriate function instead of accepting the value provided to the
-- INSERT or UPDATE call.
-- 
CREATE FUNCTION cert_trigger_proc() RETURNS trigger AS $$
BEGIN
    NEW.serial_number = get_serial_number(NEW.cert);
    NEW.not_before = get_not_before(NEW.cert);
    NEW.not_after = get_not_after(NEW.cert);
    NEW.issuer = get_issuer(NEW.cert);
    NEW.subject = get_subject(NEW.cert);

    RETURN NEW;
END
$$ LANGUAGE plpgsql;

--
-- Define a trigger on the 'certs' table that will be called whenever
-- a row is inserted or updated.
--
CREATE TRIGGER cert_trigger BEFORE INSERT OR UPDATE ON certs
    FOR EACH ROW EXECUTE PROCEDURE cert_trigger_proc();

Example

Here is an example of the results of inserting two values with a trigger in place.

insert into certs values (
cert('-----BEGIN CERTIFICATE-----
MIIDEzCCAnygAwIBAgIBATANBgkqhkiG9w0BAQQFADCBxDELMAkGA1UEBhMCWkEx
FTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYD
VQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlv
biBTZXJ2aWNlcyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEm
MCQGCSqGSIb3DQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wHhcNOTYwODAx
MDAwMDAwWhcNMjAxMjMxMjM1OTU5WjCBxDELMAkGA1UEBhMCWkExFTATBgNVBAgT
DFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYDVQQKExRUaGF3
dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNl
cyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEmMCQGCSqGSIb3
DQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQAD
gY0AMIGJAoGBANOkUG7I/1Zr5s9dtuoMaHVHoqrC2oQl/Kj0R1HahbUgdJSGHg91
yekIYfUGbTBuFRkC6VLAYttNmZ7iagxEOM3+vuNkCXDF/rFrKbYvScg71CcEJRCX
L+eQbcAoQpnXTEPew/UhbVSfXcNY4cDk2VuwuNy0e982OsK1ZiIS1ocNAgMBAAGj
EzARMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEEBQADgYEAB/pMaVz7lcxG
7oWDTSEwjsrZqG9JGubaUeNgcGyEYRGhGshIPllDfU+VPaGLtwtimHp1it2ITk6e
QNuozDJ0uW8NxuOzRAvZim+aKZuZGCg70eNAKJpaPNW15yAbi8qkq43pUdniTCxZ
qdq5snUb9kLy78fyGPmJvKP/iiMucEc=
-----END CERTIFICATE-----')),
(cert('-----BEGIN CERTIFICATE-----
MIICPDCCAaUCED9pHoGc8JpK83P/uUii5N0wDQYJKoZIhvcNAQEFBQAwXzELMAkG
A1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFz
cyAxIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2
MDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNV
BAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAxIFB1YmxpYyBQcmlt
YXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDlGb9to1ZhLZlIcfZn3rmN67eehoAKkQ76OCWvRoiC5XOooJskXQ0f
zGVuDLDQVoQYh5oGmxChc9+0WDlrbsH2FdWoqD+qEgaNMax/sDTXjzRniAnNFBHi
TkVWaR94AoDa3EeRKbs2yWNcxeDXLYd7obcysHswuiovMaruo2fa2wIDAQABMA0G
CSqGSIb3DQEBBQUAA4GBAFgVKTk8d6PaXCUDfGD67gmZPCcQcMgMCeazh88K4hiW
NWLMv5sneYlfycQJ9M61Hd8qveXbhpxoJeUwfLaJFf5n0a3hUKw8fGJLj7qE1xIV
Gx/KXQ/BUpQqEZnae88MNhPVNdwQGVnqlMEAv3WP2fr9dgTbYruQagPZRjXZ+Hxb
-----END CERTIFICATE-----'));
select * from certs;
                               cert                               |             serial_number              |        not_before        |        not_after         |         issuer         |        subject         
------------------------------------------------------------------+----------------------------------------+--------------------------+--------------------------+------------------------+------------------------
 -----BEGIN CERTIFICATE-----                                     +| 1                                      | Thu Aug 01 00:00:00 1996 | Thu Dec 31 23:59:59 2020 | C=ZA/ST=Western Cape/L | C=ZA/ST=Western Cape/L
 MIIDEzCCAnygAwIBAgIBATANBgkqhkiG9w0BAQQFADCBxDELMAkGA1UEBhMCWkEx+|                                        |                          |                          |                        | 
 FTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYD+|                                        |                          |                          |                        | 
 VQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlv+|                                        |                          |                          |                        | 
 biBTZXJ2aWNlcyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEm+|                                        |                          |                          |                        | 
 MCQGCSqGSIb3DQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wHhcNOTYwODAx+|                                        |                          |                          |                        | 
 MDAwMDAwWhcNMjAxMjMxMjM1OTU5WjCBxDELMAkGA1UEBhMCWkExFTATBgNVBAgT+|                                        |                          |                          |                        | 
 DFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYDVQQKExRUaGF3+|                                        |                          |                          |                        | 
 dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNl+|                                        |                          |                          |                        | 
 cyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEmMCQGCSqGSIb3+|                                        |                          |                          |                        | 
 DQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQAD+|                                        |                          |                          |                        | 
 gY0AMIGJAoGBANOkUG7I/1Zr5s9dtuoMaHVHoqrC2oQl/Kj0R1HahbUgdJSGHg91+|                                        |                          |                          |                        | 
 yekIYfUGbTBuFRkC6VLAYttNmZ7iagxEOM3+vuNkCXDF/rFrKbYvScg71CcEJRCX+|                                        |                          |                          |                        | 
 L+eQbcAoQpnXTEPew/UhbVSfXcNY4cDk2VuwuNy0e982OsK1ZiIS1ocNAgMBAAGj+|                                        |                          |                          |                        | 
 EzARMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEEBQADgYEAB/pMaVz7lcxG+|                                        |                          |                          |                        | 
 7oWDTSEwjsrZqG9JGubaUeNgcGyEYRGhGshIPllDfU+VPaGLtwtimHp1it2ITk6e+|                                        |                          |                          |                        | 
 QNuozDJ0uW8NxuOzRAvZim+aKZuZGCg70eNAKJpaPNW15yAbi8qkq43pUdniTCxZ+|                                        |                          |                          |                        | 
 qdq5snUb9kLy78fyGPmJvKP/iiMucEc=                                +|                                        |                          |                          |                        | 
 -----END CERTIFICATE-----                                       +|                                        |                          |                          |                        | 
                                                                  |                                        |                          |                          |                        | 
 -----BEGIN CERTIFICATE-----                                     +| 84287173645887463140025226144593929437 | Mon Jan 29 00:00:00 1996 | Wed Aug 02 23:59:59 2028 | C=US/O=VeriSign, Inc./ | C=US/O=VeriSign, Inc./
 MIICPDCCAaUCED9pHoGc8JpK83P/uUii5N0wDQYJKoZIhvcNAQEFBQAwXzELMAkG+|                                        |                          |                          |                        | 
 A1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFz+|                                        |                          |                          |                        | 
 cyAxIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2+|                                        |                          |                          |                        | 
 MDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNV+|                                        |                          |                          |                        | 
 BAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAxIFB1YmxpYyBQcmlt+|                                        |                          |                          |                        | 
 YXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUAA4GN+|                                        |                          |                          |                        | 
 ADCBiQKBgQDlGb9to1ZhLZlIcfZn3rmN67eehoAKkQ76OCWvRoiC5XOooJskXQ0f+|                                        |                          |                          |                        | 
 zGVuDLDQVoQYh5oGmxChc9+0WDlrbsH2FdWoqD+qEgaNMax/sDTXjzRniAnNFBHi+|                                        |                          |                          |                        | 
 TkVWaR94AoDa3EeRKbs2yWNcxeDXLYd7obcysHswuiovMaruo2fa2wIDAQABMA0G+|                                        |                          |                          |                        | 
 CSqGSIb3DQEBBQUAA4GBAFgVKTk8d6PaXCUDfGD67gmZPCcQcMgMCeazh88K4hiW+|                                        |                          |                          |                        | 
 NWLMv5sneYlfycQJ9M61Hd8qveXbhpxoJeUwfLaJFf5n0a3hUKw8fGJLj7qE1xIV+|                                        |                          |                          |                        | 
 Gx/KXQ/BUpQqEZnae88MNhPVNdwQGVnqlMEAv3WP2fr9dgTbYruQagPZRjXZ+Hxb+|                                        |                          |                          |                        | 
 -----END CERTIFICATE-----                                       +|                                        |                          |                          |                        | 
                                                                  |                                        |                          |                          |                        | 
(2 rows)

The truncated ‘issuer’ and ‘subject’ fields are a known bug. (Version 0.1.0, remember?) It highlights a key point though – this is not a magic bullet and a buggy function may still let things through. Modifying the functions used to create cached values means you’ll need to drop the constraints, reinitialize all cached values, and then restore the constraints.

Database security

Finally I need to come back to the point of someone dropping a constraint and then reinstating it. None of this works without solid database security. A handful of rules will go a long way.

A dedicated database user should own the schema. This user should not be used for any other purpose – it should only be used when creating and modifying the schema.

No other user should have ALTER privileges. A dedicated user isn’t enough if other users can alter the schema anyway. The owner should be the only user with ALTER privileges by default but it would not be a bad idea to double-check.

Periodically audit the schema. This doesn’t have to be very sophisticated and can be done programmatically. Periodically query the metadata for the database and ensure tables and functions are owned by the correct users, that they have the correct privileges, that there are no unexpected tables in the schema, that there are no unexpected columns in the tables.

I’ve previously mentioned providing an encryption key via a JNDI value provided by an appserver, or better yet splitting the key between that JNDI value and a file outside of the webapp directory. A recent question at work reminded me that this isn’t the only solution. Another approach is a simple key server – a webservice that provides the encryption key on demand.

This sounds wildly insecure but it’s fairly easy to lock down since the use case is so limited:

1. Run it on a private network (in a data center you own) or a VPC (in AWS). This restricts access to systems in the same data center since there should be a physical router blocking access from the internet at large.

2. Consider using IP address white-listing. You know the IP addresses of your servers, or at least their IP address range. Configure the webservice to only accept connections from these IP addresses. This can’t hurt but a knowledgeable attacker can perform ARP poisoning to hijack a white-listed address.

3. Use mutual authentication with digital certificates. You can create these certificates in-house. This forces an attacker to know something non-trivial in addition to being physically located near the key server.

4. Use SSL to prevent eavesdropping.

5. Use a keystore with a password provided by JNDI if you are returning private keys. It’s harder to protect secret keys but you could use a JNDI-provided key to wrap them. This would limit exposure if the appserver value is leaked due to poor backup security or compromised sysadmins.

I don’t think we gain anything by requiring a nonce since the code to handle it will also require secure initialization.

The biggest benefits to a key server are scalability and the ability to provide more than one key. It also prevents key disclosure if an attacker is able to look at arbitrary files on your system or, worse, obtain a backup copy of the system. The biggest drawback is that it’s one more system to maintain.

In production an AWS ‘micro’ instance should be more than adequate to support many client servers. The cost of a ‘micro’ instance is negligible. In a data center you could put it on a minimal virtual server.

For development and testing you can use a keyserver under the same appserver as your application.

I hope it goes without saying that this is not adequate for systems that require the highest security. However if you’re looking for something beyond a key provided by JNDI then this should be one of the options on the table.

A crazy idea came up during the post-mortem discussions in the Coursera security capstone project. Can a class encrypt itself during serialization?

This is mostly an academic “what if” exercise. It is hard to think of a situation where we would want to rely on an object self-encrypting instead of using an explicit encryption mechanism during persistence. I’ve only been able to identify one situation where we can’t simply make a class impossible to serialize:

HTTPSession passivation

Appservers may passivate inactive HTTPSessions to save space or to migrate a session from one server to another. This is why sessions should only contain Serializable objects. (This restriction is often ignored in small-scale applications that can fit on a single server but that can cause problems if the implementation needs to be scaled up or out.)

One approach (and the preferred approach?) is for the session to write itself to a database during passivation and reload itself during activation. The only information actually retained is what’s require to reload the data, typically just the user id. This adds a bit of complexity to the HTTPSession implementation but it has many benefits. One major benefit is that it trivial to ensure sensitive information is encrypted.

It’s not the only approach and some sites may prefer to use standard serialization. Some appservers may keep copies of serialized copies of “live” sessions in an embedded database like H2. A cautious developer may want to ensure that sensitive information is encrypted during serialization even if it should never happen.

Note: a strong argument can be made that the sensitive information shouldn’t be in the session in the first place – only retrieve it when necessary and safely discard it once it is no longer needed.

The approach

The approach I’m taking is based on the serialization chapter in Effective Java. In broad terms we want to use a serialization proxy to handle the actual encryption. The behavior is

The reason the protected class throws an exception when the deserialization methods are called is because it prevents attacks through attacker-generated serialized objects. See the discussion on the bogus byte-stream attack and internal field theft attack in the book mentioned above.

This approach has a big limitation – the class cannot be extended without the subclass reimplementing the proxy. I don’t think this is an issue in practice since this technique will only be used to protect classes containing sensitive information and it would rarely be desirable to add methods beyond the ones anticipated by the designers.

The proxy class handles encryption. The implementation below shows the use of a random salt (IV) and cryptographically strong message digest (HMAC) to detect tampering.

The code

public class ProtectedSecret implements Serializable {
    private static final long serialVersionUID = 1L;

    private final String secret;

    /**
     * Constructor.
     * 
     * @param secret
     */
    public ProtectedSecret(final String secret) {
        this.secret = secret;
    }

    /**
     * Accessor
     */
    public String getSecret() {
        return secret;
    }

    /**
     * Replace the object being serialized with a proxy.
     * 
     * @return
     */
    private Object writeReplace() {
        return new SimpleProtectedSecretProxy(this);
    }

    /**
     * Serialize object. We throw an exception since this method should never be
     * called - the standard serialization engine will serialize the proxy
     * returned by writeReplace(). Anyone calling this method directly is
     * probably up to no good.
     * 
     * @param stream
     * @return
     * @throws InvalidObjectException
     */
    private void writeObject(ObjectOutputStream stream) throws InvalidObjectException {
        throw new InvalidObjectException("Proxy required");
    }

    /**
     * Deserialize object. We throw an exception since this method should never
     * be called - the standard serialization engine will create serialized
     * proxies instead. Anyone calling this method directly is probably up to no
     * good and using a manually constructed serialized object.
     * 
     * @param stream
     * @return
     * @throws InvalidObjectException
     */
    private void readObject(ObjectInputStream stream) throws InvalidObjectException {
        throw new InvalidObjectException("Proxy required");
    }

    /**
     * Serializable proxy for our protected class. The encryption code is based
     * on https://gist.github.com/mping/3899247.
     */
    private static class SimpleProtectedSecretProxy implements Serializable {
        private static final long serialVersionUID = 1L;
        private String secret;

        private static final String CIPHER_ALGORITHM = "AES/CBC/PKCS5Padding";
        private static final String HMAC_ALGORITHM = "HmacSHA256";

        private static transient SecretKeySpec cipherKey;
        private static transient SecretKeySpec hmacKey;

        static {
            // these keys can be read from the environment, the filesystem, etc.
            final byte[] aes_key = "d2cb415e067c7b13".getBytes();
            final byte[] hmac_key = "d6cfaad283353507".getBytes();

            try {
                cipherKey = new SecretKeySpec(aes_key, "AES");
                hmacKey = new SecretKeySpec(hmac_key, HMAC_ALGORITHM);
            } catch (Exception e) {
                throw new ExceptionInInitializerError(e);
            }
        }

        /**
         * Constructor.
         * 
         * @param protectedSecret
         */
        SimpleProtectedSecretProxy(ProtectedSecret protectedSecret) {
            this.secret = protectedSecret.secret;
        }

        /**
         * Write encrypted object to serialization stream.
         * 
         * @param s
         * @throws IOException
         */
        private void writeObject(ObjectOutputStream s) throws IOException {
            s.defaultWriteObject();
            try {
                Cipher encrypt = Cipher.getInstance(CIPHER_ALGORITHM);
                encrypt.init(Cipher.ENCRYPT_MODE, cipherKey);
                byte[] ciphertext = encrypt.doFinal(secret.getBytes("UTF-8"));
                byte[] iv = encrypt.getIV();

                Mac mac = Mac.getInstance(HMAC_ALGORITHM);
                mac.init(hmacKey);
                mac.update(iv);
                byte[] hmac = mac.doFinal(ciphertext);

                // TBD: write algorithm id...
                s.writeInt(iv.length);
                s.write(iv);
                s.writeInt(ciphertext.length);
                s.write(ciphertext);
                s.writeInt(hmac.length);
                s.write(hmac);
            } catch (Exception e) {
                throw new InvalidObjectException("unable to encrypt value");
            }
        }

        /**
         * Read encrypted object from serialization stream.
         * 
         * @param s
         * @throws InvalidObjectException
         */
        private void readObject(ObjectInputStream s) throws ClassNotFoundException, IOException, InvalidObjectException {
            s.defaultReadObject();
            try {
                // TBD: read algorithm id...
                byte[] iv = new byte[s.readInt()];
                s.read(iv);
                byte[] ciphertext = new byte[s.readInt()];
                s.read(ciphertext);
                byte[] hmac = new byte[s.readInt()];
                s.read(hmac);

                // verify HMAC
                Mac mac = Mac.getInstance(HMAC_ALGORITHM);
                mac.init(hmacKey);
                mac.update(iv);
                byte[] signature = mac.doFinal(ciphertext);

                // verify HMAC
                if (!Arrays.equals(hmac, signature)) {
                    throw new InvalidObjectException("unable to decrypt value");
                }

                // decrypt data
                Cipher decrypt = Cipher.getInstance(CIPHER_ALGORITHM);
                decrypt.init(Cipher.DECRYPT_MODE, cipherKey, new IvParameterSpec(iv));
                byte[] data = decrypt.doFinal(ciphertext);
                secret = new String(data, "UTF-8");
            } catch (Exception e) {
                throw new InvalidObjectException("unable to decrypt value");
            }
        }

        /**
         * Return protected object.
         * 
         * @return
         */
        private Object readResolve() {
            return new ProtectedSecret(secret);
        }
    }
}

It should go without saying that the encryption keys should not be hard-coded or possibly even cached as shown. This was a short-cut to allow us to focus on the details of the implementation.

Different keys should be used for the cipher and message digest. You will seriously compromise the security of your system if the same key is used.

Two other things should be handled in any production system: key rotation and changing the cipher and digest algorithms. The former can be handled by adding a ‘key id’ to the payload, the latter can be handled by tying the serialization version number and cipher algorithms. E.g., version 1 uses standard AES, version 2 uses AES-256. The deserializer should be able to handle old encryption keys and ciphers (within reason).

Test code

The test code is straightforward. It creates an object, serializes it, deserializes it, and compares the results to the original value.

public class ProtectedSecretTest {

    /**
     * Test 'happy path'.
     */
    @Test
    public void testCipher() throws IOException, ClassNotFoundException {
        ProtectedSecret secret1 = new ProtectedSecret("password");
        ProtectedSecret secret2;
        byte[] ser;

        // serialize object
        try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ObjectOutput output = new ObjectOutputStream(baos)) {
            output.writeObject(secret1);
            output.flush();

            ser = baos.toByteArray();
        }

        // deserialize object.
        try (ByteArrayInputStream bais = new ByteArrayInputStream(ser); ObjectInput input = new ObjectInputStream(bais)) {
            secret2 = (ProtectedSecret) input.readObject();
        }

        // compare values.
        assertEquals(secret1.getSecret(), secret2.getSecret());
    }

    /**
     * Test deserialization after a single bit is flipped.
     */
    @Test(expected = InvalidObjectException.class)
    public void testCipherAltered() throws IOException, ClassNotFoundException {
        ProtectedSecret secret1 = new ProtectedSecret("password");
        ProtectedSecret secret2;
        byte[] ser;

        // serialize object
        try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ObjectOutput output = new ObjectOutputStream(baos)) {
            output.writeObject(secret1);
            output.flush();

            ser = baos.toByteArray();
        }
        
        // corrupt ciphertext
        ser[ser.length - 16 - 1 - 3] ^= 1;

        // deserialize object.
        try (ByteArrayInputStream bais = new ByteArrayInputStream(ser); ObjectInput input = new ObjectInputStream(bais)) {
            secret2 = (ProtectedSecret) input.readObject();
        }

        // compare values.
        assertEquals(secret1.getSecret(), secret2.getSecret());
    }
}

Final words

I cannot overemphasize this – this is primarily an intellectual exercise. As usual the biggest problem is key management, not cryptography, and with the level of effort required for the former you can probably implement a more traditional solution more quickly.

This may still be “good enough” in some situations. For instance you may only need to keep the data around during the duration of a long-running application. In this case you can create random keys at startup and simply discard all serialized data after the program ends.

Source code: https://gist.github.com/beargiles/90182af6f332830a2e0e