Invariant Properties

This is the first in a series of entries regarding the Java SecurityManager.  The JVM was designed with security as a foremost concern but most of us run without a SecurityManager for three reasons:

• we are unfamiliar with it,
• we don’t perceive a need,
• it is a real PITA to set up.

The first point is a simple matter of education.

The second point is more subtle. Most of us have always developed software in a completely open environment.  That doesn’t need to change.  Unfortunately we’ve also always run our software in a completely open environment. That wasn’t a huge problem historically because data was passive and libraries were simple enough for experienced developers to understand what was happening under the covers.

That’s changed though.  Now data is much more intelligent and at a minimum we need to be able to parse it and perform decompression and/or decryption (all of which introduces the possibility of exploitable stack overflows).  It might even include explicit or implicit scripting capabilities.  How many people would realize that XML external entities (XEE) could be used to pull in arbitrary files from the filesystem or perform a denial of service attack on a service?

Careful C developers have addressed this for years.  At the top of the main procedure they would call setrlimit() to set limits on various system resources.  If you know you already have all necessary files open there’s no reason to have permission to open another file.  You can prevent forked jobs.  You can limit the maximum amount of memory consumed.

This is crude but available everywhere.  Secure Linux (SELinux) takes this to a while new level but it requires a lot more effort to do it right.

The java equivalence is a SecurityManager and its policy.  Like C’s setrlimit you don’t have to be perfect to do have a dramatic improvement in security.

For instance, how many files do you need to open?  Really?  On a typical webapp I only count a handful of locations:

• the class path (read-only)
• the temp directory (${java.io.tmpdir}) (read, write, delete, but not execute)
• the servlet context (read-only)
• the logging facility (write-only?)
• any directories for persistent storage, if any.
• maybe the user’s home directory, for configuration information.  (${user.home})

It’s extraordinary for any webapp to require more than that – and most would only require the first four if they use a database for storage and JNDI for configuration.

Network/Socket requirements are similar.  You need to be able to find and connect to your resources – database, mail server, JMS, etc.  But after that you probably don’t need to be able to initiate any more connections.  You can either specify no permissions, or just the ‘resolve’ and ‘connect’ permissions to those specific servers.

The other permissions are a good news/bad news situation.  It’s great that you have fine control over these permissions, but it’s a pain that you have to specify them explicitly.  However there are shortcuts that are Good Enough in nearly all cases.  E.g., grant ‘read-only’ permission to all system properties.  When was the last time you needed to programmatically set a system property?  When was the last time you needed to do it after program initialization?  (The downside to this is that you could leak information if somebody managed to get a bit of core into your system — so you just don’t provide sensitive information like details about the user or system.)

The final point is very real.  The security manager was designed to be set up via an external policy file and much of it is ‘write only’. You can’t easily learn what your existing permissions are.  (I think you can do it by specifying a DomainCombiner and instrumenting the method that combines the sets of properties, but I haven’t verified this.) There’s no handy reference to all of the required permissions for various tasks, e.g., the dozens of permissions required if you need to create an object.  Any object.

However we can create our own SecurityManager programmatically, one that provides exactly the permissions we need.  It’s contrary to current “best practices”… but those best practices aren’t worth much if nobody uses them. (Besides you can still enforce the external policy file if you have a dedicated security team that’s responsible for maintaining it.)

If we want to get fancy we can even define our own Annotations that we can use on each class and/or method to specify exactly what permissions that class or method requires.  It’s not necessary for there to be a single list of required permissions.

Finally we can use an instrumented SecurityManager to learn what permissions we require.  This isn’t a game of “mother may I”, we can run our application with a transparent SecurityManagement during development and integration tests to learn exactly what permissions we should provide in the production environment.

An example instrumented SecurityManager follows.  To use it simply call System.setSecurityManager(new LoggingSecurityManager()) near the top of your application.

/**
 * Security manager that records missing permissions but otherwise
 * allows everything.
 *
 * @author bgiles
 */
public class LoggingSecurityManager extends SecurityManager {
  private AccessControlContext ctx;
  private Set missingPermissions = new HashSet();

  public LoggingSecurityManager() {
    ctx = new AccessControlContext(new ProtectionDomain[] {
      new ProtectionDomain(null, null)
    });
  }

  public LoggingSecurityManager(AccessControlContext ctx) {
    this.ctx = ctx;
  }

  @Override
  public AccessControlContext getSecurityContext() {
    return ctx;
  }

  @Override
  public void checkPermission(Permission perm) {
    try {
      ctx.checkPermission(perm);
    } catch (SecurityException e) {
      if (!missingPermissions.contains(perm)) {
        missingPermissions.add(perm);
        System.out.println(perm);
      }
    }
  }

  public Set getMissingPermissions() {
    return missingPermissions;
  }
}

A good initial set of Permissions follow.  This approach is nice because it would be easy to write the permissions as a standard Policy file.

public class LoggingSecurityManager extends SecurityManager {
  private AccessControlContext ctx;
  private Properties properties = new Properties;
  private Set missingProperties = new HashSet();

  public LoggingSecurityManager() {
    properties.add(
      new FilePermission(System.get("java.io.tmpdir") + "/-", "read,write,delete"));

    // maybe...
    properties.add(
      new FilePermission(System.get("user.home") + "/-", "read,write,delete"));

    addSystemPropertyPermissions();
    addSecurityPermissions();
    addClassPathPermissions();
    addOtherPropertyPermissions();

    permissions.add(new RuntimePermission("accessClassInPackage.sun.reflect"));
    permissions.add(new RuntimePermission("accessClassInPackage.sun.jdbc.odbc"));
    permissions.add(new RuntimePermission("accessClassInPackage.sun.security.provider"));
    permissions.add(new SocketPermission("localhost", "resolve"));
    permissions.add(new NetPermission("getProxySelector"));

    // scary permissions required for constructors
    permissions.add(new ReflectPermission("suppressAccessChecks"));  (!!)
    permissions.add(new RuntimePermission("createClassLoader")); (!!)
    permissions.add(new SecurityPermission("putProviderProperty.SUN"));
    permissions.add(new RuntimePermission("readFileDescriptor"));

    permissions.add(new RuntimePermission("writeFileDescriptor"));

    ctx = new AccessControlContext(new ProtectionDomain[] {
      new ProtectionDomain(null, permissions)
    });
  }

  /**
    * Add read-only permission to read system properties.
    * We may want to filter this list to remove sensitive information
    */
  public void addSystemPropertyPermissions() {
    for (Object key : Collections.list(System.getProperties().keys())) {
      permissions.add(new PropertyPermission((String) key, "read"));
    }
  }

  /**
   * Add read-only permissions for initializing security.
   */
  public void addSecurityPermissions() {
    permissions.add(new SecurityPermission("getPolicy"));
    permissions.add(new SecurityPermission("getProperty.random.source"));
    permissions.add(new SecurityPermission("getProperty.securerandom.source"));

    for (int i = 1; i < 10; i++) {
      permissions.add(new SecurityPermission("getProperty.security.provider." + i));
    }

    String s = Security.getProperty("securerandom.source");
    if ((s != null) && s.startsWith("file:/")) {
      permissions.add(new FilePermission(s.substring(5), "read"));
    }

    // should have been covered already but wasn't....
    permissions.add(new FilePermission("/dev/random", "read"));
  }

  /**
   * Add read-only permissions for everything on classpath.
   */
  public void addClassPathPermissions() {
    permissions.add(new FilePermission(String.format("%/lib/-",
      System.getProperty("java.home")), "read"));

    // add standard class path.
    String pathSep = System.getProperty("path.separator");
    for (String entry : System.getProperty("java.class.path").split(pathSep)) {
      File f = new File(entry);
      if (f.isFile()) {
        permissions.add(new FilePermission(entry, "read"));
      } else if (f.isDirectory()) {
        permissions.add(new FilePermission(String.format("%s/-", entry), "read"));
      } // or could be neither fish nor fowl
    }

    // add endorsed extensions.
    for (String dir : System.getProperty("java.ext.dirs").split(pathSep)) {
      permissions.add(new FilePermission(String.format("%s/-", dir), "read"));
    }
  }

  /**
   * Add other standard properties.
   */
  public void addOtherPropertyPermissions() {
    permissions.add(new PropertyPermission("jdbc.drivers", "read"));
    permissions.add(new PropertyPermission("java.security.egd", "read"));
    permissions.add(new PropertyPermission("socksProxyHost", "read"));
  }
}

Finally we can provide programmatic checks within our SecurityManager.  This is less desirable since it would not be possible to replace them with standard policy file but it might be the best (or only) way to solve the problem.

For instance, if we need to be able to connect to any MySQL server we can use

public class MySqlSecurityManager extends SecurityManager {
  /**
   * Allow connection to any mysql server
   */
  public void checkPermission(Permission perm) {
    if ((perm instanceof SocketPermission) && perm.getName().endsWith(":3306")
          && perm.getActions().contains("connect")) {
      return;
    }
    super.checkPermission(perm);
  }
}

Added in December 2011: This information is out of date. Sixapart and others now use a form of registered web service callbacks.

I’ve recently faced the problem of breaking apart a streaming Atom feed from http://updates.sixapart.com/atom-stream.xml. This is a link that is never closed – it just keeps pushing out Atom content mixed with timestamps.  You have to be prepared to get gigabytes of data across the connection.

You can feed this stream directly into a SAX parser but that’s not a good idea since any fatal error blows out the rest of the data stream. We need to process the data in chunks, ideally just one Atom entry per chunk, but how can we efficiently split the live input stream?  This reeks of a problem that is easy to describe but a pain to implement correctly.

Fortunately I remembered an old article on using java.util.Scanner.  It’s the perfect solution to this problem.

For this problem we first perform an outer scan of the URL connection using a delimiter of “<time>\\s*”.  This has three benefits:

• we can check for false positives by verifying that the start of each block is the 10 digit timestamp and closing timestamp tag.  That check would catch this blog entry, for example, if it’s not properly encoded with XML entities.  If you’re really careful you can even verify that the timestamp is just one second after the prior timestamp.
• we have a timestamp for that block.
• we know that the block remaining will contain nothing but Atom entries.

Each chunk can be written to the disk for later processing.  Do one thing at a time.

The inner scanner uses a “</feed>\\s*” delimiter to read one entry at a time.  Feed them to our Atom parser and we’re done.

Bottom score: often-overlooked java class: 1.  potentially nasty problem: 0.

(edit: another good page: Scanning text with java.util.Scanner)

Remember the old joke about the guy pounding on a screw with a hammer since that’s the only tool he knew?  We can easily fall into the same hole, especially in IT.  The market dominant solution can stagnate, the “barely better than a class project” tool can become a powerful alternative in just a dog year.

I’ve used Tomcat for years.  More importantly I’ve used it as a light appserver with container-managed resources – database, mail services, JMS.  I’ve written my own security realm so I can have container-managed authentication.  (I still usually set up Spring Security and create a HttpServletRequest proxy that makes spring-based authentication look like container-based authentication.  It’s nice to have an option though.)

I’ve also used Jetty in integration tests and one-off servers (e.g., a departmental artifactory respository).  I know it offers container-based resources but mostly see myself using it in integration tests.

So what am I missing?  Everyone and their dog uses JBoss, and I’ve used Websphere and Oracle App Server at work.  I’ve heard a lot of good things about Glassfish.  It’s been the reference implementation for a few years (Tomcat was earlier) and was the first app server to support EJB 3 and EJB 3.1.  (See: reference implementation).  It has a small but dedicated following and they’re worried that Oracle may drop it and whether the community should pick up a fork.

Is it worth it?

I decided to give it a quick spin.  So far I’ve only gotten as far as “hello, world!” (in this case installing the Hudson war and verifying that it came up) but maybe that’s enough to help other people get started.

The installation process was fairly straightforward.

1. Download package from java.sun.com.  It’s very old school – it comes as a 54 MB .sh file!
2. Create glassfish user and group
3. Create /opt/glassfish directory and change ownership to glassfish
4. ssh into system as glassfish.  (I find this the easiest way to handle X Windows issues)
5. run script.  It will launch a swing app that allows you to configure and install the server.

The installer was well behaved and didn’t require root access.

There were two small snags:

• glassfish runs on ports 4848 (admin) and 8080 (apps).  This causes problems if you also have a tomcat or jetty instance running, e.g., in a development environment.  It’s a simple configuration change but it may not be immediately obvious what the problem is.
• it took a while to find the log files.

Finally I installed Hudson from the admin console.  It went without incident.

The 30-second review is that I’ve done most of this before in Tomcat — but I had to do the installations and configuration myself.  There’s a lot of value of being ready to run the instant after the installation completes instead of several hours later but there are two caveats:

• Tomcat administration is getting better so it may no longer be necessary to edit configuration files directly
• TCat probably bundles the database/mail/jms resources already
• I don’t know how much effort it will take to change the glassfish resources from the bundled resources to existing external ones.

Bottom line: it’s worth looking at further.  I should also revisit the standard Tomcat admin console and TCat.

I’ve heard the misquote several times in the last few days – usually misunderstood – so it’s time for a quick history lesson.

The Oliver Wendell Holmes opinion in Schenck v. United States did not refer to “yelling fire in a crowded theater,” it referred to falsely yelling fire in a crowded theater.  (The actual opinion referred to “the most stringent protection of free speech would not protect a man in falsely shouting fire in a theatre and causing a panic.”)  He wrote this in living memory of several horrific fires where people were trapped in burning theaters – fires that directly lead to safety rules that we have to this day.  E.g., exits were locked and could only be opened with a key, doors opened inwards so a panicked crowd would prevent the door from being opened at all even if it were unlocked, etc.  We take safety measures like panic bars on doors that open outwards for granted today but these are the hard-earned lessons of tragedies.

In other words, yelling “fire” would inevitably cause a panicked rush to the exits.  Falsely yelling fire is a serious offense.

But not yelling fire if there is a fire?  Especially in a pre-retardant era where you could go from a small fire to an engulfing inferno in a matter of minutes?  It’s hard to think of a more reprehensible failure to act.  Of course you can and should yell fire if there really is a fire!

That brings us to the larger context.  The case arose because Schenck sent out flyers critical of the war effort in WW-I.  The government’s position, as phrased by some people, was that the hayseed military recruits were so feeble-minded that they would revolt on the basis of nothing more than a flyer.  That’s the “clear and present danger” to the rule of law posed by these flyers.

Of course that’s ludicrous. What was obvious to the entire court in 1919 was finally buried in 1969 in Brandenburg v. Ohio.  That’s the law today – speech is protected unless it is likely to incite imminent lawless acts.  The battleground today is where the line for “imminent” is drawn — we’re no longer limited to a lone man ranting to a crowd in a park.  We have radio broadcasts or podcasts or tweets or who-knows-what that can reach hundreds of thousands or millions of people.

This brings us back to the people referring to “yelling fire in a crowded theater.”  I wonder what they would think if they knew the full phrase?  If they knew the broad outline of the court case (and how “flyers” in 1919 are tweets and youtube videos in 2010)?  If they knew that the defendant was the Secretary of the Socialist Party?

I’ve been playing with Castor for object-xml mapping and ran into a notorious problem with top-level lists.  Ironically I knew this would be a problem and had written code to avoid it — and still got tripped up since I saw the usual thing and quietly assumed it was the correct thing.

The general idea is that you can map a conceptual top-level List<Order> into the following XML:

<?xml version="1.0"?>
<Orders>
  <Order>...</Order>
  <Order>...</Order>
  <Order>...</Order>
</Orders>

Some people want to drop the outer <Orders> tag since it’s nothing but syntatic sugar – no other attributes or elements, but well-formed XML requires a single root element.

The answer is to create a wrapper class that’s only used in the OXM layer.

public class ListOfOrders implements Iterable<Order>{
   private List<Order> orders = new ArrayList<Order>();

   public List<Order> getOrders() {
      return orders;
   }

   public void setOrders(List<Order> orders) {
      this.orders = orders;
   }

   public Iterator<Order> iterator() {
      return orders.iterator();
   }
}

So what went wrong?  When I wrote the unit tests I created my list of test Orders and naturally passed it into the marshaller.  That worked, but the generated XML had a top-level element of <array-list> and the unmarshaller failed with the notorious “The class for the root element ‘array-list’ could not be found” error.  I was so used to thinking “list of orders” and seeing List<Order> that it took me an hour to realize that I really needed my new ListOfOrders class here.  It was a classic variation on only seeing what you expect to see.

Dropbox is a service that lets you synchronize files between systems and even a web interface.  If you’re willing and able to install the application on your work system it’s a clean way to share files between home and office.  One obvious drawback is that you need to be careful about what you put into the folder.  Resume?  check.  Pictures of the family?  check.  Pictures of the family at the nude beach?  not such a great idea.

Sounds like a great place to put your eclipse projects.  It’s a regular folder during the day but accessible from home.   It turns out there are some pretty important details.

First attempt: create an eclipse workspace under ~/Dropbox and… sproing.  Not such a great idea – eclipse keeps some logs that can’t be synced by dropbox if they’re updated simultaneously by two eclipse instances (think work and home).  It barfed so bad that I had to nuke my workspace.

Second attempt: create an eclipse workspace in the usual place and link it to an external source folder under ~/Dropbox.  You have to create a buck-naked project and add all of the natures (java, maven, etc.) to it explicitly and/or manually.  (I just updated .project and .classpath with values from an existing project.)  It works great, you can bounce between systems without incident… until you log into your old linux box with a fairly small /home partition.  100% disk usage.

It turns out that the linux dropbox client keeps some sort of database about changes.  The Windows (and other) clients probably keep a database as well. That’s not a big deal if your dropbox contains a copy of your resume and a few family pictures but is a killer if it is continuously rebuilding even a trivial spring-enabled webapp.  In a day or two I had pumped gigabytes of data into that file.  Oops.  A 500 GB partition will delay the inevitable but at some point you’ll run out of disk space.  And really piss off dropbox.com because of your heavy network usage.

Third attempt: create a dropbox-workspaces-target under my home directory (Linux) and my /Users/me folder (NOT under My Documents!).  This folder is on my local hard disk.  I then set it as my output directory in my pom.xml file:

<build>
  <directory>${user.home}/dropbox-workspaces-target/${project.artifactId}/target</directory>
  <outputDirectory>${user.home}/dropbox-workspaces-target/${project.artifactId}/classes</outputDirectory>
  <testOutputDirectory>${user.home}/dropbox-workspaces-target/${project.artifactId}/test-classes</testOutputDirectory>
</build>

That fixes the problem for maven.  The native eclipse build still puts the files under dropbox but that’s also configurable.

You often need to build related pieces of code into complementary packages.  The classic example is a client-server tool – you want to develop the client and server together but package them separately.  The client code ends up in a .jar in the client’s application and the server code ends up in a .war on the appserver.  You definitely want to develop the code together unless you have an iron-clad agreement on the API.  That’s pretty much a requirement for public-facing APIs but agile development of internal APIs tends to have a “we’ll define it when we need it”.  And that’s a good thing!

(Quick clawback – as long as you either have a good idea of what you need or are willing to toss it if your initial assumptions are wrong.  But it’s often far cheaper to develop, toss and redevelop than to have endless meetings hammering out the API only to discover that you missed something anyway.  It will be far more expensive if you don’t know what you need but can’t make major changes if your first guess is wildly off the mark.)

You can do this with ant and some careful exclusion rules.  But can you do it in maven?

The answer is documented in a filing cabinet in the basement of a small post office in rural Britain… and guarded by a tiger.  (see: HHGTTG).  Seriously, you would think this problem comes up often enough for the answer to be reasonably easy to find.  It turns out it’s relatively straightforward.  Under your root directory create one directory per subproject, e.g., common, server and client.  Under each directory create the usual maven source tree (src/main/java, src/main/resources, etc.).

The parent pom.xml file needs to have ‘pom’ packaging and include a top-level stanza of

<modules>
  <module>common</module>
  <module>server</module>
  <module>client</module>
  <module>integration-test</module>
</modules>

Each of the subproject pom.xml files should include a header of

<parent>
   <groupId>my.parent.group</group>
   <artifactId>project-id</group>
   <version>1.0</version>
</parent>

<modelVersion>4.0.0</modelVersion>
<artifactId>project-id-server</artifactId>
<packaging>war</packaging>
...

and so on.   The parent and child group and artifact ids don’t have to be related but they usually will be.  Otherwise why would you be building the individual artifacts in the same master project?
One fine point: if you’re using eclipse-maven integration you’ll want to put all of your dependencies in the parent pom.xml so you don’t get red marks everywhere.  It sucks if you’re a purist but it’s not a huge issue.

First post to announce that I’m moving my blog over from my own jroller instance.  There’s nothing wrong with jroller, I just haven’t needed any custom modules or had time to do much with the layouts.  I’ll be republishing old posts over the next month or two.