Invariant Properties

I haven’t been posting since I’ve been very busy learning Hadoop + Kerberos for multiple client environments and getting into shape before it’s too late. (I know it’s “never too late” in principle but I’m seeing family and friends my age who are now unable to do hard workouts due to medical issues. For them it is “too late” to get into better shape so this is no longer an abstract concern for me.)

Part of my broader work is supporting applications with user-provided JDBC drivers. We bundle the datasource (typically HikariCP) and allow the user to specify the JDBC driver jar. Support has been very ad hoc and I’ve been working on parameterized tests that use aether to query the maven central repository for all versions of the datasource and JDBC jars and then verifying that I can make a connection to our test servers using all possible combinations. That’s not always the case, e.g., older JDBC drivers might not support a method required by newer versions of the datasource class, especially for more obscure databases such as hive.

(Note: I don’t mean to pick on Hikari here. I’m seeing this problem in several libraries and I’m just using it as an example.)

The test should be straightforward. With one test class per datasource version:

ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
for (Artifact artifact : /* Aether query */) {
    try {
        URL[] urls = new URL[] {
            new URL("file", "", artifact.getFile());
        }
        ClassLoader cl = new URLClassLoader(urls, oldClassLoader);
        Thread.currentThread().setContextClassLoader(cl);

        HikariConfig config = new HikariConfig();
        config.setJdbcUrl(TEST_URL);
        config.setDriverClassName(DRIVER_CLASSNAME);
        DataSource ds = new HikariDataSource(config);
        try (Connection conn = ds.getConnection();
                Statement stmt = conn.createStatement();
                ResultSet rs = stmt.executeQuery("select 1 as x")) {
            assertThat(rs.next(), equalTo(true));
            assertThat(rs.getInt("x"), equalTo(1);
        }
    } finally {
        Thread.currentThread().setContextClassLoader(oldClassLoader);
    }
}

(Note: I’m actually using a parameterized junit test that uses the loop to produce the list of parameters. Each parameterized test is then run individually. I’m using an explicit loop here to emphasize the need to restore the environment after each test.)

Only one problem – it can’t find the driver class. Looking at the source code in github reveals the problem:

public void setDriverClassName(String driverClassName) {
    Class c = HikariConfig.class.getClassLoader().loadClass(driverClassName);
    ...
}

The Hikari classes were loaded by a different classloader than the JDBC driver classes and the ‘parent’ relationship between the classloaders goes the wrong way.

The fix isn’t hard – I need to modify my classloader so it loads both the Hikari datasource library and the JDBC driver library. This requires the use of reflection to create and configure the HikariConfig and HikariDataSource classes but that’s not too hard if I use commons-lang3 helper classes. There’s even a benefit to this approach – I can specify both datasource and JDBC driver jars in the test parameters and no longer need a separate test class for each version of the datasource.

Unfortunately it doesn’t work. I haven’t dug deeper into the class but I noticed the setter only verifies that the class is visible. It’s actually loaded and used elsewhere and it might use a different classloader at that point. Research continues….

But wait, it gets worse!

As an alternative I tried to explicitly register the JDBC driver in order to create the datasource without explicitly naming the JDBC driver classname (if possible):

ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
Driver driver = null;
for (Artifact artifact : /* Aether query */) {
    try {
        URL[] urls = new URL[] {
            new URL("file", "", artifact.getFile());
        }
        ClassLoader cl = new URLClassLoader(urls, oldClassLoader);
        Thread.currentThread().setContextClassLoader(cl);

        Class driverClass = (Class) cl.loadClass(DRIVER_NAME);
        driver = driverClass.newInstance();
        DriverManager.registerDriver(driver);

        ...

    } finally {
        if (driver != null) {
            DriverManager.deregisterDriver(driver);
        }
        Thread.currentThread().setContextClassLoader(oldClassLoader);
    }
}

Incredibly this fails – the deregisterDriver() call throws a SecurityException! This happens even when I explicitly set a permissive SecurityManager in the test setup. Digging into the code I discovered that the DriverManager checks whether the caller has the ability to load the class being deregistered. That sounds like a basic sanity check against malicious behavior but it introduces a classloader dependency. Again it’s not using the classloader I created in order to isolate my tests. The DriverManager is a core class so there’s no solution to this problem.

Edited to add…

I meant that there’s no clean solution to this problem. The DriverManager class uses reflection to learn the classloader of the calling method and verifies that the driver is accessible to it. In our case it’s not – we created a new classloader and it’s still our thread’s contextClassLoader but we’re calling the deregisterDriver() method from a class loaded by the original classloader.

One solution is to write and maintain another class that exists solely to deregister the driver class. That is non-obvious and will be a pain to maintain.

The other solution is to use reflection to make the internal registeredDrivers collection accessible and directly manipulate it in our ‘finally’ clause. That was my final solution.

Lessons learned

If we’re writing a library that allows the user to specify a classname at runtime we MUST test the scenario where the user loads the containing jar in a separate classloader. It’s not enough to only test it when containing jar is in the same classpath as our library – the jar might ultimately be provided by the end user and not the developer.