Invariant Properties

  • rss
  • Home

DataSource Classloader Headaches

Bear Giles | January 2, 2017

I haven’t been posting since I’ve been very busy learning Hadoop + Kerberos for multiple client environments and getting into shape before it’s too late. (I know it’s “never too late” in principle but I’m seeing family and friends my age who are now unable to do hard workouts due to medical issues. For them it is “too late” to get into better shape so this is no longer an abstract concern for me.)

Part of my broader work is supporting applications with user-provided JDBC drivers. We bundle the datasource (typically HikariCP) and allow the user to specify the JDBC driver jar. Support has been very ad hoc and I’ve been working on parameterized tests that use aether to query the maven central repository for all versions of the datasource and JDBC jars and then verifying that I can make a connection to our test servers using all possible combinations. That’s not always the case, e.g., older JDBC drivers might not support a method required by newer versions of the datasource class, especially for more obscure databases such as hive.

(Note: I don’t mean to pick on Hikari here. I’m seeing this problem in several libraries and I’m just using it as an example.)

The test should be straightforward. With one test class per datasource version:

  1. ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
  2. for (Artifact artifact : /* Aether query */) {
  3.     try {
  4.         URL[] urls = new URL[] {
  5.             new URL("file", "", artifact.getFile());
  6.         }
  7.         ClassLoader cl = new URLClassLoader(urls, oldClassLoader);
  8.         Thread.currentThread().setContextClassLoader(cl);
  9.  
  10.         HikariConfig config = new HikariConfig();
  11.         config.setJdbcUrl(TEST_URL);
  12.         config.setDriverClassName(DRIVER_CLASSNAME);
  13.         DataSource ds = new HikariDataSource(config);
  14.         try (Connection conn = ds.getConnection();
  15.                 Statement stmt = conn.createStatement();
  16.                 ResultSet rs = stmt.executeQuery("select 1 as x")) {
  17.             assertThat(rs.next(), equalTo(true));
  18.             assertThat(rs.getInt("x"), equalTo(1);
  19.         }
  20.     } finally {
  21.         Thread.currentThread().setContextClassLoader(oldClassLoader);
  22.     }
  23. }
ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
for (Artifact artifact : /* Aether query */) {
    try {
        URL[] urls = new URL[] {
            new URL("file", "", artifact.getFile());
        }
        ClassLoader cl = new URLClassLoader(urls, oldClassLoader);
        Thread.currentThread().setContextClassLoader(cl);

        HikariConfig config = new HikariConfig();
        config.setJdbcUrl(TEST_URL);
        config.setDriverClassName(DRIVER_CLASSNAME);
        DataSource ds = new HikariDataSource(config);
        try (Connection conn = ds.getConnection();
                Statement stmt = conn.createStatement();
                ResultSet rs = stmt.executeQuery("select 1 as x")) {
            assertThat(rs.next(), equalTo(true));
            assertThat(rs.getInt("x"), equalTo(1);
        }
    } finally {
        Thread.currentThread().setContextClassLoader(oldClassLoader);
    }
}

(Note: I’m actually using a parameterized junit test that uses the loop to produce the list of parameters. Each parameterized test is then run individually. I’m using an explicit loop here to emphasize the need to restore the environment after each test.)

Only one problem – it can’t find the driver class. Looking at the source code in github reveals the problem:

  1. public void setDriverClassName(String driverClassName) {
  2.     Class c = HikariConfig.class.getClassLoader().loadClass(driverClassName);
  3.     ...
  4. }
public void setDriverClassName(String driverClassName) {
    Class c = HikariConfig.class.getClassLoader().loadClass(driverClassName);
    ...
}

The Hikari classes were loaded by a different classloader than the JDBC driver classes and the ‘parent’ relationship between the classloaders goes the wrong way.

The fix isn’t hard – I need to modify my classloader so it loads both the Hikari datasource library and the JDBC driver library. This requires the use of reflection to create and configure the HikariConfig and HikariDataSource classes but that’s not too hard if I use commons-lang3 helper classes. There’s even a benefit to this approach – I can specify both datasource and JDBC driver jars in the test parameters and no longer need a separate test class for each version of the datasource.

Unfortunately it doesn’t work. I haven’t dug deeper into the class but I noticed the setter only verifies that the class is visible. It’s actually loaded and used elsewhere and it might use a different classloader at that point. Research continues….

But wait, it gets worse!

As an alternative I tried to explicitly register the JDBC driver in order to create the datasource without explicitly naming the JDBC driver classname (if possible):

  1. ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
  2. Driver driver = null;
  3. for (Artifact artifact : /* Aether query */) {
  4.     try {
  5.         URL[] urls = new URL[] {
  6.             new URL("file", "", artifact.getFile());
  7.         }
  8.         ClassLoader cl = new URLClassLoader(urls, oldClassLoader);
  9.         Thread.currentThread().setContextClassLoader(cl);
  10.  
  11.         Class driverClass = (Class) cl.loadClass(DRIVER_NAME);
  12.         driver = driverClass.newInstance();
  13.         DriverManager.registerDriver(driver);
  14.  
  15.         ...
  16.  
  17.     } finally {
  18.         if (driver != null) {
  19.             DriverManager.deregisterDriver(driver);
  20.         }
  21.         Thread.currentThread().setContextClassLoader(oldClassLoader);
  22.     }
  23. }
ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
Driver driver = null;
for (Artifact artifact : /* Aether query */) {
    try {
        URL[] urls = new URL[] {
            new URL("file", "", artifact.getFile());
        }
        ClassLoader cl = new URLClassLoader(urls, oldClassLoader);
        Thread.currentThread().setContextClassLoader(cl);

        Class driverClass = (Class) cl.loadClass(DRIVER_NAME);
        driver = driverClass.newInstance();
        DriverManager.registerDriver(driver);

        ...

    } finally {
        if (driver != null) {
            DriverManager.deregisterDriver(driver);
        }
        Thread.currentThread().setContextClassLoader(oldClassLoader);
    }
}

Incredibly this fails – the deregisterDriver() call throws a SecurityException! This happens even when I explicitly set a permissive SecurityManager in the test setup. Digging into the code I discovered that the DriverManager checks whether the caller has the ability to load the class being deregistered. That sounds like a basic sanity check against malicious behavior but it introduces a classloader dependency. Again it’s not using the classloader I created in order to isolate my tests. The DriverManager is a core class so there’s no solution to this problem.

Edited to add…

I meant that there’s no clean solution to this problem. The DriverManager class uses reflection to learn the classloader of the calling method and verifies that the driver is accessible to it. In our case it’s not – we created a new classloader and it’s still our thread’s contextClassLoader but we’re calling the deregisterDriver() method from a class loaded by the original classloader.

One solution is to write and maintain another class that exists solely to deregister the driver class. That is non-obvious and will be a pain to maintain.

The other solution is to use reflection to make the internal registeredDrivers collection accessible and directly manipulate it in our ‘finally’ clause. That was my final solution.

Lessons learned

If we’re writing a library that allows the user to specify a classname at runtime we MUST test the scenario where the user loads the containing jar in a separate classloader. It’s not enough to only test it when containing jar is in the same classpath as our library – the jar might ultimately be provided by the end user and not the developer.

Categories
java
Comments rss
Comments rss
Trackback
Trackback

« Encrypt your usernames and email addresses! Building Hadoop on Ubuntu 16.10 »

Leave a Reply

Click here to cancel reply.

You must be logged in to post a comment.

Archives

  • May 2020 (1)
  • March 2019 (1)
  • August 2018 (1)
  • May 2018 (1)
  • February 2018 (1)
  • November 2017 (4)
  • January 2017 (3)
  • June 2016 (1)
  • May 2016 (1)
  • April 2016 (2)
  • March 2016 (1)
  • February 2016 (3)
  • January 2016 (6)
  • December 2015 (2)
  • November 2015 (3)
  • October 2015 (2)
  • August 2015 (4)
  • July 2015 (2)
  • June 2015 (2)
  • January 2015 (1)
  • December 2014 (6)
  • October 2014 (1)
  • September 2014 (2)
  • August 2014 (1)
  • July 2014 (1)
  • June 2014 (2)
  • May 2014 (2)
  • April 2014 (1)
  • March 2014 (1)
  • February 2014 (3)
  • January 2014 (6)
  • December 2013 (13)
  • November 2013 (6)
  • October 2013 (3)
  • September 2013 (2)
  • August 2013 (5)
  • June 2013 (1)
  • May 2013 (2)
  • March 2013 (1)
  • November 2012 (1)
  • October 2012 (3)
  • September 2012 (2)
  • May 2012 (6)
  • January 2012 (2)
  • December 2011 (12)
  • July 2011 (1)
  • June 2011 (2)
  • May 2011 (5)
  • April 2011 (6)
  • March 2011 (4)
  • February 2011 (3)
  • October 2010 (6)
  • September 2010 (8)

Recent Posts

  • 8-bit Breadboard Computer: Good Encapsulation!
  • Where are all the posts?
  • Better Ad Blocking Through Pi-Hole and Local Caching
  • The difference between APIs and SPIs
  • Hadoop: User Impersonation with Kerberos Authentication

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Pages

  • About Me
  • Notebook: Common XML Tasks
  • Notebook: Database/Webapp Security
  • Notebook: Development Tips

Syndication

Java Code Geeks

Know Your Rights

Support Bloggers' Rights
Demand Your dotRIGHTS

Security

  • Dark Reading
  • Krebs On Security Krebs On Security
  • Naked Security Naked Security
  • Schneier on Security Schneier on Security
  • TaoSecurity TaoSecurity

Politics

  • ACLU ACLU
  • EFF EFF

News

  • Ars technica Ars technica
  • Kevin Drum at Mother Jones Kevin Drum at Mother Jones
  • Raw Story Raw Story
  • Tech Dirt Tech Dirt
  • Vice Vice

Spam Blocked

53,793 spam blocked by Akismet
rss Comments rss valid xhtml 1.1 design by jide powered by Wordpress get firefox