Investigate workarounds for build systems that change JAR files, but don't close their classloaders

[uschindler]

Some build systems (Maven sometimes, but also Gradle) opened URLClassLoaders on a JAR file they created. They just release the ClassLoader, but don't close it (as only Java 7+ allows this, and most build system are Java 5 or even below). You can check for Closeable interface and close if it is, but not all users do this correctly.

The problem with forbiddenapis is: It also uses a URLClassLoader (but closes correctly, so no issue here), but if the build system previously opened a URLClassLoader, then rebuilt the Jar file for some reason, the internal cache gets invalidated and breaks. Loading classes fails with FileNotFoundException (on older pre-Java 7 JVMs it sometimes crushed completely with mmap SIGSEGV).

The workarounds are:

• Add a setting to the Checker (one more enum Option), so it disables the URLConnection cache when fetching resources. This may slowdown forbiddenapis, but only users affected by this must use it.
• Use a hack I investigated: If opening resource fails with IOException (FNFE only), do the following: If JarURLConnection, call getJarFile(), close it, and get a new URLConnection(). Closing the JAR file removes the open file from Cache. A new URLConnection will then reopen it. The problem with that approach is: It breaks other threads accessing the same jar files possibly in other ClassLoaders. So we should carefully only apply that hack on URLs we know that they are from our URLClassLoader.

[uschindler]

I created a PR: #76

[uschindler]

I am not sure if we really need this workarounds. @rjernst has only seen the problem in Gradle builds, but was not able to reproduce. It looked like this was caused by a cranky Gradle Daemon running in background and changing JARs.

I will keep this issue/PR open for a while, but will not merge the stuff. At least we have a solution, but it is far from good to add workarounds for broken stuff anywhere else.

[tbroyer]

FWIW, I got hit by this in my build (Gradle with daemon), at least I suppose this is this issue; I experienced FileNotFoundException for classes that are present in my JARs:

* What went wrong:
Execution failed for task ':oasis-webapp:forbiddenApisMain'.
> de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API calls failed: java.io.FileNotFoundException: JAR entry oasis/model/applications/v2/SimpleCatalogEntry.class not found in …/oasis-model/build/libs/oasis-model.jar

[uschindler]

It is known that this happens with the daemon. The reason is some resource leak in the daemon, that does not close its classloaders or the JAR files. The daemon keeps some classloader alive that refers to your JAR file. If it then gets overwritten by another build step, forbiddenapis falls on this, because th cache is global per JVM.

[tbroyer]

Has this been reported upstream? Do you know what in Gradle keeps the classloaders alive?

[uschindler]

@tbroyer the problem is the Gradle daemon. In the PR we added a detection so it disables caching if the build is running in daemon.

But no, we have no idea why this happens. I think it might be caused by the fact that Gradle is still on Java 6, where URLClassLoader does not have a close() method. But I have no idea where to search for it. @rjernst figured out that the daemon "touches" JAR files in the JAR task, although they are not rebuilt. This makes the JDK think that they changed (although they did not change), so leading to the problem.