Java 9: Better detection of runtime classes

[uschindler]

In Java 9 we get modules, so the bootclasspath is no longer useful. We added support for the new URL style of class loader URLs using "jrt:" URIs, but the check for runtime classes just uses this protocol prefix to define that a class is coming from the runtime.

This is not 100% correct, in later stages of the JDK9 development, the JAR file format will be "depecated", too. Application classes will then be loaded from jrt: URLs, too (I was talking with Brian Goetz about this).

To correctly detect classes form the runtime, we need to maintain a list of module names/patterns (like the list of packages) containing stuff like "java.", "jdk." and so on. If a class resolves to an URL with this modules in the URL, we can assume that its a runtime class. The current code may also detect classes from outside the JVM as runtime classes (if the application code is installed as a module).

[rmuir]

See also this response from jigsaw-dev list: http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-September/004482.html

[uschindler]

The problem is that you cannot use/check the file system with older Java versions that are still supported by forbiddenapis.

In addition, the whole answer only gives you a comment about this initial check for the supported JDK. If this check fails (as it does), forbiddenapis cannot use the runtime for static analysis: The problem is more that forbiddenapis really wants to have the bytecode of the classes for the static analysis (so like the Lucene test that broke, it MUST load the "java/lang/String.class" bytecode). And for that it calls getSystemClassLoader#getResourceAsStream() :(

I think this is a bug in JIGSAW. The system classloader is no longer a standard URLClassLoader (of course, to enforce module restrictions), but that still needs to make the class bytecode as resources available.

This special classloader needs to load the classes anyways after doing its checks. Why can't it do the checks not also for getResource*() methods?

[uschindler]

I made a first preview on solving the problem: https://github.com/policeman-tools/forbidden-apis/compare/features/java9_module_support

I have to test this now. It passes all tests with earlier versions, so the idea here is the one by Alan Bateman (http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-September/004485.html):

• As a first step, the Checker class uses reflection to lookup the public Java 9 API (which is not accessible from earlier Java versions). It looks up Class#getModule() and also the module's Module#getResourceAsStream()
• If this works, we can assume we have a Java 9 module system so we skip the legacy stuff

During running checker, whenever it needs to load a Class' bytecode, we use the following approach:

• First try conventional way by using ClassLoader#getResource(). If this returns non-null, we use the legacy behaviour. This affects earlier JDKs but also classes from the unnamed public module (the application classes we are checking).
• If getResource() returns null, we try to use module system: Unfortunately we have to actually load the requested class (although we never need it, so we use resolve=false - but this is still a bad overhead). Once we have a class object, we use the above 2 reflected methods to get the module and then call GetResourceAsStream on the module. This should return the bytecode. If it does, we also know that the class is coming from the runtime, so we can set the flag isRuntime=true.

[uschindler]

This was merged into master. I will keep this issue open, because the new module system is not yet in the final state. We should also make sure which modules we count as "runtime" (shipped with JDK). There is no decision yet on the JDK mailing lists / JEP about Jigsaw. After that is solved, we can also use this information also to improve the checker for internal runtime apis (which is a simple regex on the class name + package at the moment).

[plevart]

Hi Uwe,

If you already load the class and obtain j.l.Class object for it (in order to get to the Module and .class resource), then you could also ask for Class#getClassLoader() and see if it is one of bootstrap or extension class loaders:

Class clazz = ...
boolean isRuntimeClass = (clazz.getClassLoader() == null || clazz.getClassLoader() == ClassLoader.getSystemClassLoader().getParent());

Would that work?

[uschindler]

Hi, this is a good idea for Java 9, although I am not sure if the condition is fine for that. I would like to detect - based on the module object / name /... - if its part of the runtime or maybe some external module added by a 3rd party.

The other stuff (public or private API) can be figured out by exported packages; in pre-Java 9 versions using the black/whitelist we currently use...

[plevart]

I think ClassLoader identity - whether it is bootstrap (== null) or extension (== ClassLoder.getSystemClassLoader().getParent) - is still a good indication of whether it is part of Java runtime or some external library. But you may not need to load the class to find that out. There might be a solution in the new Module/Layer/etc APIs. Stay tuned.

[plevart]

I played with jigsaw API a bit and came up with the following:

package myapp;

import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Layer;
import java.lang.reflect.Module;
import java.util.Map;
import java.util.TreeMap;

public class ReadableModulesResourceLoader {

    private final Map<String, Module> pkg2moduleMap = new TreeMap<>();

    /**
     * Constructs a ReadableModulesResourceLoader that is able to load resources
     * from any of readable modules according to given target module.
     *
     * @param target                    the target module that reads modules that
     *                                  contain resources to be loaded.
     * @param includeUnexportedPackages if true, returned loader will also load resources
     *                                  from un-exported packages of all readable modules.
     */
    public ReadableModulesResourceLoader(Module target, boolean includeUnexportedPackages) {
        Layer layer = target.getLayer();
        if (layer == null) {
            // unnamed target has no layer, so we take the boot layer
            layer = Layer.boot();
        }
        // search all layers for readable modules
        for (; layer != null; layer = layer.parent().orElse(null)) {
            for (Module module : layer.modules()) {
                // only consider named modules that are readable by sourceModule
                if (module.isNamed() && target.canRead(module)) {
                    // register packages
                    for (String pkg : module.getPackages()) {
                        // include all packages if requested or only those exported to target
                        if (includeUnexportedPackages || module.isExported(pkg, target)) {
                            Module oldModule = pkg2moduleMap.putIfAbsent(pkg, module);
                            if (oldModule != null) {
                                throw new IllegalStateException(
                                    "Package: " + pkg +
                                    " is defined in at least two modules: " + oldModule.getName() +
                                    " and " + module.getName());
                            }
                        }
                    }
                }
            }
        }
    }

    /**
     * Returns an input stream for reading a resource in a readable module
     * containing package of the resource, {@code null} if the resource is not in
     * that module or if the package that would contain it is not exported and
     * this loader was not constructed with {@code includeUnexportedPackages}
     * being true.
     *
     * @throws IOException If an I/O error occurs
     */
    public InputStream getResourceAsStream(String name) throws IOException {
        int lastSlash = name.lastIndexOf('/');
        if (lastSlash < 0) {
            // root package -> can't be part of named module
            return null;
        }
        String pkgName = name.substring(0, lastSlash).replace('/', '.');
        Module module = pkg2moduleMap.get(pkgName);
        return module == null ? null : module.getResourceAsStream(name);
    }


    // testing...

    private void test(String resource) throws IOException {
        InputStream ins = getResourceAsStream(resource);
        if (ins != null) {
            System.out.println(resource + ": " + ins.readAllBytes().length + " bytes");
            ins.close();
        } else {
            System.out.println(resource + ": not found");
        }
    }

    public static void main(String[] args) throws IOException {
        System.out.println("\nLoader for exported packages only...\n");
        ReadableModulesResourceLoader publicLoader = new ReadableModulesResourceLoader(
            ReadableModulesResourceLoader.class.getModule(),
            false // only include exported packages
        );
        publicLoader.test("java/lang/Object.class");
        publicLoader.test("sun/misc/Unsafe.class");
        publicLoader.test("jdk/internal/HotSpotIntrinsicCandidate.class");
        publicLoader.test("javax/swing/JFileChooser.class");

        System.out.println("\nLoader for all packages...\n");
        ReadableModulesResourceLoader privateLoader = new ReadableModulesResourceLoader(
            ReadableModulesResourceLoader.class.getModule(),
            true // include all packages of readable modules
        );
        privateLoader.test("java/lang/Object.class");
        privateLoader.test("sun/misc/Unsafe.class");
        privateLoader.test("jdk/internal/HotSpotIntrinsicCandidate.class");
        privateLoader.test("javax/swing/JFileChooser.class");
    }
}

If I compile this and run it as a module that does not "require" any modules (just java.base implicitly), I get the following output:

Loader for exported packages only...

java/lang/Object.class: 1640 bytes
sun/misc/Unsafe.class: 13093 bytes
jdk/internal/HotSpotIntrinsicCandidate.class: not found
javax/swing/JFileChooser.class: not found

Loader for all packages...

java/lang/Object.class: 1640 bytes
sun/misc/Unsafe.class: 13093 bytes
jdk/internal/HotSpotIntrinsicCandidate.class: 435 bytes
javax/swing/JFileChooser.class: not found

JFileChooser is in java.desktop module, so it is not "readable" by the module of this app. If I compile and run this code on the classpath (as an unnamed module), I get this:

Loader for exported packages only...

java/lang/Object.class: 1640 bytes
sun/misc/Unsafe.class: 13093 bytes
jdk/internal/HotSpotIntrinsicCandidate.class: not found
javax/swing/JFileChooser.class: 22119 bytes

Loader for all packages...

java/lang/Object.class: 1640 bytes
sun/misc/Unsafe.class: 13093 bytes
jdk/internal/HotSpotIntrinsicCandidate.class: 435 bytes
javax/swing/JFileChooser.class: 22119 bytes

The unnamed module (classpath classes) reads all modules by default.

So this is probably a way to get to the bytecodes of classes. To find out whether they are part of runtime or any other library, you can test whether the Module#getClassLoader is either bootstrap (== null) or extension (== ClassLoader.getClassLoader().getParent()). The Module you ask is the module used for loading the resource.

[uschindler]

In #95, I implemented a new code that checks the module name. So this is solved after this is committed.

http://openjdk.java.net/projects/jigsaw/spec/sotms/ states: "The remaining platform modules will share the “java.” name prefix and are likely to include, e.g., java.sql for database connectivity, java.xml for XML processing, and java.logging for logging. Modules that are not defined in the Java SE 9 Platform Specification but instead specific to the JDK will, by convention, share the “jdk.” name prefix."

The code there checks for those prefixes. The module name comes from the Class#getModule() API (Jigsaw) or from the jrt:/module/class-URL.

[uschindler]

Resolved through commit of #95.