Remove annotation-only Jars from runtime classpath (in Gradle)

[jjohannes]

Proposal to remove the following Jars from the runtime classpath (in Gradle projects):

• checker-qual-*.jar
• error_prone_annotations-*.jar
• jsr305-*.jar

Afterwards, they are still transitively visible on the compile classpath, which can not be expressed in POMs, but in Gradle Metadata which is published starting with the next Guava release (33). In a Gradle build setup, this would be "compileOnlyApi" scope. See also #2824 (comment) and the linked Maven feature request for the discussion about having such a "compileOnlyApi scope" in Maven.

Context:

• Original issue on this topic: Unnecessary-looking dependencies #2824
• It was considered to do this as part of Publish Gradle Module Metadata with Variants #3683 but not done as this might be a breaking change for some users.
• I think for most (almost all?) users these Jars are not needed at runtime because of Guava. If they are needed for other reasons, they should be declared directly by the users in any case. For rare cases where only the use of Guava requires these to be added (if someone has examples for such cases, please share) the user can still add them them as runtimeOnly. You could consider them as an "optional feature" that you have to add yourself if needed.

[OrangeDog]

which can not be expressed in POMs

<scope>provided</scope>?

[cpovirk]

Sadly, <scope>provided</scope> also removes the annotations at compile time, and that can lead to problems.

[jjohannes]

POM's <scope> can be a bit confusing, because it is used both for describing the dependencies of the local project to build with the Maven Tool and for metadata in repositories. Depending on which usage of POM you look at, the <scope> has different meaning.

At build time of the local project, all scopes may have some meaning for the Maven Tool.

In the repository – POM as metadata – only these two have meaning:

• <scope>compile</scope> + <optional>false</optional>
• <scope>runtime</scope> + <optional>false</optional>

Anything with another scope (or with optional=true) has no meaning in metadata. It is ignored by both Maven and Gradle when parsing the file as metadata of an already built dependency.

Another interesting tidbit is that the meaning of runtime is a bit different in the pulished metadata. You can use it there for dependencies that you needed to compile your own code, but want to hide on the compile classpath of consumers. Because at this point the code is already compiled, runtime is sufficient.

In Gradle, for the local project you have more scopes that influence the metadata. But only some of them map correctly to POM as metadata (which Gradle generates in addition to the Gradle Metadata for Maven consumers).

• api -> <scope>compile</scope>
• implementation -> <scope>runtime</scope>
• runtimeOnly -> <scope>runtime</scope> (same as above, once published it dosen't matter if it was visible locally during compilation or not)
• compileOnly -> nothing (only for local compilation, not visible transitively for compilation or runtime)
• compileOnlyApi -> this one is missing (so it maps to <scope>compile</scope>)

[ben-manes]

In the past compiler bugs like JDK-8152174, scala/bug#7751, and earlier meant putting annotations on the classpath was the safest option. Would this change allow for encountering those types of bugs again? (No preference, just backstory)

[jjohannes]

That's what this is about: not running into problems like #6606 (comment) again but still solving #2824.

• You want the annotation libraries at compile time to avoid compilation issues as the ones @ben-manes listed above.
• You do not want them at runtime (i.e. package them with your application) - that's what Unnecessary-looking dependencies #2824 is about.

For Gradle, we can solve this now: have dependencies at compile time, but not at runtime. That's what this PR does for the annotation libraries.

[cpovirk]

https://github.com/google/guava/wiki/UseGuavaInYourBuild#what-about-guavas-own-dependencies mentions at least 2 runtime problems, one of which is still probably a concern for one of our deps:

• JDK-8152174 [edit: cited by Ben above, sorry], fixed in JDK9, but I'd assume that we have enough JDK8 users for someone to be affected.
• JDK-8247797, still present but maybe not actually a problem in all the situations it claims, including in Guava?

JDK-8152174 is potential trouble, albeit only for checker-qual. (That said, checker-qual is the largest dependency, so it's probably the one that users would most like to see us cut. The plan is still to move to jspecify once it's ready. That would help with the size.) Perhaps our next step should be to hide deps other than checker-qual.

JDK-8247797 is probably fine, which is interesting: Despite what its writeup suggests, I can successfully reflect on ParametricNullness, which is annotated with javax.annotation.Nonnull, including using an enum element with value javax.annotation.meta.When.UNKNOWN, even when jsr305 is not on the classpath:

$ cat GuavaReflectionNoDeps.java
import com.google.common.base.Function;
import java.lang.annotation.Annotation;
import java.lang.reflect.Method;

class GuavaReflectionNoDeps {
  public static void main(String[] args) {
    for (Method method : Function.class.getMethods()) {
      for (Annotation onFunction : method.getAnnotations()) {
        System.out.println(onFunction);
        for (Annotation onAnnotation : onFunction.annotationType().getAnnotations()) {
          System.out.println("  " + onAnnotation);
        }
      }
    }
  }
}

$ ( CP=$HOME/.m2/repository/com/google/guava/guava/32.1.0-jre/guava-32.1.0-jre.jar:$HOME/.m2/repository/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar; ~/openjdk-8u342/bin/javac -cp $CP GuavaReflectionNoDeps.java && ~/openjdk-8u342/bin/java -cp .:$CP GuavaReflectionNoDeps )
@com.google.common.base.ParametricNullness()
  @java.lang.annotation.Retention(value=RUNTIME)
  @java.lang.annotation.Target(value=[FIELD, METHOD, PARAMETER])

It looks like the problem actually surfaces only when Nonnull is on the classpath and When is not. That of course won't happen here, since both are part of the same jsr305 artifact: Either both will be present, or neither will:

$ cp ~/.m2/repository/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar /tmp

$ zip -d /tmp/jsr305-3.0.2.jar javax/annotation/meta/When.class
deleting: javax/annotation/meta/When.class

$ ( CP=$HOME/.m2/repository/com/google/guava/guava/32.1.0-jre/guava-32.1.0-jre.jar:$HOME/.m2/repository/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar; ~/openjdk-8u342/bin/javac -cp $CP GuavaReflectionNoDeps.java && ~/openjdk-8u342/bin/java -cp .:$CP:/tmp/jsr305-3.0.2.jar GuavaReflectionNoDeps )
@com.google.common.base.ParametricNullness()
Exception in thread "main" java.lang.NoClassDefFoundError: javax/annotation/meta/When
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.getDeclaredMethods(Class.java:1975)
        at sun.reflect.annotation.AnnotationType$1.run(AnnotationType.java:112)
        at sun.reflect.annotation.AnnotationType$1.run(AnnotationType.java:109)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.reflect.annotation.AnnotationType.<init>(AnnotationType.java:109)
        at sun.reflect.annotation.AnnotationType.getInstance(AnnotationType.java:85)
        at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:266)
        at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
        at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
        at java.lang.Class.createAnnotationData(Class.java:3521)
        at java.lang.Class.annotationData(Class.java:3510)
        at java.lang.Class.getAnnotations(Class.java:3446)
        at GuavaReflectionNoDeps.main(GuavaReflectionNoDeps.java:10)
Caused by: java.lang.ClassNotFoundException: javax.annotation.meta.When
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 15 more

[jjohannes]

Thanks for @cpovirk for clarifying that some issues are also about runtime. My understanding though is that you have to be in a very specific situation to get into such an issue. You need to use reflection on a class that uses these annotations. Which you could do an a Guava class. (But is that an intended use of Guava?)
And IIUC, it is only a bug in JDK8 that this may cause an problematic error, while in other cases there is no "real" problem.

I guess my take on this is, that if users do some of these things, it is most likely that they should directly declare dependencies on the annotation libraries they want to deal with and not rely on Guava brining them in transitively. I don't know "real" projects doing something like this, though. So my assumption here might be wrong.

I don't know how big the part of the user base is that still uses JDK8 and does something so specific that they run into the bug. It feels like it could be ok to document that you need to add a certain <runtime> / runtimeOnly dependency yourself (or move to a newer Java) if you run into it. (But maybe I am not conservative enough :))

We could also drive the variant setup further and have a separate dependency setup for JDK8 (that keeps one or all dependencies) and do the clean setup for everyone using JDK9+.

[cpovirk]

There seem to be frameworks out there that try to automatically (e.g.) serialize arbitrary objects by looking at their fields. That is of course not a great idea, and it's one of the reasons that the Java module system exists (and perhaps a reason for us to release a jar with a module-info someday). Maybe it's not a bad idea for us to "gently" break such users now in preparation for breaking them more firmly in the future.... And the results could be educational for us in general.

I would probably still start with omitting the other deps first, but I'm warming a little to trying this out for checker-qual, too.

We'll see if others have something to add. Given the flood of releases lately, we'll probably wait a little before publishing another one, so we have time to figure this out.

[vlsi]

You need to use reflection on a class that uses these annotations. Which you could do an a Guava class. (But is that an intended use of Guava?)

Well, type annotations add information to the type.
Do you think every reflection should be disabled unless explicitly permitted by the library authors?
If the reflection for public methods should be available, it should probably provide type information like "acceptable parameter type" and "result type", and it is important to know if the type is nullable or not.

For instance, a JSON parser might user object constructor when parsing JSON, and it might inspect if the parameters are @Nullable. If the parameter is non-nullable, and JSON misses the value, then the parser could produce a better error message than a mere "parameter X is null".

[OrangeDog]

@vlsi in such cases the tool depends on those annotation types itself. Or in the case of "nullable" just looks at the name and none of the details of the annotation.

[vlsi]

Or in the case of "nullable" just looks at the name and none of the details of the annotation

If the annotation class is not provided in the runtime, the reflection API does not return the annotation, so the tool can't even detect annotation was present in the first place.
That is like the "parameter type varies depending on the runtime classpath" issue. It is not the best issue to debug.

Of course, there's a backdoor like getResourceAsStream("ClassName.class") and then parsing the bytecode, however, it would sound strange. Of course, Java specification says Java must ignore annotations that are not present on the classpath, yet bugs like #6606 (comment) exist :-(

In other words, I believe it is fair to expect that "types used in the public signatures" (e.g. return types of the public APIs) should be present in "direct runtime dependencies".

At the same time, the annotations from error-prone do not sound like "type annotations", and they are much more like "compile-time-only" annotations, so having them as compileOnlyApi (==transitive compile dependencies) rather than compile (==transitive compile + runtime) probably makes sense.

[cpovirk]

(breadcrumb: If we do this, we'll want to watch out for GWT issues like #7134. That's not a big concern, and I remain more nervous about omitting dependencies for other reasons than for GWT reasons.)