pants with java 9/10 fails with exception failed: u'sun.boot.class.path'.

[justinas-dabravolskas]

Building pants/zinc on java 9 and 10 fails with exception failed: u'sun.boot.class.path'.

System Software Overview:
System Version: macOS 10.13.3 (17D102)
Kernel Version: Darwin 17.4.0

Tested tags: 1.5.0, 1.6.0rc0 and master

Tested jvms:
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

Java(TM) SE Runtime Environment 18.3 (build 10+46)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode)

Output of ./pants test zinc:

16:33:10 00:00 [main]
(To run a reporting server: ./pants server)
16:33:10 00:00 [setup]
16:33:11 00:01 [parse]
Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> jvm-platform-validate -> resolve -> resources -> compile -> pyprep -> test
16:33:11 00:01 [bootstrap]
16:33:11 00:01 [substitute-aliased-targets]
16:33:11 00:01 [jar-dependency-management]
16:33:11 00:01 [bootstrap-jvm-tools]
16:33:11 00:01 [provide-tools-jar]
16:33:11 00:01 [imports]
16:33:11 00:01 [ivy-imports]
16:33:11 00:01 [unpack-jars]
16:33:11 00:01 [unpack-jars]
16:33:11 00:01 [deferred-sources]
16:33:11 00:01 [deferred-sources]
16:33:11 00:01 [gen]
16:33:11 00:01 [antlr-java]
16:33:11 00:01 [antlr-py]
16:33:11 00:01 [jaxb]
16:33:11 00:01 [protoc]
16:33:11 00:01 [ragel]
16:33:11 00:01 [thrift-java]
16:33:11 00:01 [thrift-py]
16:33:11 00:01 [wire]
16:33:11 00:01 [avro-java]
16:33:11 00:01 [go-thrift]
16:33:11 00:01 [jax-ws]
16:33:11 00:01 [scrooge]
16:33:11 00:01 [thrifty]
16:33:11 00:01 [jvm-platform-validate]
16:33:11 00:01 [jvm-platform-validate]
16:33:11 00:01 [resolve]
16:33:11 00:01 [ivy]
16:33:12 00:02 [coursier]
16:33:12 00:02 [go]
16:33:12 00:02 [scala-js-compile]
16:33:12 00:02 [scala-js-link]
16:33:12 00:02 [node]
16:33:12 00:02 [resources]
16:33:12 00:02 [prepare]
16:33:12 00:02 [services]
16:33:12 00:02 [compile]
16:33:12 00:02 [node]
16:33:12 00:02 [compile-jvm-prep-command]
16:33:12 00:02 [jvm_prep_command]
16:33:12 00:02 [compile-prep-command]
16:33:12 00:02 [compile]
16:33:12 00:02 [zinc]
Invalidated 6 targets.
16:33:12 00:02 [isolation-zinc-pool-bootstrap]
[1/6] Compiling 1 zinc source in 1 target (src/scala/org/pantsbuild/zinc/util:util).
[2/6] Compiling 2 zinc sources in 1 target (src/scala/org/pantsbuild/zinc/options:options).
[3/6] Compiling 2 zinc sources in 1 target (src/scala/org/pantsbuild/zinc/cache:cache).
16:33:12 00:02 [compile]

16:33:12 00:02 [compile]

16:33:12 00:02 [compile]

16:33:12 00:02 [cache]
No cached artifacts for 1 target.
Invalidated 1 target.
16:33:12 00:02 [cache].
16:33:12 00:02 [cache]
No cached artifacts for 1 target.
Invalidated 1 target.
16:33:12 00:02 [cache].
compile(src/scala/org/pantsbuild/zinc/util:util) failed: u'sun.boot.class.path'
16:33:12 00:02 [cache]
No cached artifacts for 1 target.
Invalidated 1 target.
16:33:12 00:02 [cache].
compile(src/scala/org/pantsbuild/zinc/options:options) failed: u'sun.boot.class.path'
compile(src/scala/org/pantsbuild/zinc/cache:cache) failed: u'sun.boot.class.path'
FAILURE: Compilation failure: Failed jobs: compile(src/scala/org/pantsbuild/zinc/cache:cache), compile(src/scala/org/pantsbuild/zinc/util:util), compile(src/scala/org/pantsbuild/zinc/options:options)

           Waiting for background workers to finish.

16:33:12 00:02 [complete]
FAILURE

[baroquebobcat]

So. I think what's going on here is that java 9 removed the sun.boot.class.path system property. And Pants is loading the value of that property in a couple places.

• pants/src/python/pants/backend/jvm/tasks/jvm_dependency_analyzer.py

  Lines 138 to 141 in a9d3e19

  	# Note: assumes HotSpot, or some JVM that supports sun.boot.class.path.
  	# TODO: Support other JVMs? Not clear if there's a standard way to do so.
  	# May include loose classes dirs.
  	boot_classpath = get_path('sun.boot.class.path')

• pants/src/python/pants/backend/jvm/subsystems/shader.py

  Line 354 in a9d3e19

  boot_classpath = distribution.system_properties['sun.boot.class.path']

I think the jvm_dependency_analyzer.py location is probably the one that's blowing up in this path. It's surprising to me that you aren't getting a stacktrace. I'm surprised because I think you are running in the pants repo, and the config here is set to print them.

[justinas-dabravolskas]

Does it mean pants aren't compatible with java >= 9 projects?

[baroquebobcat]

I think so. I don't know exactly how much work it would be to add support but there are a few related issues.

For example,
#5080
#4288

Some of these have had work towards fixing them.

[benjyw]

FWIW I have Pants running the Java9 compiler successfully, producing Java9 classfiles, on a java 8 JVM.

[baroquebobcat]

Cool!

[stuhood]

I've recently begun rebasing #4729, which should unblock this from a zinc perspective... but I might not be able to finish it until closer to the end of this month. @cheister might also be able to open source a "zincless" java compilation task that he came up with. cc @youngderekm

[justinas-dabravolskas]

I've tried pants on jdk8 and zinc on jdk9 by passing -java-home. Indeed it works but has some side effects.

1. Zinc forks javac for every compilation, this eliminates benefits of nailgun and makes compilation very slow (I would estimate 10x slowdown - 1.5 hour for our project). Maybe it's worth starting nailgun with a different jdk instead?
2. There is a warning for each file : [warn] Error reading API from class file : java.lang.UnsupportedClassVersionError: com/r9/nucleus/io/ArrayViewPool has been compiled by a more recent version of the Java Runtime (class file version 53.0), this version of the Java Runtime only recognizes class file versions up to 52.0 Interesting what gets broken by it?

[stuhood]

Assuming this is a pure Java project, a short term workaround is coming in the form of #5743, and a longer term workaround is coming in #4729.

[justinas-dabravolskas]

#5743 spawns javac without nailgun? From performance perspective it will be similar to passing -java-home to zinc as @benjyw suggested. So we are still waiting for #4729