Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remote MacOS builds and classpath issues #13492

Closed
thna123459 opened this issue May 19, 2021 · 5 comments
Closed

Remote MacOS builds and classpath issues #13492

thna123459 opened this issue May 19, 2021 · 5 comments
Labels
P2 We'll consider working on this in future. (Assignee optional) platform: apple stale Issues or PRs that are stale (no activity for 30 days) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@thna123459
Copy link

Description of the problem:

Observed frequently on MacOS when building a project with a few libraries:

ERROR: /private/tmp/benchmark/java/src/test/java/com/orga/lib59/BUILD.bazel:2:10: Building src/test/java/com/orga/lib59/Class0Test.jar (1 source file) failed: (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_thna/07706e3b761d28de6140eeb7b91560d5/execroot/__main__ && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/private/tmp/benchmark \
  external/remotejdk11_macos/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--patch-module=java.compiler=external/remote_java_tools_darwin/java_tools/java_compiler.jar' '--patch-module=jdk.compiler=external/remote_java_tools_darwin/java_tools/jdk_compiler.jar' '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' -jar external/remote_java_tools_darwin/java_tools/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/test/java/com/orga/lib59/Class0Test.jar-0.params @bazel-out/darwin-fastbuild/bin/src/test/java/com/orga/lib59/Class0Test.jar-1.params)
Execution platform: @local_config_platform//:host
compiler message file broken: key=compiler.misc.msg.bug arguments=11.0.6, {1}, {2}, {3}, {4}, {5}, {6}, {7}
java.lang.NoClassDefFoundError: Could not initialize class sun.util.calendar.ZoneInfoFile
	at java.base/sun.util.calendar.ZoneInfo.getTimeZone(ZoneInfo.java:588)
	at java.base/java.util.TimeZone.getTimeZone(TimeZone.java:577)
	at java.base/java.util.TimeZone.setDefaultZone(TimeZone.java:682)
	at java.base/java.util.TimeZone.getDefaultRef(TimeZone.java:653)
	at java.base/java.util.TimeZone.getDefault(TimeZone.java:642)
	at java.base/java.time.ZoneId.systemDefault(ZoneId.java:272)
	at jdk.zipfs/jdk.nio.zipfs.ZipUtils.dosToJavaTime(ZipUtils.java:120)
	at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.cen(ZipFileSystem.java:2075)
	at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:2062)
	at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getEntry(ZipFileSystem.java:1347)
	at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:334)
	at jdk.zipfs/jdk.nio.zipfs.ZipPath.getAttributes(ZipPath.java:740)
	at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:292)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
	at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2760)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager$ArchiveContainer.<init>(JavacFileManager.java:520)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.getContainer(JavacFileManager.java:316)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:712)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.list(ClassFinder.java:734)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.scanPlatformPath(ClassFinder.java:696)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:547)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:299)
	at jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:642)
	at jdk.compiler/com.sun.tools.javac.code.Symbol$PackageSymbol.members(Symbol.java:1131)
	at jdk.compiler/com.sun.tools.javac.code.Symtab.listPackageModules(Symtab.java:832)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:345)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:529)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:286)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:301)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.complete(Enter.java:576)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.main(Enter.java:560)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:1066)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:937)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:104)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:147)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94)
	at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:123)
	at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.fallback(ReducedClasspathJavaLibraryBuilder.java:103)
	at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:65)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:110)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:97)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:77)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:61)

or

INFO: Analyzed 400 targets (6 packages loaded, 1530 targets configured).
INFO: Found 400 targets...
ERROR: /private/tmp/benchmark/java/src/main/java/com/orga/lib95/BUILD.bazel:4:13: Building src/main/java/com/orga/lib95/liblib95.jar (1 source file) failed: (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_thna/07706e3b761d28de6140eeb7b91560d5/execroot/__main__ && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/private/tmp/benchmark \
  external/remotejdk11_macos/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--patch-module=java.compiler=external/remote_java_tools_darwin/java_tools/java_compiler.jar' '--patch-module=jdk.compiler=external/remote_java_tools_darwin/java_tools/jdk_compiler.jar' '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' -jar external/remote_java_tools_darwin/java_tools/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/main/java/com/orga/lib95/liblib95.jar-0.params @bazel-out/darwin-fastbuild/bin/src/main/java/com/orga/lib95/liblib95.jar-1.params)
Execution platform: @local_config_platform//:host
Exception in thread "main" java.lang.UnsatisfiedLinkError: no zip in java.library.path: [/Users/user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2660)
	at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:829)
	at java.base/java.lang.System.loadLibrary(System.java:1870)
	at java.base/java.util.zip.ZipUtils.loadLibrary(ZipUtils.java:285)
	at java.base/java.util.zip.Deflater.<clinit>(Deflater.java:193)
	at java.base/java.util.zip.ZipOutputStream.<init>(ZipOutputStream.java:134)
	at java.base/java.util.zip.ZipOutputStream.<init>(ZipOutputStream.java:120)
	at java.base/java.util.jar.JarOutputStream.<init>(JarOutputStream.java:74)
	at com.google.devtools.build.buildjar.jarhelper.JarCreator.execute(JarCreator.java:241)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:154)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:120)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:97)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:77)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:61)
INFO: Elapsed time: 81.219s, Critical Path: 72.82s
INFO: 253 processes: 76 remote cache hit, 51 internal, 126 remote.
FAILED: Build did NOT complete successfully
       81.31 real         0.02 user         0.03 sys

or even:

ERROR: /private/tmp/benchmark/java/src/main/java/com/orga/lib31/BUILD.bazel:4:13: Building src/main/java/com/orga/lib31/liblib31.jar (1 source file) failed: (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_thna/07706e3b761d28de6140eeb7b91560d5/execroot/__main__ && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/private/tmp/benchmark \
  external/remotejdk11_macos/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--patch-module=java.compiler=external/remote_java_tools_darwin/java_tools/java_compiler.jar' '--patch-module=jdk.compiler=external/remote_java_tools_darwin/java_tools/jdk_compiler.jar' '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' -jar external/remote_java_tools_darwin/java_tools/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/main/java/com/orga/lib31/liblib31.jar-0.params @bazel-out/darwin-fastbuild/bin/src/main/java/com/orga/lib31/liblib31.jar-1.params)
Execution platform: @local_config_platform//:host
error: An unhandled exception was thrown by the Error Prone static analysis plugin.
     Please report this at https://github.com/google/error-prone/issues/new and include the following:

     error-prone version: 2.4.0
     BugPattern: (see stack trace)
     Stack Trace:
     java.lang.NoClassDefFoundError: Could not initialize class com.google.errorprone.bugpatterns.time.TemporalAccessorGetChronoField
  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  	at com.google.errorprone.scanner.ScannerSupplierImpl.instantiateChecker(ScannerSupplierImpl.java:84)
  	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
  	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
  	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
  	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
  	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
  	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
  	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
  	at com.google.errorprone.scanner.ScannerSupplierImpl.get(ScannerSupplierImpl.java:102)
  	at com.google.errorprone.scanner.ScannerSupplierImpl.get(ScannerSupplierImpl.java:40)
  	at com.google.errorprone.ErrorProneAnalyzer.lambda$scansPlugins$0(ErrorProneAnalyzer.java:79)
  	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167)
  	at com.google.errorprone.ErrorProneAnalyzer.finished(ErrorProneAnalyzer.java:152)
  	at com.google.devtools.build.buildjar.javac.plugins.errorprone.ErrorPronePlugin.postFlow(ErrorPronePlugin.java:161)
  	at com.google.devtools.build.buildjar.javac.BlazeJavaCompiler.flow(BlazeJavaCompiler.java:115)
  	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.flow(JavaCompiler.java:1365)
  	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:960)
  	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:104)
  	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:147)
  	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100)
  	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94)
  	at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:123)
  	at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.fallback(ReducedClasspathJavaLibraryBuilder.java:103)
  	at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:65)
  	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:110)
  	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118)
  	at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:97)
  	at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:77)
  	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:61)

Errors only occur when passing the remote executor flag. There are no errors on the remote side (buildfarm logs). It is not clear if the error occurs from the local bazel process or from the remote process (how to determine that?).

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

A test project is attached, run a remote build with java libraries, use bazel build --remote_executor=grpc://host:port //... with 50 jobs.

bazel-java-remote-issue.tar.gz

What operating system are you running Bazel on?

MacOS (20.3.0 Darwin Kernel Version 20.3.0, also observed on 19.4.0 Darwin Kernel Version 19.4.0)

What's the output of bazel info release?

release 4.1.0rc4, but 4.0.0 also exhibits errors.

Have you found anything relevant by searching the web?

The closest issue I could find is #13484 but it is for Windows.
@jin jin added platform: apple team-Remote-Exec Issues and PRs for the Execution (Remote) team untriaged labels May 24, 2021
@thna123459
Copy link
Author

As we understand it, the MacOS remote builds are not hermetic at this point. Our current theory is that library files are resolved from their cached paths. Since folders are continuously added and removed during a parallel remote build, and since the buildfarm worker process links files in order to re-use/cache them, some java processes eventually attempt to resolve dynamic libraries from deleted folders.

@thna123459
Copy link
Author

The same builds succeed without a problem when a native java version is in use:

--define=ABSOLUTE_JAVABASE=/Library/Java/JavaVirtualMachines/java-11.jdk/Contents/Home/
--javabase=@bazel_tools//tools/jdk:absolute_javabase
--host_javabase=@bazel_tools//tools/jdk:absolute_javabase
--host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
--java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla

@coeuvre
Copy link
Member

coeuvre commented Apr 8, 2022

We don't have mac remote workers to produce this. But there are some significant changes regarding to toolchains since 5.0. Can you try with newer Bazel?

@coeuvre coeuvre added type: bug P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Apr 8, 2022
@github-actions
Copy link

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

@github-actions github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label Jun 13, 2023
@github-actions
Copy link

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P2 We'll consider working on this in future. (Assignee optional) platform: apple stale Issues or PRs that are stale (no activity for 30 days) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jin @coeuvre @thna123459