bazel/remote-execution is broken

[aehlig]

The bazel CI for remote execution is broken[1] since 452141f.
The error message reads

Executing tests from //src/test/java/com/google/devtools/build/lib:sandbox-tests
-----------------------------------------------------------------------------
/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/1082dd75df5fc6b2f01ea01672d6d73c/execroot/io_bazel/bazel-out/k8-fastbuild/bin/src/test/java/com/google/devtools/build/lib/sandbox-tests.runfiles/io_bazel/src/test/java/com/google/devtools/build/lib/sandbox-tests: line 353: /usr/lib/jvm/java-8-openjdk-amd64/bin/java: No such file or directory

[1] https://buildkite.com/bazel/remote-execution/builds/2101#cc63b8e7-bf87-4dcf-9093-5d7d932cb54a

[aehlig]

CC @tomlu @laurentlb from the duplicate issue

[ulfjack]

I have no idea. Maybe an issue with the Buildkite setup? Why are these tests failing, but none of the others? Does this have to do with the Jdk9 migration?

[tomlu]

Yeah, how on Earth can a rollback of a Java rule change break remote execution? Surely the exact commit is a red herring, and this has more to do with some change in the CI enviroment?

[tomlu]

I know bazel on the CI machines was upgraded on the same day to 0.14 around the exact time that this rollback went in, maybe that is related

[fweikert]

Timeline on Saturday, June 2nd:

6:00pm CEST: Last green build starts [1]
6:38pm CEST: New image with Bazel 0.14.0 gets pushed
7:00pm CEST: First broken build starts [2]

Note that the broken build references an old toolchain from the previous release:
--crosstool_top=@bazel_toolchains//configs/ubuntu16_04_clang/1.0/bazel_0.13.0/default:toolchain

[1] https://buildkite.com/bazel/remote-execution/builds/2100
[2] https://buildkite.com/bazel/remote-execution/builds/2101

[fweikert]

#5376 for root cause analysis