Skip to content

Commit

Permalink
[7.0.0] Fix Java runtime toolchain resolution in cross-compilation sc…
Browse files Browse the repository at this point in the history
…enarios (#19900)

The Java rules used toolchains of type
`@bazel_tools//tools/jdk:runtime_toolchain_type` for two different
purposes requiring different exec/target platform constraints, which led
to issues when cross-compiling: The runtime added to the runfiles of an
executable Java target has to have constraints on the target platform,
whereas the runtime used to extract the bootclasspath should not have
constraints on the target platform. As a result:
1. `@local_jdk` did not define any target constraints for its runtime,
which allowed the runtime to provide the bootclasspath required by
Android rules, but could also end up building `java_binary` targets that
wouldn't run on the target (see #18265).
2. Remote JDKs defined a target constraint for the their runtimes, which
prevented them from being added to the runfiles of targets they couldn't
run on, but broke Android compilation due to the failure to resolve a
runtime for bootclasspath extraction (see #17085).

This is resolved by adding a third toolchain type,
`bootstrap_runtime_toolchain_type`, that is only used by the
`bootclasspath` rule (realizes
#17085 (comment)).
Detailed explanations of the different types and their intended
constraints have been added to `@bazel_tools//tools/jdk:BUILD`.

Addressing the cross-compilation errors then required the following
changes:
1. Direct dependencies on the Java runtime have been removed from rules
that do not need them (e.g. `android_library` and `java_binary` with
`create_executable = False`).
3. Implicit dependencies on the Java runtime through the `bootclasspath`
rule have been replaced with dependencies on the bootstrap runtime.
4. `{local,remote}_java_repository` now use the user-provided exec
constraints of the Java toolchain as the target constraints of the
associated runtime.
5. `@local_jdk` uses the auto-detected host constraints as exec
constraints for the local Java toolchain.

Fixes #17085
Fixes #18265
Fixes bazelbuild/rules_java#64

RELNOTES[INC]: Java runtime toolchains created via
`local_java_repository` from
`@bazel_tools//tools/jdk:local_java_repository.bzl`, which includes
`local_jdk`, now have `target_compatible_with` set to the auto-detected
host constraints. This can result in errors about toolchain resolution
failures for `@bazel_tools//tools/jdk:runtime_toolchain_type`,
especially when cross-compiling. These failures can be fixed in the
following ways (listed in decreasing order of preference):
* Replace `java_binary` targets that aren't meant to be run with `bazel
run` or as tools during the build with `java_single_jar` (available in
`@rules_java//java:java_single_jar.bzl`). Such targets do not require a
Java runtime for the target configuration.
* Set `--java_runtime_version=remotejdk_N` for some Java version `N` to
let Bazel choose and download an appropriate remote JDK for the current
target platform. This setting defaults to `local_jdk`, which means that
Bazel can only use the local JDK, which isn't compatible with any other
platform.
* Manually define and register a `local_java_runtime` with no value set
for `exec_compatible_with` (defaults to `[]`) and select it by setting
`--java_runtime_version` to its `name`. This fully restores the previous
behavior, but can result in incorrect results when cross-compiling (see
#18265).

Closes #18262.

Commit
f79ca02

PiperOrigin-RevId: 574914446
Change-Id: I6cbfb7ffa2fbfd62e5f6fb49532b36be658dfa40

Co-authored-by: Fabian Meumertzheim <[email protected]>
  • Loading branch information
iancha1992 and fmeum authored Oct 19, 2023
1 parent 9e8ae1a commit 2cb3159
Show file tree
Hide file tree
Showing 17 changed files with 761 additions and 593 deletions.
6 changes: 6 additions & 0 deletions .bazelrc
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ build:remote_shared --tool_java_runtime_version=rbe_jdk
build:remote_shared --noexperimental_check_desugar_deps

# Configuration to build and test Bazel on RBE on Ubuntu 18.04 with Java 11
# Workaround for https://github.com/bazelbuild/bazel/issues/19837.
# TODO(bazel-team): Remove this toolchain when .bazelversion is 7.0.0rc2 or later.
build:ubuntu2004_java11 --extra_toolchains=//:bazel_rbe_java_toolchain_definition
build:ubuntu2004_java11 --extra_toolchains=@rbe_ubuntu2004_java11//java:all
build:ubuntu2004_java11 --crosstool_top=@rbe_ubuntu2004_java11//cc:toolchain
build:ubuntu2004_java11 --extra_toolchains=@rbe_ubuntu2004_java11//config:cc-toolchain
Expand All @@ -34,6 +37,9 @@ build:windows --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolch

build:windows_arm64 --platforms=//:windows_arm64
build:windows_arm64 --extra_toolchains=@local_config_cc//:cc-toolchain-arm64_windows
# When cross-compiling, the local Java runtime, which is used by default, will not be compatible
# with the target.
build:windows_arm64 --java_runtime_version=remotejdk_11

# Enable Bzlmod
common:bzlmod --enable_bzlmod
Expand Down
17 changes: 17 additions & 0 deletions BUILD
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Bazel - Google's Build System

load("@bazel_skylib//rules:write_file.bzl", "write_file")
load("@rules_java//toolchains:default_java_toolchain.bzl", "default_java_toolchain")
load("@rules_license//rules:license.bzl", "license")
load("@rules_pkg//pkg:tar.bzl", "pkg_tar")
load("@rules_python//python:defs.bzl", "py_binary")
Expand Down Expand Up @@ -293,3 +294,19 @@ REMOTE_PLATFORMS = ("rbe_ubuntu2004_java11",)
)
for platform_name in REMOTE_PLATFORMS
]

# Workaround for https://github.com/bazelbuild/bazel/issues/19837.
# TODO(bazel-team): Remove these two targets when .bazelversion is 7.0.0rc2 or later.
default_java_toolchain(
name = "bazel_java_toolchain",
bootclasspath = ["@rules_java//toolchains:platformclasspath"],
source_version = "11",
tags = ["manual"],
target_version = "11",
)

default_java_toolchain(
name = "bazel_rbe_java_toolchain",
source_version = "11",
target_version = "11",
)
14 changes: 9 additions & 5 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ bazel_dep(name = "zstd-jni", version = "1.5.2-3.bcr.1")
bazel_dep(name = "blake3", version = "1.3.3.bcr.1")
bazel_dep(name = "zlib", version = "1.3")
bazel_dep(name = "rules_cc", version = "0.0.9")
bazel_dep(name = "rules_java", version = "6.3.1")
bazel_dep(name = "rules_java", version = "7.0.6")
bazel_dep(name = "rules_proto", version = "5.3.0-21.7")
bazel_dep(name = "rules_jvm_external", version = "5.2")
bazel_dep(name = "rules_python", version = "0.24.0")
Expand Down Expand Up @@ -231,10 +231,10 @@ use_repo(
"remotejdk17_macos_aarch64",
"remotejdk17_win",
"remotejdk17_win_arm64",
"remotejdk20_linux",
"remotejdk20_macos",
"remotejdk20_macos_aarch64",
"remotejdk20_win",
"remotejdk21_linux",
"remotejdk21_macos",
"remotejdk21_macos_aarch64",
"remotejdk21_win",
)

# =========================================
Expand Down Expand Up @@ -309,6 +309,10 @@ register_toolchains("@local_config_winsdk//:all")

register_toolchains("//src/main/res:empty_rc_toolchain")

# Workaround for https://github.com/bazelbuild/bazel/issues/19837.
# TODO(bazel-team): Remove when .bazelversion is 7.0.0rc2 or later.
register_toolchains("//:bazel_java_toolchain_definition")

# =========================================
# Android tools dependencies
# =========================================
Expand Down
572 changes: 287 additions & 285 deletions MODULE.bazel.lock

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions distdir_deps.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -106,17 +106,17 @@ DIST_DEPS = {
"rules_java_builtin",
"rules_java_builtin_for_testing",
],
"archive": "rules_java-6.3.1.tar.gz",
"sha256": "117a1227cdaf813a20a1bba78a9f2d8fb30841000c33e2f2d2a640bd224c9282",
"urls": ["https://github.com/bazelbuild/rules_java/releases/download/6.3.1/rules_java-6.3.1.tar.gz"],
"archive": "rules_java-7.0.6.tar.gz",
"sha256": "e81e9deaae0d9d99ef3dd5f6c1b32338447fe16d5564155531ea4eb7ef38854b",
"urls": ["https://github.com/bazelbuild/rules_java/releases/download/7.0.6/rules_java-7.0.6.tar.gz"],
"workspace_file_content": "",
"used_in": [
"additional_distfiles",
],
"license_kinds": [
"@rules_license//licenses/spdx:Apache-2.0",
],
"package_version": "6.1.1",
"package_version": "7.0.6",
},
# Used in src/test/java/com/google/devtools/build/lib/blackbox/framework/blackbox.WORKSAPCE
"rules_proto": {
Expand Down
2 changes: 1 addition & 1 deletion src/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -585,7 +585,7 @@ filegroup(
"@rules_testing//:LICENSE",
] + [
"@remotejdk%s_%s//:WORKSPACE" % (version, os)
for version in ("17", "20")
for version in ("17", "21")
for os in ("macos", "macos_aarch64", "linux", "win")
] + ["@bazel_tools_repo_cache//:files"],
)
3 changes: 2 additions & 1 deletion src/MODULE.tools
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# NOTE: When editing this file, also update the lockfile.
# bazel run //src/test/tools/bzlmod:update_default_lock_file
# bazel mod deps --lockfile_mode=update

module(name = "bazel_tools")

bazel_dep(name = "rules_cc", version = "0.0.9")
bazel_dep(name = "rules_java", version = "6.3.1")
bazel_dep(name = "rules_java", version = "7.0.6")
bazel_dep(name = "rules_license", version = "0.0.3")
bazel_dep(name = "rules_proto", version = "4.0.0")
bazel_dep(name = "rules_python", version = "0.4.0")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ java_library(
),
"//conditions:default": ["NoAndroidSdkStubTest.java"],
}),
javacopts = ["-source 7 -target 7"], # we run this through dx for testing, so no lambdas!
javacopts = ["-source 8 -target 8"],
resources = ["testresource.txt"],
deps = [
"//src/test/java/com/google/devtools/build/lib/testutil",
Expand Down
10 changes: 5 additions & 5 deletions src/test/py/bazel/test_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,11 +89,11 @@ class TestBase(absltest.TestCase):
'remotejdk17_macos_aarch64',
'remotejdk17_win',
'remotejdk17_win_arm64',
'remotejdk20_linux',
'remotejdk20_macos',
'remotejdk20_macos_aarch64',
'remotejdk20_win',
'remotejdk20_win_arm64',
'remotejdk21_linux',
'remotejdk21_macos',
'remotejdk21_macos_aarch64',
'remotejdk21_win',
'remotejdk21_win_arm64',
'rules_cc',
'rules_java',
'rules_java_builtin_for_testing',
Expand Down
8 changes: 4 additions & 4 deletions src/test/shell/bazel/bazel_java17_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -173,11 +173,11 @@ EOF
bazel build //pkg:Main \
--extra_toolchains=//pkg:java_toolchain_definition \
--java_language_version=17 \
--java_runtime_version=remotejdk_20 \
--java_runtime_version=remotejdk_21 \
&>"${TEST_log}" && fail "Expected build to fail"

expect_log "error: \[BazelJavaConfiguration\] The Java 17 runtime used to run javac is not " \
"recent enough to compile for the Java 20 runtime in external/remotejdk20_[a-z0-9]*\. Either " \
"recent enough to compile for the Java 21 runtime in external/remotejdk21_[a-z0-9]*\. Either " \
"register a Java toolchain with a newer java_runtime or specify a lower " \
"--java_runtime_version\."
}
Expand Down Expand Up @@ -220,11 +220,11 @@ EOF
bazel build //pkg:gen \
--extra_toolchains=//pkg:java_toolchain_definition \
--tool_java_language_version=17 \
--tool_java_runtime_version=remotejdk_20 \
--tool_java_runtime_version=remotejdk_21 \
&>"${TEST_log}" && fail "Expected build to fail"

expect_log "error: \[BazelJavaConfiguration\] The Java 17 runtime used to run javac is not " \
"recent enough to compile for the Java 20 runtime in external/remotejdk20_[a-z0-9]*\. Either " \
"recent enough to compile for the Java 21 runtime in external/remotejdk21_[a-z0-9]*\. Either " \
"register a Java toolchain with a newer java_runtime or specify a lower " \
"--tool_java_runtime_version\."
}
Expand Down
131 changes: 131 additions & 0 deletions src/test/shell/bazel/bazel_java_test_defaults.sh
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,52 @@ EOF
//pkg:foo_deploy.jar &>"$TEST_log" || fail "Build should succeed"
}

function test_java_library_compiles_for_any_platform_with_local_jdk() {
mkdir -p pkg
cat > pkg/BUILD.bazel <<'EOF'
platform(name = "exotic_platform")
java_library(
name = "foo",
srcs = ["Foo.java"],
)
EOF

cat > pkg/Foo.java <<'EOF'
package com.example;
public class Foo {
public static void main(String[] args) {
System.out.println("Hello World!");
}
}
EOF

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=local_jdk \
//pkg:foo &>"$TEST_log" || fail "Build should succeed"
}

function test_java_library_compiles_for_any_platform_with_remote_jdk() {
mkdir -p pkg
cat > pkg/BUILD.bazel <<'EOF'
platform(name = "exotic_platform")
java_library(
name = "foo",
srcs = ["Foo.java"],
)
EOF

cat > pkg/Foo.java <<'EOF'
package com.example;
public class Foo {
public static void main(String[] args) {
System.out.println("Hello World!");
}
}
EOF

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=remotejdk_11 \
//pkg:foo &>"$TEST_log" || fail "Build should succeed"
}

function test_non_executable_java_binary_compiles_for_any_platform_with_local_jdk() {
mkdir -p pkg
cat > pkg/BUILD.bazel <<'EOF'
Expand All @@ -332,4 +378,89 @@ EOF
//pkg:foo_deploy.jar &>"$TEST_log" || fail "Build should succeed"
}

function test_non_executable_java_binary_compiles_for_any_platform_with_remote_jdk() {
mkdir -p pkg
cat > pkg/BUILD.bazel <<'EOF'
platform(name = "exotic_platform")
java_binary(
name = "foo",
srcs = ["Foo.java"],
create_executable = False,
)
EOF

cat > pkg/Foo.java <<'EOF'
package com.example;
public class Foo {
public static void main(String[] args) {
System.out.println("Hello World!");
}
}
EOF

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=remotejdk_11 \
//pkg:foo &>"$TEST_log" || fail "Build should succeed"

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=remotejdk_11 \
//pkg:foo_deploy.jar &>"$TEST_log" || fail "Build should succeed"
}

function test_executable_java_binary_fails_without_runtime_with_local_jdk() {
mkdir -p pkg
cat > pkg/BUILD.bazel <<'EOF'
platform(name = "exotic_platform")
java_binary(
name = "foo",
srcs = ["Foo.java"],
main_class = "com.example.Foo",
)
EOF

cat > pkg/Foo.java <<'EOF'
package com.example;
public class Foo {
public static void main(String[] args) {
System.out.println("Hello World!");
}
}
EOF

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=local_jdk \
//pkg:foo &>"$TEST_log" && fail "Build should fail"
expect_log "While resolving toolchains for target //pkg:foo ([0-9a-f]*): No matching toolchains found for types @bazel_tools//tools/jdk:runtime_toolchain_type"

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=local_jdk \
//pkg:foo_deploy.jar &>"$TEST_log" && fail "Build should fail"
expect_log "While resolving toolchains for target //pkg:foo_deployjars_internal_rule ([0-9a-f]*): No matching toolchains found for types @bazel_tools//tools/jdk:runtime_toolchain_type"
}

function test_executable_java_binary_fails_without_runtime_with_remote_jdk() {
mkdir -p pkg
cat > pkg/BUILD.bazel <<'EOF'
platform(name = "exotic_platform")
java_binary(
name = "foo",
srcs = ["Foo.java"],
main_class = "com.example.Foo",
)
EOF

cat > pkg/Foo.java <<'EOF'
package com.example;
public class Foo {
public static void main(String[] args) {
System.out.println("Hello World!");
}
}
EOF

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=remotejdk_11 \
//pkg:foo &>"$TEST_log" && fail "Build should fail"
expect_log "While resolving toolchains for target //pkg:foo ([0-9a-f]*): No matching toolchains found for types @bazel_tools//tools/jdk:runtime_toolchain_type"

bazel build --platforms=//pkg:exotic_platform --java_runtime_version=remotejdk_11 \
//pkg:foo_deploy.jar &>"$TEST_log" && fail "Build should fail"
expect_log "While resolving toolchains for target //pkg:foo_deployjars_internal_rule ([0-9a-f]*): No matching toolchains found for types @bazel_tools//tools/jdk:runtime_toolchain_type"
}

run_suite "Java toolchains tests, configured using flags or the default_java_toolchain macro."
3 changes: 2 additions & 1 deletion src/test/shell/bazel/bazel_with_jdk_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,10 @@ function set_up() {

mkdir -p java/main
cat >java/main/BUILD <<EOF
java_library(
java_binary(
name = 'JavaExample',
srcs = ['JavaExample.java'],
main_class = 'JavaExample',
)
EOF

Expand Down
Loading

0 comments on commit 2cb3159

Please sign in to comment.