[Bazel CI] Log error: 'no such package '@@rules_java//toolchains': BUILD file not found in directory 'toolchains' of external repository @@rules_java'

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4122#0191e992-eff8-4964-ae5d-195e882df5b8

Platform: Ubuntu

Logs:

ERROR: �[0mno such package '@@rules_java//toolchains': BUILD file not found in directory 'toolchains' of external repository @@rules_java. Add a BUILD file to a directory to mark it as a package.
ERROR: �[0m/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/66716e53d95df22d9f586c1d34099b28/external/remote_jdk8_linux_aarch64_toolchain_config_repo/BUILD.bazel:27:10: no such package '@@rules_java//toolchains': BUILD file not found in directory 'toolchains' of external repository @@rules_java. Add a BUILD file to a directory to mark it as a package. and referenced by '@@remote_jdk8_linux_aarch64_toolchain_config_repo//:bootstrap_runtime_toolchain'
ERROR: �[0m/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/66716e53d95df22d9f586c1d34099b28/external/bazel_tools/src/tools/launcher/BUILD:10:14: Analysis failed
ERROR: �[0mAnalysis of target '//src/app_charts/base:app_management_test' failed; build aborted: Analysis failed

Culprit:

Steps:

git clone https://github.com/googlecloudrobotics/core
git reset 495f81b04aeffe6e854921404f8b2e7081802235  --hard
export USE_BAZEL_VERSION=3f480ea26cfbadc04c701b9058100e37a5a24dc2
bazel build //... 

CC Greenteam @meteorcloudy

[meteorcloudy]

@ensonic Please take a look if you want to stay compatible with Bazel@HEAD, you may need to upgrade rules_java and ideally migrate to Bzlmod for external deps.

[ensonic]

We're not using rules_java anymore and I am currently in the process of trying to migrate to blzmod. Hopefully done by end of week.

[ensonic]

Not sure about the buildkite setup, but a test dependency is missing in the runfiles

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4134#01920879-54ba-4090-b81f-721132344aff

this test fails:
https://github.com/googlecloudrobotics/core/blame/main/src/app_charts/base/app_management_test.sh#L17

log:
https://storage.googleapis.com/bazel-untrusted-buildkite-artifacts/01920879-54ba-4090-b81f-721132344aff/src/app_charts/base/app_management_test/test.log

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //src/app_charts/base:app_management_test
-----------------------------------------------------------------------------
/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/86590d0664c879d867354ce7ad029d5b/sandbox/linux-sandbox/2734/execroot/_main/bazel-out/k8-fastbuild/bin/src/app_charts/base/app_management_test.runfiles/_main/src/app_charts/base/app_management_test: line 34: external/_main~non_module_deps~kubernetes_helm/helm: No such file or directory
TEST FAILED: "external/_main~non_module_deps~kubernetes_helm/helm template src/app_charts/base/base-cloud-0.0.1.tgz --set-string app_management=true" failed

but we have adjusted the new layout in the external folder and on our ci the test passes:
https://github.com/googlecloudrobotics/core/actions/runs/10934200983/job/30353889949#step:5:195

[ensonic]

Now with #439 referencing a sh_test data dependency via TESTSRC_DIR we get:
https://storage.googleapis.com/bazel-untrusted-buildkite-artifacts/01920d9f-0c23-4acd-8ebd-c0d5540d82dc/src/app_charts/base/app_management_test/test.log

/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/753c46cd17f260bf584a727a9c58ca06/sandbox/linux-sandbox/2734/execroot/_main/bazel-out/k8-fastbuild/bin/src/app_charts/base/app_management_test.runfiles/_main/src/app_charts/base/app_management_test: line 34: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/753c46cd17f260bf584a727a9c58ca06/sandbox/linux-sandbox/2734/execroot/_main/bazel-out/k8-fastbuild/bin/src/app_charts/base/app_management_test.runfiles/_main/external/_main~non_module_deps~kubernetes_helm/helm: No such file or directory

@meteorcloudy do you know why bazel behaves differently on buildkite?

[meteorcloudy]

Probably due to bazelbuild/bazel#23127

You should avoid using the canonical repo name whenever possible, please use the runfiles library to access data file.

[ensonic]

Probably due to bazelbuild/bazel#23127

You should avoid using the canonical repo name whenever possible, please use the runfiles library to access data file.

Thanks for the info. I hope we can find a better way. I am reluctant to add 10 lines of yucky shell code to each sh_test.

Does bazelbuild/bazel#23127 mean the '~' will change to '+' ?

[meteorcloudy]

Yes, and _main is omitted. So _main~non_module_deps~kubernetes_helm -> +non_module_deps+kubernetes_helm

[ensonic]

Lets see if #445 helps ...

[ensonic]

Sadly had to revert it, as it was then failing on our CI. Hence we need to update those 3 places when we bump the bazel version again :/

[ensonic]

Its green now:
https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4150#01922c85-003c-40ee-aa6f-8487bd300c66