'grep: MANIFEST: No such file or directory' with remote caching

[ash2k]

What version of gazelle are you using?

49a5b63

What version of rules_go are you using?

bazel-contrib/rules_go@cbb70ea

What version of Bazel are you using?

v3.4.1

Does this issue reproduce with the latest releases of all the above?

These versions are not too old, newer than latest released versions. Haven't tried the latest commits.

What operating system and processor architecture are you using?

linux, x86_64

What did you do?

Recently changed the build to use remote caching with a GCP Bucket.

What did you expect to see?

Passing build with no errors.

What did you see instead?

Now the build is failing with:

bazel run //:gazelle
(08:22:22) INFO: Invocation ID: 916da195-cdb1-48e5-b67b-3377b681ed74
(08:22:23) INFO: Current date is 2020-08-27
(08:22:23) Loading: 
(08:22:23) Loading: 0 packages loaded
(08:22:23) Analyzing: target //:gazelle (0 packages loaded, 0 targets configured)
(08:22:24) INFO: Analyzed target //:gazelle (24 packages loaded, 298 targets configured).
(08:22:24) INFO: Found 1 target...
(08:22:24) [0 / 5] [Prepa] BazelWorkspaceStatusAction stable-status.txt
(08:22:25) [15 / 68] GoCompilePkg external/org_golang_x_tools/go/analysis/passes/buildtag/go_tool_library.a [for host]; 0s remote-cache ... (2 actions running)
(08:22:26) [35 / 83] GoCompilePkg external/org_golang_x_tools/go/analysis/passes/httpresponse/go_tool_library.a [for host]; 0s remote-cache ... (2 actions running)
(08:22:27) [48 / 83] GoLink nogo_for_ci_/nogo_for_ci [for host]; 0s remote-cache
(08:22:28) [60 / 83] GoCompilePkg external/org_golang_x_tools/cmd/goyacc/goyacc.a [for host]; 0s remote-cache ... (2 actions running)
(08:22:29) [78 / 83] GoCompilePkg external/bazel_gazelle/language/go_default_library.a [for host]; 0s remote-cache
Target //:gazelle up-to-date:
  bazel-bin/gazelle-runner.bash
  bazel-bin/gazelle
(08:22:29) INFO: Elapsed time: 6.864s, Critical Path: 2.66s
(08:22:29) INFO: 75 processes: 75 remote cache hit.
(08:22:29) INFO: Build completed successfully, 83 total actions
(08:22:29) INFO: Running command line: bazel-bin/gazelle -bazel_run
(08:22:29) INFO: Build completed successfully, 83 total actions
grep: MANIFEST: No such file or directory
error: could not locate gazelle binary
make: *** [update-bazel] Error 1

I didn't notice the problem because our build does not run gazelle on branches, only on master. This is not needed, so I'll remove the invocation as a fix, but it probably shouldn't be happening.

For your convenience, here are some relevant links:

• WORKSPACE
• root BUILD.bazel
• CI configuration
• .bazelrc. Please also see the link above for some extra bazel command line flags that are used in CI only. Remote Caching configuration is there.
• Makefile

[jayconrod]

I tried running bazel run //:gazelle in the Gazelle repo with remote caching enabled, but I didn't see this problem. I'm probably not going to be able to reproduce your setup or the issue you're seeing unless there's a more minimal test case.

The error sounds like the Gazelle runner script (generated from gazelle.bash.in) is failing to find runfiles. It will check the local directory first, then will fall back to MANIFEST, which is used on Windows. Maybe try running with -s to see how Bazel is invoking the runner script, or add -x at the end of the first line to see what commands the script is executing?

[ash2k]

Hm, I've removed Gazelle invocation but now rules_docker fails in a similar way:

(10:42:01) INFO: Analyzed target //cmd/agentk:push_docker_latest (0 packages loaded, 0 targets configured).
(10:42:01) INFO: Found 1 target...
(10:42:01) [0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //cmd/agentk:push_docker_latest up-to-date:
  bazel-bin/cmd/agentk/push_docker_latest.digest
  bazel-bin/cmd/agentk/push_docker_latest
(10:42:01) INFO: Elapsed time: 0.628s, Critical Path: 0.02s
(10:42:01) INFO: 0 processes.
(10:42:01) INFO: Build completed successfully, 1 total action
(10:42:01) INFO: Running command line: bazel-bin/cmd/agentk/push_docker_latest
(10:42:01) INFO: Build completed successfully, 1 total action
/root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/execroot/gitlab_k8s_agent/bazel-out/k8-fastbuild/bin/cmd/agentk/push_docker_latest: line 31: /root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/execroot/gitlab_k8s_agent/bazel-out/k8-fastbuild/bin/cmd/agentk/push_docker_latest.runfiles/gitlab_k8s_agent/../io_bazel_rules_docker/container/go/cmd/pusher/pusher_/pusher: No such file or directory
make: *** [release-latest-all-ci] Error 1

It feels like bazel run is not working properly with --remote_download_minimal. I have found these issues which may be relevant:

• remotely-built java_binarys can't run locally bazelbuild/bazel#10021
• remote_download_outputs=toplevel doesn't download libraries in cc_import needed to run. bazelbuild/bazel#11532
• Build event file creation fails with symlinks when using --remote_download_minimal bazelbuild/bazel#11942

Using --remote_download_toplevel instead of --remote_download_minimal fixes the rules_docker issue. I'll try this with Gazelle to verify and also try the flags you suggested later today. Thanks.

[ash2k]

I've tried to use gazelle repository to reproduce the issue, but something else is happening here.

I've added the following to .bazelrc:

build --remote_download_minimal
build --remote_cache=https://storage.googleapis.com/my_bucket
build --google_credentials=creds.json

And then:

bazel run //:gazelle  # OK
bazel clean # OK
bazel run -s //:gazelle > ./log.txt 2>&1
cat ./log.txt 

INFO: Invocation ID: 5d377f40-2ea0-4bef-8bfe-e13b9089e21f
Loading: 
Loading: 0 packages loaded
Analyzing: target //:gazelle (1 packages loaded, 0 targets configured)
INFO: Repository bazel_gazelle_go_repository_tools instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule go_repository_tools defined at:
  /Users/mikhail/src/bazel-gazelle/internal/go_repository_tools.bzl:117:38: in <toplevel>
ERROR: An error occurred during the fetch of repository 'bazel_gazelle_go_repository_tools':
   list_repository_tools_srcs: 2020/08/28 11:26:45 generated file internal/go_repository_tools_srcs.bzl is not up to date
exit status 1
ERROR: /Users/mikhail/src/bazel-gazelle/cmd/gazelle/BUILD.bazel:10:11: no such package '@com_github_pmezard_go_difflib//difflib': no such package '@bazel_gazelle_go_repository_config//': no such package '@bazel_gazelle_go_repository_tools//': list_repository_tools_srcs: 2020/08/28 11:26:45 generated file internal/go_repository_tools_srcs.bzl is not up to date
exit status 1
 and referenced by '//cmd/gazelle:gazelle_lib'
ERROR: Analysis of target '//:gazelle' failed; build aborted: Analysis failed
INFO: Elapsed time: 1.137s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (65 packages loaded, 6925 targets configured)
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully (65 packages loaded, 6925 targets configured)

Ok, this happens even without any modifications, so it's some other issue probably.

Edit: I think this was happening because the first gazelle run removed @com_github_bazelbuild_buildtools//build:go_default_library from //cmd/gazelle:gazelle_lib. If I put it back I still cannot make it work both with and without any changes. Even after bazel clean --expunge it's giving me this error:

Starting local Bazel server and connecting to it...
Loading: 
Loading: 0 packages loaded
Analyzing: target //:gazelle (1 packages loaded, 0 targets configured)
Analyzing: target //:gazelle (13 packages loaded, 14 targets configured)
Analyzing: target //:gazelle (13 packages loaded, 14 targets configured)
Analyzing: target //:gazelle (44 packages loaded, 6851 targets configured)
INFO: Repository bazel_gazelle_go_repository_tools instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule go_repository_tools defined at:
  /Users/mikhail/src/bazel-gazelle/internal/go_repository_tools.bzl:117:38: in <toplevel>
ERROR: An error occurred during the fetch of repository 'bazel_gazelle_go_repository_tools':
   list_repository_tools_srcs: 2020/08/28 11:43:39 generated file internal/go_repository_tools_srcs.bzl is not up to date
exit status 1
ERROR: /Users/mikhail/src/bazel-gazelle/language/go/BUILD.bazel:14:11: no such package '@com_github_pelletier_go_toml//': no such package '@bazel_gazelle_go_repository_config//': no such package '@bazel_gazelle_go_repository_tools//': list_repository_tools_srcs: 2020/08/28 11:43:39 generated file internal/go_repository_tools_srcs.bzl is not up to date
exit status 1
 and referenced by '//language/go:go'
ERROR: Analysis of target '//:gazelle' failed; build aborted: Analysis failed
INFO: Elapsed time: 11.883s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (63 packages loaded, 6899 targets configured)
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully (63 packages loaded, 6899 targets configured)

I'm probably doing something silly but I don't understand what.

[ash2k]

I've created bazelbuild/bazel#12015 as it seem to be a bazel issue.

[jayconrod]

It looks like --remote_download_minimal expands to a bunch of flags including --nobuild_runfile_links. I wonder if enabling that flag would help? Though if the runfiles aren't there then there would be nothing to link to. In any case, it does sound like a Bazel issue.

For the Gazelle build issue, let me know if you see that in the default configuration at v0.22.0. I think there was a similar error at an earlier untagged commit, might have been this one.

[ash2k]

I'm closing this one as it's a bazel issue. For the gazelle weirdness I'm seeing I've created #894.

[pcj]

Note: this error seems to occur in a nix environment as well.