--remote_download_outputs=toplevel results in output files being deleted

[adam-azarchs]

Description of the bug:

Running bazel test --nobuild_runfile_links --build_tests_only --remote_download_outputs=toplevel deletes previously-downloaded files from the output directory.

Removing any of the three flags seems to make the problem go away, however at a very significant performance cost (adding several minutes to a build that's otherwise just a few seconds) during development. Deleting those runfiles causes significant pain as people may have e.g. active jupyter notebooks running against files from there.

This behavior seems weird to me because most of the time bazel will not delete files which it wasn't planning to touch during a particular build (which has, in the past, caused a different set of problems when things escape the runfiles symlink tree to find things they shouldn't be finding).

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

With remote execution configured,

1. bazel build //:my_target, where my_target has stuff in runfiles which may also be in the runfiles for some tests in the repo.
2. bazel test --nobuild_runfile_links --build_tests_only --remote_download_outputs=toplevel //....

After the second command is executed, many things in bazel-bin/my_target.runfiles/ will be broken symlinks because the files they pointed to were deleted by the second command.

Which operating system are you running Bazel on?

linux

What is the output of bazel info release?

release 5.2.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[coeuvre]

I believe this should be fixed by recent changes. Also see #18580 about --nobuild_runfile_links.