action sandbox elides empty directories from tree artifacts

[ghost]

Description of the problem / feature request:

I happened upon a tree artifact edge case involving empty directories.

Consider two tree artifacts, treeA and treeB. Both contain an empty directory named empty, but treeB also contains a file named dummy.

$ tree tree*
treeA
└── empty
treeB
├── dummy
└── empty

When these tree artifacts are provided as inputs to a sandboxed action, Bazel elides the empty directory from treeB. If, however, a tree artifact's only content is an empty directory, then the empty directory appears in the action's sandbox. A find -L . in a genrule having both treeA and treeB reports the following to stdout:

.
./external
./external/bazel_tools
./external/bazel_tools/tools
./external/bazel_tools/tools/genrule
./external/bazel_tools/tools/genrule/genrule-setup.sh
./bazel-out
./bazel-out/k8-fastbuild
./bazel-out/k8-fastbuild/bin
./bazel-out/k8-fastbuild/bin/treeA
./bazel-out/k8-fastbuild/bin/treeA/empty
./bazel-out/k8-fastbuild/bin/treeB
./bazel-out/k8-fastbuild/bin/treeB/dummyfile

(Observe that treeB/empty is missing.)

Motivation

I'm trying to implement pkg_tree in rules_pkg (bazelbuild/rules_pkg#388) but my test cases fail because something in spawn/sandbox-land is eating my empty input directories.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

See repro at https://github.com/beasleyr-vmw/repro-treeartifact-emptysubdir

What operating system are you running Bazel on?

CentOS 8

What's the output of bazel info release?

release 4.2.1

If bazel info release returns "development version" or "(@Non-Git)", tell us how you built Bazel.

n/a

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

n/a

Have you found anything relevant by searching the web?

The following appear to be somewhat related.

• Sandbox does not create empty directory for aar_import with empty resources treeartifact #1928
• 3f008a0

Any other information, logs, or outputs that you want to share?

[meisterT]

cc @larsrc-google

[larsrc-google]

@alexjski What is the proper behaviour for tree artifacts wrt empty directories?

[alexjski]

@alexjski What is the proper behaviour for tree artifacts wrt empty directories?

This is not at all an issue of the sandbox -- tree artifact values currently do not collect empty directories. The difference between a local build and a sandboxed one is that a local one would pick up those directories which are not represented in the metadata held by Bazel (as such it's kind of like picking up auxiliary files).

I think a workaround you could apply would be to declare those empty directories as nested tree artifacts, generated by the same action:

outer = ctx.actions.declare_directory("dir")
empty = ctx.actions.declare_directory("dir/empty")
...
ctx.actions.run(..., outputs=[outer, empty, ...], ...)

@coeuvre Do we get empty directories within tree artifacts when we run an action generating trees in RBE?

[coeuvre]

Sorry I missed the notification. For remote execution, only files are downloaded, empty directories are not downloaded.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!