pants package :: can corrupt directory target output by leaving stale/deleted contents

[huonw]

Describe the bug

A packageable target can sometimes be a directory. For instance, pex_binary(..., layout="packed") (or layout="loose"). If someone has run pants package :: once to output this target, makes a change that removes files, and then reruns pants package ::, the removed files are left there.

In theory, writing to dist/ should exactly synchronise the contents of a target-that-outputs-directory (removing stale contents). This will need to be done at an appropriate level, e.g. for a packed pex pants package path/to:target, it'll output dist/path.to/target.pex/..., but it should only synchronise dist/path.to/target.pex/, not removing anything from dist/ or dist/path.to/.

This likely applies to other methods of writing to dist/, like export-codegen.

Reproducer:

cd $(mktemp -d)
cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.15.0"
backend_packages = ["pants.backend.python"]

[anonymous-telemetry]
enabled = false
EOF

cat > BUILD <<EOF
python_sources(name="sources")
pex_binary(name="pex", layout="packed", dependencies=[":sources"])
EOF

touch a.py b.py

# initial export:
pants package :pex
tree dist

rm b.py
pants package :pex
# BUG: b.py is still in dist/pex.pex/
tree dist

# clear out and start again to validate the expected contents (no b.py)
rm -rf dist
pants package :pex
tree dist

Output:

1. initial export

   08:43:37.96 [INFO] Wrote dist/pex.pex
   dist
   └── pex.pex
       ├── PEX-INFO
       ├── __main__.py
       ├── __pex__
       │   └── __init__.py
       ├── a.py
       └── b.py
   
   2 directories, 5 files
   

2. BUG: b.py is still in dist/pex.pex/

   08:43:38.33 [WARN] Unmatched glob from //b.py:sources's `source` field: "b.py"
   
   Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to https://www.pantsbuild.org/v2.15/docs/troubleshooting#pants-cannot-find-a-file-in-your-project.
   08:43:38.33 [INFO] Wrote dist/pex.pex
   dist
   └── pex.pex
       ├── PEX-INFO
       ├── __main__.py
       ├── __pex__
       │   └── __init__.py
       ├── a.py
       └── b.py
   
   2 directories, 5 files
   

3. clear out and start again ...

   08:43:38.74 [INFO] Wrote dist/pex.pex
   dist
   └── pex.pex
       ├── PEX-INFO
       ├── __main__.py
       ├── __pex__
       │   └── __init__.py
       └── a.py
   
   2 directories, 4 files
   

Pants version
2.15.0

OS
macOS

Additional info

Potentially vaguely related: #17758, #18809.

[jsirois]

@huonw its quibbling, but Pants has always just populated dist/ back to v0 days and never synchronized. You're really describing new semantics you desire / a new feature.

[huonw]

Feel free to reframe it as a feature request (although, from a user perspective, the end result of silently corrupting a target-that-outputs-a-directory is very surprising, and is very different to target-that-outputs-a-file).

[jsirois]

Ah, I read too fast. Ok, yeah - not removing the complete dir then replacing it is odd. Pex used alone does that; so Pants is doing something weird here.