export-codegen is not resilient to existing contents in dist/codegen

[huonw]

Describe the bug

When running pants export-codegen ... when there's existing entries in overlapping locations in dist/, it doesn't clear out those entries, which can lead to problems:

1. if the existing entries are of a different type (e.g. a file in dist/, and writing a directory, or vice versa), running export-codegen throws errors like IntrinsicError: Error opening file .../dist/codegen/entry for writing: Os { code: 21, kind: IsADirectory, message: "Is a directory" }
2. if there's generated directories that have stale contents, they're not removed
3. (if there's symlinks, overwriting the symlinks with the same symlinks failed. This was worked-around in IntrinsicError: File exists (os error 17) when attempting to overwrite a symlink in dist/ #18809 / Remove existing entry if any when materialising symlink #18873, but potentially that work-around could be removed.)

Reproducer:

cd $(mktemp -d)

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.16.0rc1"
backend_packages = ["pants.backend.shell"]

[anonymous-telemetry]
enabled = false
EOF

cat > BUILD <<EOF
shell_command(
  name="directory",
  command="mkdir -p entry; touch entry/new_file.txt",
  tools=["mkdir", "touch"],
  output_directories=["entry"],
)
shell_command(
  name="file",
  command="touch entry",
  tools=["touch"],
  output_files=["entry"]
)
EOF

pants version

# simulate having done a previous export that had different files:
mkdir -p dist/codegen/entry/existing_file.txt

# BUG 2: existing_file.txt still exists
pants export-codegen :directory
ls dist/codegen/entry
#> existing_file.txt new_file.txt

# BUG 1: errors if `dist/codegen/entry` already exists of a different kind
pants export-codegen :file
#> IntrinsicError: Error opening file .../dist/codegen/entry for writing: Os { code: 21, kind: IsADirectory, message: "Is a directory" }

Pants version
2.16.0rc1, but likely affects all versions

OS
macOS

Additional info

This is the equivalent of #17758 and #18849 for export-codegen instead of package. #18930 built infrastructure with write_digest(..., clear_paths=...) that makes fixing this easier.

[huonw]

I started a fix in #18871, but it's not a priority for me in the near term (was far less critical than #18849 was), so I haven't pushed it to completion. That PR notes what was missing.

I'm happy to help someone pick it back up, if you're interested.