[FR] Automatically clean-up egg-info left in directory on isolated builds

[effigies]

setuptools version

setuptools==65.3.0

Python version

Python 3.8.5

OS

Ubuntu 22.04

Additional environment information

Running using python -m build. This occurs with tags before and after moving to pyproject.toml.

Description

Building with python -m build, a <package>.egg-info/ directory is left behind along with the dist/ after cleanup. This includes a SOURCES.txt that will be reused during the next build, which can lead to unexpected behavior when modifying MANIFEST.in.

Expected behavior

I would expect egg-info/ directories to be removed during non-editable builds.

How to Reproduce

git clone https://github.com/pypa/sampleproject.git
python -m build
git status --ignored

Output

$ git clone https://github.com/pypa/sampleproject.git
Cloning into 'sampleproject'...
remote: Enumerating objects: 543, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 543 (delta 4), reused 10 (delta 4), pack-reused 529
Receiving objects: 100% (543/543), 140.03 KiB | 1.25 MiB/s, done.
Resolving deltas: 100% (269/269), done.

$ cd sampleproject
$ python -m build
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools>=43.0.0, wheel)
* Getting dependencies for sdist...
running egg_info
creating src/sampleproject.egg-info
writing src/sampleproject.egg-info/PKG-INFO
writing dependency_links to src/sampleproject.egg-info/dependency_links.txt
writing entry points to src/sampleproject.egg-info/entry_points.txt
writing requirements to src/sampleproject.egg-info/requires.txt
writing top-level names to src/sampleproject.egg-info/top_level.txt
writing manifest file 'src/sampleproject.egg-info/SOURCES.txt'
reading manifest file 'src/sampleproject.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file 'src/sampleproject.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing src/sampleproject.egg-info/PKG-INFO
writing dependency_links to src/sampleproject.egg-info/dependency_links.txt
writing entry points to src/sampleproject.egg-info/entry_points.txt
writing requirements to src/sampleproject.egg-info/requires.txt
writing top-level names to src/sampleproject.egg-info/top_level.txt
reading manifest file 'src/sampleproject.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file 'src/sampleproject.egg-info/SOURCES.txt'
running check
creating sampleproject-2.0.0
creating sampleproject-2.0.0/data
creating sampleproject-2.0.0/src
creating sampleproject-2.0.0/src/sample
creating sampleproject-2.0.0/src/sampleproject.egg-info
copying files to sampleproject-2.0.0...
copying LICENSE.txt -> sampleproject-2.0.0
copying README.md -> sampleproject-2.0.0
copying pyproject.toml -> sampleproject-2.0.0
copying setup.cfg -> sampleproject-2.0.0
copying setup.py -> sampleproject-2.0.0
copying data/data_file -> sampleproject-2.0.0/data
copying src/sample/__init__.py -> sampleproject-2.0.0/src/sample
copying src/sample/package_data.dat -> sampleproject-2.0.0/src/sample
copying src/sample/simple.py -> sampleproject-2.0.0/src/sample
copying src/sampleproject.egg-info/PKG-INFO -> sampleproject-2.0.0/src/sampleproject.egg-info
copying src/sampleproject.egg-info/SOURCES.txt -> sampleproject-2.0.0/src/sampleproject.egg-info
copying src/sampleproject.egg-info/dependency_links.txt -> sampleproject-2.0.0/src/sampleproject.egg-info
copying src/sampleproject.egg-info/entry_points.txt -> sampleproject-2.0.0/src/sampleproject.egg-info
copying src/sampleproject.egg-info/requires.txt -> sampleproject-2.0.0/src/sampleproject.egg-info
copying src/sampleproject.egg-info/top_level.txt -> sampleproject-2.0.0/src/sampleproject.egg-info
Writing sampleproject-2.0.0/setup.cfg
Creating tar archive
removing 'sampleproject-2.0.0' (and everything under it)
* Building wheel from sdist
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools>=43.0.0, wheel)
* Getting dependencies for wheel...
running egg_info
writing src/sampleproject.egg-info/PKG-INFO
writing dependency_links to src/sampleproject.egg-info/dependency_links.txt
writing entry points to src/sampleproject.egg-info/entry_points.txt
writing requirements to src/sampleproject.egg-info/requires.txt
writing top-level names to src/sampleproject.egg-info/top_level.txt
reading manifest file 'src/sampleproject.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file 'src/sampleproject.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/sample
copying src/sample/__init__.py -> build/lib/sample
copying src/sample/simple.py -> build/lib/sample
copying src/sample/package_data.dat -> build/lib/sample
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/sample
copying build/lib/sample/__init__.py -> build/bdist.linux-x86_64/wheel/sample
copying build/lib/sample/simple.py -> build/bdist.linux-x86_64/wheel/sample
copying build/lib/sample/package_data.dat -> build/bdist.linux-x86_64/wheel/sample
running install_data
creating build/bdist.linux-x86_64/wheel/sampleproject-2.0.0.data
creating build/bdist.linux-x86_64/wheel/sampleproject-2.0.0.data/data
creating build/bdist.linux-x86_64/wheel/sampleproject-2.0.0.data/data/my_data
copying data/data_file -> build/bdist.linux-x86_64/wheel/sampleproject-2.0.0.data/data/my_data
running install_egg_info
running egg_info
writing src/sampleproject.egg-info/PKG-INFO
writing dependency_links to src/sampleproject.egg-info/dependency_links.txt
writing entry points to src/sampleproject.egg-info/entry_points.txt
writing requirements to src/sampleproject.egg-info/requires.txt
writing top-level names to src/sampleproject.egg-info/top_level.txt
reading manifest file 'src/sampleproject.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file 'src/sampleproject.egg-info/SOURCES.txt'
Copying src/sampleproject.egg-info to build/bdist.linux-x86_64/wheel/sampleproject-2.0.0-py3.8.egg-info
running install_scripts
adding license file "LICENSE.txt" (matched pattern "LICENSE.txt")
creating build/bdist.linux-x86_64/wheel/sampleproject-2.0.0.dist-info/WHEEL
creating '/home/chris/Projects/sampleproject/dist/tmpyw2to0dj/sampleproject-2.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'sample/__init__.py'
adding 'sample/package_data.dat'
adding 'sample/simple.py'
adding 'sampleproject-2.0.0.data/data/my_data/data_file'
adding 'sampleproject-2.0.0.dist-info/LICENSE.txt'
adding 'sampleproject-2.0.0.dist-info/METADATA'
adding 'sampleproject-2.0.0.dist-info/WHEEL'
adding 'sampleproject-2.0.0.dist-info/entry_points.txt'
adding 'sampleproject-2.0.0.dist-info/top_level.txt'
adding 'sampleproject-2.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built sampleproject-2.0.0.tar.gz and sampleproject-2.0.0-py3-none-any.whl

$ git status --ignored
On branch main
Your branch is up to date with 'origin/main'.

Ignored files:
  (use "git add -f <file>..." to include in what will be committed)
	dist/
	src/sampleproject.egg-info/

nothing to commit, working tree clean

[abravalheri]

Hi @effigies, thank you very much for bringing this topic for discussion.

I would classify this as a feature request instead of a bug.
Classic setuptools would require the user to clean the working dir, so I believe the behaviour was just carried over.

Would you like to work on a PR for that?

[effigies]

Thanks for the response! I agree, I wasn't sure if this was intended, unintended or just legacy behavior. I think the fix would be adding a step to remove the egg-info to the sdist process, similar to

removing 'sampleproject-2.0.0' (and everything under it)

Should we also be deleting .egg-info/ directories that are found at the start of the build, to ensure reproducibility? What should happen if I have an editable installation and want to build an sdist?

I would be up for working on a PR, but my time is pretty limited, so if someone else is motivated I don't want them to wait on me.

[abravalheri]

The "final" solution would be fulfilling https://github.com/pypa/setuptools/milestone/3.

But we can think about intermediate steps as the one you proposed.
Personally, I think setting egg_base in the egg_info command to a directory contained in the metadata_directory (PEP 517) is a more interesting solution, but there are definitely challenges with that approach...

[sbidoul]

Chiming in to mention that it'd be nice if the .egg-info directory was not generated during a PEP 660 editable install either.

[astrojuanlu]

Do I understand correctly that a series of PRs were opened that would close gh-1386 and possibly this issue as well?

[abravalheri]

Hi @astrojuanlu, I don't think so.
While I was investigating this, I noticed that there are some parts of setuptools that are very intertwined with egg-info and not running the command the way it is right now would break other parts.

(Automatically removing the directory is also problematic because it can cause race conditions with importlib.metadata...)

The series of open PRs (still in progress) is only a step on this direction, but not the full solution.