How to treat non-stdlib packages that are available by default in [all] venvs ?

[Nour-Mws]

This is actually 2 issues for the price of one :D

In one conversation in PR #70, it turned out that some packages are available in virtual environments / part of the Python distribution, while not belonging to stdlib.

We'd like to know:

1. What are these packages (besides pkg_resources)?
2. Do we need to treat those the same as stdlib (i.e. filter them out of parsed imports). FD's current behavior is to report them as imports (and subsequently to report them as missing if undeclared).

Original conversation
@mknorps:

pkg_resources is not considered STDLIB:

• pkg_resources in locations:
  /home/maria/Tweag/FawltyDeps/fawltydeps/extract_dependencies.py:9

@jherland:

Searching for pkg_resources in the stdlib yields really only this page: https://docs.python.org/3/library/importlib.resources.html?highlight=pkg_resources which links to https://setuptools.pypa.io/en/latest/pkg_resources.html, showing that pkg_resources is part of setuptools, and that setuptool is not a part of the stdlib. Thus we should almost certainly add setuptools to our list of main dependencies. The reason it's gone under our radar is that setuptools (along with pip) is automatically installed into every(?) virtualenv...

[jherland]

• What are these packages (besides pkg_resources)?

(pkg_resources is part of setuptools)

Heres' what I get on my machine (NixOS, inside our dev shell) when using the venv module in stdlib for creating virtualenvs:

$ rm -rf foo && python3.7 -m venv foo && foo/bin/pip list
Error: Command '['/home/jherland/code/fawltydeps/foo/bin/python3.7', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
$ rm -rf foo && python3.8 -m venv foo && foo/bin/pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 56.0.0
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/home/jherland/code/fawltydeps/foo/bin/python3.8 -m pip install --upgrade pip' command.
$ rm -rf foo && python3.9 -m venv foo && foo/bin/pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 58.1.0
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/home/jherland/code/fawltydeps/foo/bin/python3.9 -m pip install --upgrade pip' command.
$ rm -rf foo && python3.10 -m venv foo && foo/bin/pip list
Package    Version
---------- -------
pip        22.3.1
setuptools 65.5.0
$ rm -rf foo && python3.11 -m venv foo && foo/bin/pip list
Package    Version
---------- -------
pip        22.3.1
setuptools 65.5.0

So it seems pip and setuptools are the only ones installed by default on this particular setup. I'd be surprised if the venv stdlib module behaved differently on other distros, but there are other mechanisms for creating virtualenvs, most notably the virtualenv tool (which is seems that Poetry might be using, as I found it already present in our dev shell:

$ rm -rf foo && virtualenv foo && foo/bin/pip list
created virtual environment CPython3.10.9.final.0-64 in 99ms
  creator CPython3Posix(dest=/home/jherland/code/fawltydeps/foo, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/jherland/.local/share/virtualenv)
    added seed packages: pip==22.3.1, setuptools==65.6.3, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Package    Version
---------- -------
pip        22.3.1
setuptools 65.6.3
wheel      0.38.4

That one includes wheel as well.

In any case, I'm pretty sure we're not importing anything from pip or wheel in our code, only pkg_resources from setuptools.

[jherland]

BTW, I came across this article (https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/#pip-a-privileged-player) that, in a footnote, linked to python/cpython#101039: Apparently setuptools is no longer being installed by default in a venv, starting with Python 3.12. Good thing we found this now, instead of getting an ugly surprise when upgrading later.

[jherland]

The opinion we have arrived at is that any undeclared non-stdlib dependency should be flagged, even if it's a package (like setuptools) that is available ~everywhere. There are so many different deployment scenarios for Python that there is always somewhere where this package might be missing. Declaring it is always the Right Thing™️ to do.