Invoking _get_runnable_pip to install setuptools is problematic

[matteius]

Description

This may or may not be a pip bug, but I am stumped trying to figure it out and does involve pip so I am hoping for some help. In pipenv we used to invoke pip but recently changed to invoke _get_runnable_pip to ensure we use our vendor'd version of pip which is currently the latest 22.2.2. The problem is that get this released, I excluded a list of BAD_PACKAGES from the install phase which includes setuptools because I figured its already provided by the virtualenv, but some users wish to install a specific version of setuptools different from this --which is where the problem exists.

pipenv itself has setuptools in its lock file as well, and when I remove the BAD_PACKAGES work-around for this issue, I find that the pip runnable removes setuptools and then the new setuptools fails to install because pkg_resources is no longer available, presumable because the prior version of setuptools had just been removed. I am having trouble figuring out how to get this working the way it was working when just pip was invoked and this scenario worked.

This issue occurs during the dependency setup before the test runner ever has a chance to run, but I am not sure why its happening like this and would love any help trying to understand what the difference is.

Expected behavior

the _get_runnable_pip should be able to upgrade setuptools the way pip does when its invoked.

pip version

22.2.2

Python version

3.7,3.8,3.9 (weirdly python 3.10 works)

OS

linux/windows/mac

How to Reproduce

Haven't been able to reproduce this locally I was just able to reproduce it by creating my pipenv virtualenv with python3.8 - it works fine on python 3.10. github actions for pipenv cannot get to running the tests due to this issue trying to upgrade setuptools using _get_runnable_pip as well on anything other than python 3.10

Output

Successfully built pipenv
Installing collected packages: distlib, zipp, virtualenv-clone, typing-extensions, platformdirs, filelock, certifi, importlib-metadata, virtualenv, pipenv
Successfully installed certifi-2022.6.15 distlib-0.3.5 filelock-3.8.0 importlib-metadata-4.12.0 pipenv-2022.8.18.dev0 platformdirs-2.5.2 typing-extensions-4.3.0 virtualenv-20.16.3 virtualenv-clone-0.5.7 zipp-3.8.1
Creating a virtualenv for this project...
Pipfile: /home/runner/work/pipenv/pipenv/Pipfile
Using /opt/hostedtoolcache/Python/3.7.13/x64/bin/python (3.7.13) to create virtualenv...
created virtual environment CPython3.7.13.final.0-64 in 481ms
  creator CPython3Posix(dest=/home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/runner/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator


Successfully created virtual environment!
Virtualenv location: /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5
Installing dependencies from Pipfile.lock (a4c141)...
Ignoring atomicwrites: markers 'sys_platform == "win32"' don't match your environment
Ignoring colorama: markers 'platform_system == "Windows"' don't match your environment
An error occurred while installing setuptools==65.1.0; python_version >= '3.7' --hash=sha256:10602cd0a6f5feab6656e9587f9075292ab777c5200f3bf00293ecd23d9f2788 --hash=sha256:d2e010624c781b26ad6629a8de9832327cf853dea93894487979e55f9ad06857! Will try again.
Installing initially failed dependencies...
[pipenv.exceptions.InstallError]: Collecting setuptools==65.1.0
[pipenv.exceptions.InstallError]:   Using cached setuptools-65.1.0-py3-none-any.whl (1.2 MB)
[pipenv.exceptions.InstallError]: Installing collected packages: setuptools
[pipenv.exceptions.InstallError]:   Attempting uninstall: setuptools
[pipenv.exceptions.InstallError]:     Found existing installation: setuptools 63.4.1
[pipenv.exceptions.InstallError]:     Uninstalling setuptools-63.4.1:
[pipenv.exceptions.InstallError]:       Successfully uninstalled setuptools-63.4.1
[pipenv.exceptions.InstallError]:   Rolling back uninstall of setuptools
[pipenv.exceptions.InstallError]:   Moving to /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/_distutils_hack/
[pipenv.exceptions.InstallError]:    from /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/~distutils_hack
[pipenv.exceptions.InstallError]:   Moving to /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/distutils-precedence.pth
[pipenv.exceptions.InstallError]:    from /tmp/pip-uninstall-8_ao60oj/distutils-precedence.pth
[pipenv.exceptions.InstallError]:   Moving to /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/pkg_resources/
[pipenv.exceptions.InstallError]:    from /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/~kg_resources
[pipenv.exceptions.InstallError]:   Moving to /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/setuptools-63.4.1.dist-info/
[pipenv.exceptions.InstallError]:    from /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/~etuptools-63.4.1.dist-info
[pipenv.exceptions.InstallError]:   Moving to /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/setuptools-63.4.1.virtualenv
[pipenv.exceptions.InstallError]:    from /tmp/pip-uninstall-8_ao60oj/setuptools-63.4.1.virtualenv
[pipenv.exceptions.InstallError]:   Moving to /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/setuptools/
[pipenv.exceptions.InstallError]:    from /home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/~etuptools
[pipenv.exceptions.InstallError]: ERROR: Exception:
[pipenv.exceptions.InstallError]: Traceback (most recent call last):
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
[pipenv.exceptions.InstallError]:     status = run_func(*args)
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/cli/req_command.py", line 247, in wrapper
[pipenv.exceptions.InstallError]:     return func(self, options, args)
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/commands/install.py", line 470, in run
[pipenv.exceptions.InstallError]:     pycompile=options.compile,
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/req/__init__.py", line 81, in install_given_reqs
[pipenv.exceptions.InstallError]:     pycompile=pycompile,
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/req/req_install.py", line 759, in install
[pipenv.exceptions.InstallError]:     prefix=prefix,
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/locations/__init__.py", line 256, in get_scheme
[pipenv.exceptions.InstallError]:     prefix=prefix,
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/locations/_distutils.py", line 141, in get_scheme
[pipenv.exceptions.InstallError]:     scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix)
[pipenv.exceptions.InstallError]:   File "/home/runner/work/pipenv/pipenv/pipenv/patched/pip/_internal/locations/_distutils.py", line 66, in distutils_scheme
[pipenv.exceptions.InstallError]:     obj = d.get_command_obj("install", create=True)
[pipenv.exceptions.InstallError]:   File "/home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 864, in get_command_obj
[pipenv.exceptions.InstallError]:     klass = self.get_command_class(command)
[pipenv.exceptions.InstallError]:   File "/home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/setuptools/dist.py", line 954, in get_command_class
xceptions.InstallError]:     self.cmdclass[command] = cmdclass = ep.load()
[pipenv.exceptions.InstallError]:   File "/home/runner/.local/share/virtualenvs/pipenv-XWpgSlS5/lib/python3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 194, in load
[pipenv.exceptions.InstallError]:     module = import_module(match.group('module'))
[pipenv.exceptions.InstallError]:   File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/importlib/__init__.py", line 127, in import_module
[pipenv.exceptions.InstallError]:     return _bootstrap._gcd_import(name[level:], package, level)
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
[pipenv.exceptions.InstallError]: ModuleNotFoundError: No module named 'setuptools.command.install'
ERROR: Couldn't install package: setuptools
 Package installation failed...
Error: Process completed with exit code 1.

For more details on the dependency installation failure runs: https://github.com/pypa/pipenv/runs/7909760557?check_suite_focus=true

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[matteius]

What are the implications of patching this to be for python 3.7 and great, not just 3.10 and greater? https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/__init__.py#L44

It seems to have gotten past the issue I described on the other python versions by doing so.

[uranusjr]

sysconfig was not properly patched in a lot of systems prior to 3.10, and it took us quite a few fixes in both CPython and Linux vendors to get this working. This is most not an issue for pipenv since it mostly only works with virtual environments, and would likely work for everything except --system invocations

[matteius]

Thanks @uranusjr -- that as my workaround for this issue. I'll be keeping a look out for new issues that might be side-effects of this change, but initial testing with the --system flag also allowed upgrading setuptools to a specific version on Ubuntu 21.10, but perhaps older systems might have a side effect. I will close this out, but anyone should feel free to let me know additional insights about this edge case.