-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] unable to do a local install with pyproject.toml in a venv, created with --system-site-packages #3240
Comments
Hi @skirpichev, after some investigation it seems that This is what I learned so far: A. We can replicate the problem in a Debian installation (
B. This error is caused by
C. Repeat the process A but in a fedora container (
So far it does not look to me that this problem is particularly caused by version 61, but rather by how |
I don't know if this is related to https://sources.debian.org/src/python3.10/3.10.4-2/Lib/_distutils_system_mod.py/ somehow. |
Hmm, I don't think it's a Debian-specific problem. In fact, the used python3.10 binary (I use the stable, i.e. bullseye - and it has CPython 3.9, not 3.10!) was locally-built (./confiqure -q && make -s && make -s install :)) So, if there are Debian-specific patches for the distutils stuff - I doubt they were applied. The binary was rebuilt recently:
Sorry for an incomplete bugreport, I hope this helps. |
I am replicating the bug of from where setuptools is being loaded even with version |
These are the steps to replicate the problem with > docker run --rm -it python:3.10-bullseye /bin/bash
mkdir /tmp/proj
cd /tmp/proj
cat <<EOS > pyproject.toml
[build-system]
requires = ['setuptools==60.9.3']
build-backend = 'setuptools.build_meta'
EOS
python3 -m venv --system-site-packages .venv
.venv/bin/python -m pip install -U pip 'setuptools==60.9.3'
# Let's make the global setuptools broken on purpose
sed -i '/def get_requires_for_build_wheel/a\ raise SystemError("setuptools loaded from /usr", setuptools.__version__)' /usr/local/lib/python3.10/site-packages/setuptools/build_meta.py .venv/bin/python -m pip install .
Processing /tmp/proj
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Traceback (most recent call last):
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
return hook(config_settings)
File "/usr/local/lib/python3.10/site-packages/setuptools/build_meta.py", line 153, in get_requires_for_build_wheel
raise SystemError("setuptools loaded from /usr", setuptools.__version__)
SystemError: ('setuptools loaded from /usr', '58.1.0')
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip. |
I can confirm the same error for alpine. I ran the same example as above via: docker run --rm -it python:3.10-alpine3.14 /bin/ash The error persists even if we do the following: # Make sure manipulations of the PYTHONPATH are not used in the virtual environment
rm -rf .venv/lib/python3.10/site-packages/*.pth
# Let's add a `setup.cfg` just in case
cat <<EOS > setup.cfg
[metadata]
name = proj
version = 42
EOS Alpine Linux does not rely on a specific hook: |
@skirpichev, I am not sure if there is any action point for setuptools here. I don't know how |
If you want to report this problem to the other related tools, I recommend the following reproducer: docker run --rm -it python:3.10-alpine3.14 /bin/ash
# OR
# docker run --rm -it python:3.10-bullseye /bin/bash
mkdir /tmp/proj
cd /tmp/proj
cat <<EOS > pyproject.toml
[build-system]
requires = ['setuptools==60.9.3']
build-backend = 'setuptools.build_meta'
EOS
cat <<EOS > setup.cfg
[metadata]
name = proj
version = 42
EOS
python3 -m venv --system-site-packages .venv
.venv/bin/python -m pip install -U pip 'setuptools==60.9.3'
# Let's make the global setuptools broken on purpose
sed -i '/def get_requires_for_build_wheel/a\ raise SystemError("setuptools loaded from /usr", setuptools.__version__)' /usr/local/lib/python3.10/site-packages/setuptools/build_meta.py
.venv/bin/python -m pip install . Processing /tmp/proj
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Traceback (most recent call last):
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
return hook(config_settings)
File "/usr/local/lib/python3.10/site-packages/setuptools/build_meta.py", line 153, in get_requires_for_build_wheel
raise SystemError("setuptools loaded from /usr", setuptools.__version__)
SystemError: ('setuptools loaded from /usr', '58.1.0')
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip. Personally, I would expect |
Sorry, I doubt this should be silently closed. There was no difference before introducing support for the pyproject.toml. Please reconsider. |
Hi @skirpichev, thanks for the feedback. I will reopen the issue, but I don't see what can be done in setuptools.
The example in #3240 (comment) shows the opposite doesn't it? According to that example, What would be your suggestion? |
I didn't get it. You break something, right? But if you don't, both variants (with and without options) - work for me with the old setuptools versions.
Unless I miss something, the above configuration is an equivalent for the original version from the bugreport. The differences are: using setup.cfg for the project info and the old setuptools version, of course. In fact, the latter doesn't matter: you can drop the harcoded setuptools version and use 61.3.1. The thing that does create the difference in behaviour for different venv's - is using the pyproject.toml for the project info. Maybe it's an accident that is was working this way. But it was so for years...
So far - removing the invalid label:) Probably, both configs (with and without using the pyproject.toml for the project info) - should work in a same way for any given virtual environment. E.g. fail both if the |
The fact that it works is a coincidence between the version that you have installed system-wide in your machine (outside of the virtual environment) and the feature you need. Somehow the installer/front-end is loading Let's illustrate this problem once more: A. Create a project with a specific version of setuptools. Set a virtual environment that fulfils the build dependencies. > docker run --rm -it python:3.10-alpine3.14 /bin/ash
mkdir /tmp/proj
cd /tmp/proj
cat <<EOS > pyproject.toml
[build-system]
requires = ['setuptools==47']
build-backend = 'setuptools.build_meta'
EOS
python3 -m venv --system-site-packages .venv
.venv/bin/python -m pip install -U pip 'setuptools==47' Here I choose setuptools 47 because it is a version that does not try to mess with .venv/bin/python3 -c 'import setuptools; print(setuptools.__version__)'
# 47.0.0
ls .venv/lib/python3.10/site-packages/*.pth
# ls: .venv/lib/python3.10/site-packages/*.pth: No such file or directory B. Install an older version of setuptools in the system wide Python installation (outside of the virtual environment) python3 -m pip uninstall -y setuptools
python3 -m pip install 'setuptools==39'
# Meanwhile the version inside the virtual environment exists and has the correct version
.venv/bin/python -c 'import setuptools; print(setuptools.__version__)'
# 47.0.0 C. Run the installation and see it failing because it is loading the version of setuptools from outside the virtual environment. .venv/bin/python -m pip install .
Processing /tmp/proj
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [38 lines of output]
Traceback (most recent call last):
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 124, in get_requires_for_build_wheel
backend = _build_backend()
File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
obj = import_module(mod_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/local/lib/python3.10/site-packages/setuptools/__init__.py", line 12, in <module>
import setuptools.version
File "/usr/local/lib/python3.10/site-packages/setuptools/version.py", line 1, in <module>
import pkg_resources
File "/usr/local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 77, in <module>
__import__('pkg_resources.extern.packaging.requirements')
File "/usr/local/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 9, in <module>
from pkg_resources.extern.pyparsing import stringStart, stringEnd, originalTextFor, ParseException
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
File "/usr/local/lib/python3.10/site-packages/pkg_resources/extern/__init__.py", line 43, in load_module
__import__(extant)
File "/usr/local/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 943, in <module>
collections.MutableMapping.register(ParseResults)
AttributeError: module 'collections' has no attribute 'MutableMapping'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip. From the backtrace we can see the frontend/installer is trying to load setuptools from Please note here the current code for |
I think the best thing to do here is to ask pip maintainers if this behaviour is intended or if they have any clues why this might be happening. |
Just for record system-wide versions: ls -l /usr/local/lib/python3.10/site-packages/
total 32K
drwxr-s--- 3 sk staff 4.0K Nov 8 05:33 _distutils_hack
-rw-r----- 1 sk staff 152 Nov 8 05:33 distutils-precedence.pth
drwxr-s--- 5 sk staff 4.0K Nov 8 05:33 pip
drwxr-s--- 2 sk staff 4.0K Nov 8 05:34 pip-21.2.4.dist-info
drwxr-s--- 6 sk staff 4.0K Nov 8 05:33 pkg_resources
-rw-r--r-- 1 sk staff 119 Feb 19 07:15 README.txt
drwxr-s--- 7 sk staff 4.0K Nov 8 05:33 setuptools
drwxr-s--- 2 sk staff 4.0K Nov 8 05:33 setuptools-58.1.0.dist-info I'll debug this further, thanks. On another hand, I've used setup.cfg since 2018 which was fine with/without the |
Hi @skirpichev, I opened an issue with
It might be the case the version you had installed in the system's site-packages/dist-packages were always enough to provide the features you needed. I don't know if you can do that in your machine, but is there any chance you can try to "uninstall" setuptools from your global system's site-packages/dist-packages just to see if the problem persists? For example, if you are running: python3.10 -m venv --system-site-packages ~/venv/xxx Is it a possibility for you to |
(Alternatively, if you have the rights you can also try to update the setuptools version on the system installation to see if the problem persists). |
@skirpichev, there seems to be a problem with pip: pypa/pip#6264. Please let me know if you can somehow confirming that once the system's site-packages/dist-packages does not contain Otherwise I think the best thing to do right now is to close this ticket, because we cannot do much about it in setuptools. |
Yes, you are right. Thanks for patience. On another hand, I don't understand why this is working (setup.cfg-based example above with minor edit):
Shouldn't this fail due to a bad setuptools version in the requires? Of course, probably this is just another side of the build frontend issue (i.e. pip) you reported. |
If you setup.cfg based example does not require any advanced feature, building ir with setuptools 58.1.0 is probably fine and does not cause errors. |
Well, the question was why the pip (or someone else?) does ignore the (strict!) version requirement for setuptools? Instead, it just silently uses a different version... I would expect a failure in this situation (c.f. with the case you add an install_requires in the setup.cfg). And maybe it's a different problem wrt can or can't the pip use the setuptools from the system-wide environment. I.e. if it's ok that it can - I still would expect, that only with an acceptable version... |
I am sure that is not intentional and they pretty much would prefer to have a failure or warning. But all software is subject to bugs, and not always we realize they exist when we are coding. Other times it might also be super difficult to find the origin of the problem and fix it. |
Ok, let me know if you feel that this deserve a separate issue for pip. |
setuptools version
61.3.1
Python version
3.10
OS
Debian Linux
Additional environment information
No response
Description
Initially reported here. Local installation of the package doesn't work in the venv, created with
--system-site-packages
.The example of package:
Expected behavior
There should be no difference wrt presence of the
--system-site-packages
option.How to Reproduce
Output
The text was updated successfully, but these errors were encountered: