Isolated build environment has access to build's environment when run from a Python environment #97

pganssle · 2020-09-10T16:55:58Z

It seems that when constructing the isolated environment, python-build still uses the packages in the outer environment if they are present, rather than installing into a "clean" environment.

One consequence of this is that if you have an older version of your build backend in the environment that contains python-build, pip install won't install an upgraded version of it. For example, construct a project with a pyproject.toml like so:

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

And a virtualenv like so:

virtualenv venv --no-wheel
source venv/bin/activate
pip install build==0.0.4
pip install setuptools==39.0.0

When you run python-build -w ., when satisfying setuptools, pip will not upgrade setuptools, since it is "already satisfied":

Requirement already satisfied: setuptools in ./venv/lib/python3.8/site-packages (from -r /tmp/build-reqs-r6nyhzz2.txt (line 2)) (39.0.0)

Preferably, pip would consider the isolated environment a blank slate, and you'd get the latest setuptools, unless --no-isolation is passed.

This is not only a problem with pip — the build environment used contains all the packages from the environment python-build was invoked in, as you can see if you create a setup.py like so:

import setuptools

try:
    import dateutil
    raise Exception("dateutil should not be present during isolated builds!")
except ImportError
    print("dateutil not found")

setuptools.setup()

If you run this in the virtualenv we created above, it will succeed and print the "dateutil not found" message. If you then run pip install python-dateutil in the virtualenv, the python-build -w . will fail, indicating that the build is being carried out in an environment not isolated from the base environment.

These results on Arch Linux using virtualenv==20.0.26, build==0.0.4 and Python version 3.8.2 (via pyenv). I have not attempted to replicate this without a virtualenv.

The text was updated successfully, but these errors were encountered:

FFY00 · 2020-09-10T17:00:00Z

Hum, I am pretty sure I was passing --ignore-installed to pip, maybe that got removed in one of the reviews. Can you check if it fixes your issue?

pganssle · 2020-09-10T17:06:02Z

Hum, I am pretty sure I was passing --ignore-installed to pip, maybe that got removed in one of the reviews. Can you check if it fixes your issue?

That would only partially fix the issue — pip probably shouldn't need --ignore-installed because it shouldn't be seeing any packages installed in the isolated environment when it's invoked. It probably would solve the issue of the default upgrading behavior, but it won't solve the other issue I mentioned, which is that packages installed in the outer environment are visible in the isolated environment (see the example about dateutil in my earlier post here).

FFY00 · 2020-09-11T00:01:42Z

I will keep this short, otherwise I will rant about this. It happens because of the presence of pyvenv.cfg which overrides the PYTHONHOME behavior.

FFY00 · 2020-09-11T00:25:24Z

Opened pypa/pyproject-hooks#92 which is required to fix this issue.

FFY00 · 2020-09-11T00:30:15Z

Keeping it short. IMO PYTHONHOME should override pyvenv.cfg. PEP 405 tried to improve things by introducing pyvenv.cfg but it made PYTHONHOME unreliable. I would actually call this a breaking change, it should have never happened between minor versions. I am very bummed out from finding out about this.

pganssle · 2020-09-11T02:56:32Z

I would actually call this a breaking change, it should have never happened between minor versions.

You mean PEP 405 was a breaking change? When PEPs propose a change Python semantics, they're never implemented in minor versions. Looks like PEP 405 was introduced in Python 3.3 (Python does not use semver — backwards-incompatible changes are allowed in point releases like 3.2 → 3.3).

Keeping it short. IMO PYTHONHOME should override pyvenv.cfg. PEP 405 tried to improve things by introducing pyvenv.cfg but it made PYTHONHOME unreliable.

This is a somewhat vague description of the problem. I don't really understand the isolation / environment code well enough to understand why this would be a problem. From what I can tell, PYTHONHOME refers to the location of the standard library, but the problem we're having is not from the standard library, it's from site-packages. I'll note that it's entirely possible to prevent access to these by removing the relevant directories from sys.path (though I'm not sure if it's possible to do that with environment variables).

Opened pypa/pep517#92 which is required to fix this issue.

One possible (very hacky) workaround in the meantime might be to monkey-patch sys.executable during the run in versions of pep517 that don't support using a custom executable.

FFY00 · 2020-09-11T14:45:39Z

You mean PEP 405 was a breaking change? When PEPs propose a change Python semantics, they're never implemented in minor versions. Looks like PEP 405 was introduced in Python 3.3 (Python does not use semver — backwards-incompatible changes are allowed in point releases like 3.2 → 3.3).

Yes, but AFAIK you don't change behaviors, you deprecate stuff and remove them, but I might be wrong. Changing how something behaves is very problematic.

In this case, I see absolutely no reason for pyvenv.cfg to be the first step of the search.

This is a somewhat vague description of the problem. I don't really understand the isolation / environment code well enough to understand why this would be a problem. From what I can tell, PYTHONHOME refers to the location of the standard library, but the problem we're having is not from the standard library, it's from site-packages. I'll note that it's entirely possible to prevent access to these by removing the relevant directories from sys.path (though I'm not sure if it's possible to do that with environment variables).

Okay, this is the issue:

(on /usr/bin/python, no pyvenv.cfg is found)

$ PYTHONHOME=/tmp/build-env-kit8jz7e PYTHONPATH=/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/lib-dynload:/usr/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages /usr/bin/python -m sysconfig | head -n 15
Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"

Paths:
	data = "/tmp/build-env-kit8jz7e"
	include = "/tmp/build-env-kit8jz7e/include/python3.8"
	platinclude = "/tmp/build-env-kit8jz7e/include/python3.8"
	platlib = "/tmp/build-env-kit8jz7e/lib/python3.8/site-packages"
	platstdlib = "/tmp/build-env-kit8jz7e/lib/python3.8"
	purelib = "/tmp/build-env-kit8jz7e/lib/python3.8/site-packages"
	scripts = "/tmp/build-env-kit8jz7e/bin"
	stdlib = "/tmp/build-env-kit8jz7e/lib/python3.8"

(on a virtual environment, pyvenv.cfg is found)

$ PYTHONHOME=/tmp/build-env-kit8jz7e PYTHONPATH=/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/
lib-dynload:/usr/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages ~/.virtualenvs/test-env/bin/python -m sysconfig | head -n 15
Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"

Paths:
	data = "/home/anubis/.virtualenvs/test-env"
	include = "/tmp/build-env-kit8jz7e/include/python3.8"
	platinclude = "/tmp/build-env-kit8jz7e/include/python3.8"
	platlib = "/home/anubis/.virtualenvs/test-env/lib/python3.8/site-packages"
	platstdlib = "/home/anubis/.virtualenvs/test-env/lib/python3.8"
	purelib = "/home/anubis/.virtualenvs/test-env/lib/python3.8/site-packages"
	scripts = "/home/anubis/.virtualenvs/test-env/bin"
	stdlib = "/tmp/build-env-kit8jz7e/lib/python3.8"

So, if pyvenv.cfg is present, even if I provide a PYTHONHOME, which should relocate the Python installation to other path, the paths gets overridden. PYTHONHOME should override the prefix, but for some reason pyvenv.cfg gets precedence, making PYTHONHOME useless.

With PEP 405, the Python interpreter is not self-contained, it depends on external environment variables we might not control. I chose PYTHONHOME because it did not require us to run a different interpreter, but it turns out it does. I think PEP 405, which was motivated by the lack of symlink support in Windows, in its attempt to make things better killed this use case, I wish just a little bit of more thought had been put into it.

pganssle · 2020-09-11T15:18:34Z

Interesting, I wonder if it's worth tweaking the documentation to make it more clear that PYTHONHOME / pyvenv.cfg's home value is setting the base path for a whole bunch of stuff, not just the standard library. Maybe that's more obvious to people familiar with the way all those other paths are derived?

layday · 2020-09-15T14:31:17Z

Can you not use the Python -S flag to prevent the import of the site module, which will prevent the parsing of pyvenv.cfg? You can then manipulate PYTHONPATH normally, by breaking out of the virtual environment. Although not importing site might have other implications that I'm not aware of.

FFY00 · 2020-09-15T14:42:16Z

That does work, yes.

Paths:
	data = "/tmp/build-env-65qowjyv"
	include = "/tmp/build-env-65qowjyv/include/python3.8"
	platinclude = "/tmp/build-env-65qowjyv/include/python3.8"
	platlib = "/tmp/build-env-65qowjyv/lib/python3.8/site-packages"
	platstdlib = "/tmp/build-env-65qowjyv/lib/python3.8"
	purelib = "/tmp/build-env-65qowjyv/lib/python3.8/site-packages"
	scripts = "/tmp/build-env-65qowjyv/bin"
	stdlib = "/tmp/build-env-65qowjyv/lib/python3.8"

FFY00 · 2020-10-07T01:10:48Z

@pganssle can you confirm that the current master fixes the issue for you?

FFY00 · 2020-10-23T14:36:33Z

Ping. Should this be closed now?

gaborbernat · 2020-10-23T15:07:29Z

I think so.

FFY00 · 2020-10-23T15:17:38Z

Okay, we can reopen if needed.

pganssle mentioned this issue Sep 10, 2020

Missing backend error when PEP 518 requirements are not already satisfied #98

Closed

FFY00 changed the title ~~Isolated build environment has access to build's environment~~ Isolated build environment has access to build's environment when run from a Python environment Sep 11, 2020

pganssle mentioned this issue Sep 11, 2020

env: fix islation when pyvenv.cfg is present #99

Closed

pganssle mentioned this issue Sep 14, 2020

Migrate from pep517 to using build in tests and workflows scikit-hep/pyhf#1065

Closed

pganssle mentioned this issue Sep 30, 2020

Old setuptools may be used when "setuptools" is required (more likely to affect older Pythons like 3.7) #108

Closed

henryiii mentioned this issue Sep 30, 2020

build uninstalls setuptools in a venv #109

Closed

gaborbernat mentioned this issue Oct 4, 2020

Debian disables ensurepip #106

Closed

FFY00 added the bug Something isn't working label Oct 5, 2020

FFY00 added the release blocker label Oct 7, 2020

pganssle mentioned this issue Oct 20, 2020

Failure to build a package within Docker with "No such file or directory: 'dist'" #152

Closed

FFY00 closed this as completed Oct 23, 2020

pypa locked and limited conversation to collaborators Jan 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Isolated build environment has access to build's environment when run from a Python environment #97

Isolated build environment has access to build's environment when run from a Python environment #97

pganssle commented Sep 10, 2020

FFY00 commented Sep 10, 2020

pganssle commented Sep 10, 2020

FFY00 commented Sep 11, 2020

FFY00 commented Sep 11, 2020

FFY00 commented Sep 11, 2020 •

edited

Loading

pganssle commented Sep 11, 2020 •

edited

Loading

FFY00 commented Sep 11, 2020

pganssle commented Sep 11, 2020

layday commented Sep 15, 2020 •

edited

Loading

FFY00 commented Sep 15, 2020

FFY00 commented Oct 7, 2020

FFY00 commented Oct 23, 2020

gaborbernat commented Oct 23, 2020

FFY00 commented Oct 23, 2020

Isolated build environment has access to build's environment when run from a Python environment #97

Isolated build environment has access to build's environment when run from a Python environment #97

Comments

pganssle commented Sep 10, 2020

FFY00 commented Sep 10, 2020

pganssle commented Sep 10, 2020

FFY00 commented Sep 11, 2020

FFY00 commented Sep 11, 2020

FFY00 commented Sep 11, 2020 • edited Loading

pganssle commented Sep 11, 2020 • edited Loading

FFY00 commented Sep 11, 2020

pganssle commented Sep 11, 2020

layday commented Sep 15, 2020 • edited Loading

FFY00 commented Sep 15, 2020

FFY00 commented Oct 7, 2020

FFY00 commented Oct 23, 2020

gaborbernat commented Oct 23, 2020

FFY00 commented Oct 23, 2020

FFY00 commented Sep 11, 2020 •

edited

Loading

pganssle commented Sep 11, 2020 •

edited

Loading

layday commented Sep 15, 2020 •

edited

Loading