pyproject.toml-based builds lose proxy config for build-dependency installation

[mjpieters]

Environment

• pip version: 18.1
• Python version: 3.7.1
• OS: MacOS, Windows

Description

When behind a proxy, installing a project with a pyproject.toml defining a build-system.requires option will fail, because the BuildEnvironment.install_requirements() implementation runs pip as a child process without copying across proxy configuration.

Expected behavior

Even in an isolated build environment, proxy access needs to be configured, so that build dependencies can be downloaded.

How to Reproduce

Block direct access to PyPI, and use a proxy to install pyinstaller, e.g. pip install pyinstaller --proxy=http://webdefence.global.blackspider.com:80 --trusted-host=pypi.python.org.

Disabling build isolation with --no-build-isolation and manually installing the build requirements first lets you work around this issue.

Output

$ pip install pyinstaller --proxy=http://webdefence.global.blackspider.com:80 --trusted-host=pypi.python.org
Collecting pyinstaller
  Using cached https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz
  Installing build dependencies ... error
  Complete output from command c:\users\blaggr\appdata\local\programs\python\python37-32\scripts\python.exe -m pip install --ignore-installed --no-user --prefix C:\Users\blaggr\AppData\Local\Temp\pip-build-env-vd3w15r1 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --trusted-host pypi.python.org -- setuptools wheel:
  Collecting setuptools
    Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
    Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
    Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
    Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
    Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
    Could not find a version that satisfies the requirement setuptools (from versions: )
  No matching distribution found for setuptools

I'm filing this ticket as a follow-up to a Stack Overflow question, where a Windows user ran into this problem. They shared their pip -vvv output to help diagnosing.

[pradyunsg]

Thanks for filing this! ^>^

[mjpieters]

This is the same basic issue as #5502, where network configuration switches are not being forwarded to the isolated child process.

[gaborbernat]

I also found out that --no-cache-dir is also not forwarded, or cache dir in general.

[chrisinmtown]

@mjpieters you suggest as a workaround using flag --no-build-isolation. Please say, where/how do I implement that workaround? Forgive my ignorance about .toml files, the section that seems to cause the problem is this one in my case:

[build-system]
requires = ["poetry_core>=1.0", "setuptools"]
build-backend = "poetry.core.masonry.api"

The first build step tries to install these but fails in my proxy-afflicted environment.

[turribeach]

We seem to be affected by this issue but it doesn't seem that it has got any traction, given that it's nearly 4 years old. Any ideas when it will be fixed? Thanks

[uranusjr]

It’d be awesome if you’d be interested in working on it and submit a pull request! Someone being actively affected is in the best position to debug the issue and test the changes.

[turribeach]

I will be happy to test any potential fixes but I got no idea where the problem is and I am not a hardcore Python developer, I just write some occasional Python.

[pfmoore]

To be clear, this is probably going to wait until someone who is affected by the issue (and therefore has an environment that can be used to test potential solutions) can create a PR. None of the pip developers work in an environment where this functionality is needed, so we're dependent on the community for help here.

I'll add the "awaiting PR" label to the issue to reflect the position here.

[turribeach]

I will be happy to do a PR but I got no idea where the bug is. Any pointers?

[pfmoore]

See the OP's comment:

When behind a proxy, installing a project with a pyproject.toml defining a build-system.requires option will fail, because the BuildEnvironment.install_requirements() implementation runs pip as a child process without copying across proxy configuration.

I don't have much more than that. But please understand that this isn't a simple "find the bug and fix it". There's a question of what command line options should be copied over, should the related PIP_xxx environment variables be respected in the build environment setup, etc. These sorts of questions are the things that I can't answer, precisely because I don't need this functionality. I can at best say what seems sensible to me, if someone proposes an approach, but even then I wouldn't be sure.

[chrisinmtown]

Update Dec 2022: with suitable options for trusted hosts, pip version 22.3.1 on python 3.9.12 happily installs packages to a virtualenv on my MacOS laptop in my proxy-afflicted network environment. I cannot reproduce the problem that @mjpieters posted in 2018. Please note I have no ~/.pip/pip.conf file at all. First here's a transcript of a failure when no proxy server is used:

% pip install pyinstaller --trusted-host=pypi.org --trusted-host=files.pythonhosted.org   
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pyinstaller/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pyinstaller/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pyinstaller/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pyinstaller/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pyinstaller/
ERROR: Could not find a version that satisfies the requirement pyinstaller (from versions: none)
ERROR: No matching distribution found for pyinstaller

And here's the success, I anonymized the proxy server domain name (obvs):

% pip install pyinstaller --proxy=http://the.proxy.mybigcompany.com:8080 --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
Collecting pyinstaller
  Downloading pyinstaller-5.7.0-py3-none-macosx_10_13_universal2.whl (923 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 923.6/923.6 kB 3.3 MB/s eta 0:00:00
Requirement already satisfied: macholib>=1.8 in /Users/chris/.virtualenvs/py39/lib/python3.9/site-packages (from pyinstaller) (1.16.2)
Requirement already satisfied: pyinstaller-hooks-contrib>=2021.4 in /Users/chris/.virtualenvs/py39/lib/python3.9/site-packages (from pyinstaller) (2022.14)
Requirement already satisfied: setuptools>=42.0.0 in /Users/chris/.virtualenvs/py39/lib/python3.9/site-packages (from pyinstaller) (57.4.0)
Requirement already satisfied: altgraph in /Users/chris/.virtualenvs/py39/lib/python3.9/site-packages (from pyinstaller) (0.17.3)
Installing collected packages: pyinstaller
Successfully installed pyinstaller-5.7.0

So maybe this is no longer a problem? Happy new year everyone.

[pradyunsg]

Thanks @chrisinmtown! I'll close this then. :)

For posterity, if this is locked and you're hitting this issue, check that you have the latest pip version and, if so, please file a new issue with clear reproduction steps. :)

[chrisinmtown]

@pradyunsg perhaps you could please give this a little more time? I'm only reporting that pip seems to work behind a proxy when invoked properly. The original problem description states that a pyproject.toml configuration fails. However I don't see all the details for that situation. I'd be glad to test if someone could provide those details.

[mjpieters]

@chrisinmtown Your setup has a local cache so never has to download the requirements. Can you retest without the cache? The proxy was used to download the initial package but the build isolation sandbox never had to use the proxy.

I'd move the cache aside rather than clear it; pip cache dir gives you the location. While it might be possible to use configuration to point to a temporary cache location for a test, I'm not 100% certain that such config will make it to the isolated build environment, so it'd be preferable that the standard cache location is empty for the test.

[pradyunsg]

It’s not using the cache — pip prints “Using cached …” when it uses the cache.

[chrisinmtown]

FWIW I tried to clear the pip cache before I ran the tests posted above, like this:

% pip cache purge
Files removed: 1

Thanks for the directions to use pip cache dir, I removed that directory entirely:

% rm -fr ~/Library/Caches/pip

I then uninstalled pyinstaller and reran the test with exactly the same result as posted above. Then I removed the cache yet again, uninstalled that package's dependencies and ran yet again, I watched all get downloaded successfully:

% pip install pyinstaller --proxy=http://the.proxy.mybigcompany.com:8080 --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
Collecting pyinstaller
  Downloading pyinstaller-5.7.0-py3-none-macosx_10_13_universal2.whl (923 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 923.6/923.6 kB 1.5 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=42.0.0 in /Users/chris/.virtualenvs/py39/lib/python3.9/site-packages (from pyinstaller) (57.4.0)
Collecting altgraph
  Downloading altgraph-0.17.3-py2.py3-none-any.whl (21 kB)
Collecting macholib>=1.8
  Downloading macholib-1.16.2-py2.py3-none-any.whl (38 kB)
Collecting pyinstaller-hooks-contrib>=2021.4
  Downloading pyinstaller_hooks_contrib-2022.14-py2.py3-none-any.whl (252 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 252.6/252.6 kB 2.6 MB/s eta 0:00:00
Installing collected packages: altgraph, pyinstaller-hooks-contrib, macholib, pyinstaller
Successfully installed altgraph-0.17.3 macholib-1.16.2 pyinstaller-5.7.0 pyinstaller-hooks-contrib-2022.14

[pradyunsg]

Taking a more careful look, this issue is about projects that need to be built via pyproject.toml-based builds, and your example isn't performing such a build; it's using a compatible .whl file directly.

[chrisinmtown]

@pradyunsg so the right test to run is installing a Python package "A" that depends on a package "B" for which no pre-built .whl file is available? Do you have a candidate for "A" that I should try?

[pradyunsg]

pip install https://files.pythonhosted.org/packages/a3/50/c4d2727b99052780aad92c7297465af5fe6eec2dbae490aa9763273ffdc1/pip-22.3.1.tar.gz

You "just" need something that'd get built from source -- passing a source tarball containing a pyproject.toml file is sufficient.

[chrisinmtown]

Thanks @pradyunsg for the new information. It looks like the proxy config is not preserved, the problem apparently still exists. Here's the output:

% pip install https://files.pythonhosted.org/packages/a3/50/c4d2727b99052780aad92c7297465af5fe6eec2dbae490aa9763273ffdc1/pip-22.3.1.tar.gz --proxy=http://the.proxy.mybigcompany.com:8080 --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
Collecting https://files.pythonhosted.org/packages/a3/50/c4d2727b99052780aad92c7297465af5fe6eec2dbae490aa9763273ffdc1/pip-22.3.1.tar.gz
  Downloading pip-22.3.1.tar.gz (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 2.1 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
      ERROR: No matching distribution found for setuptools
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[chrisinmtown]

I tried to follow @mjpieters suggestion that the install_requirements function needs to pass proxy information to the pip subprocess. I don't immediately see a source for that. I added it very crudely at line 265 and then the install of dependencies worked:

        args.extend(["--proxy", "http://the.proxy.mybigcompany.com:8080"])

I see that the finder object (class PackageFinder) has a trusted_host attribute but no proxy attribute. Please give me your opinion, does the PackageFinder class need to be extended? Or is the proxy available from a different place?

[chrisinmtown]

Thanks @q0w for the suggestion. I tried getting the PIP_PROXY environment variable like this without success:

        proxy = os.environ.get("PIP_PROXY")
        if proxy:
            args.extend(["--proxy", proxy])

[q0w]

No, without modifying pip code? Does it work wit PIP_ env variables

[chrisinmtown]

Yes, setting the PIP_PROXY environment variable first then running the install without the --proxy command-line argument works in my environment. Here's the transcript.

% export PIP_PROXY=http://the.proxy.mybigcompany.com:8080
% pip install https://files.pythonhosted.org/packages/a3/50/c4d2727b99052780aad92c7297465af5fe6eec2dbae490aa9763273ffdc1/pip-22.3.1.tar.gz --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
Collecting https://files.pythonhosted.org/packages/a3/50/c4d2727b99052780aad92c7297465af5fe6eec2dbae490aa9763273ffdc1/pip-22.3.1.tar.gz
  Downloading pip-22.3.1.tar.gz (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 851.7 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

For what it's worth, in my proxy-afflicted environment this install fails if PIP_PROXY is not set or if either trusted-host argument is missing.

[junqfisica]

This issue seems to be related to the following discussion:

#10739 (comment)

[notatallshaw]

One workaround not mentioned in this issue is that pip is using requests for making http calls, so anything that would cause requests to use a proxy will cause pip to use a proxy. E.g. you can set the standard environmental variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY.

[martinezlc99]

FYI - I do have a PR to add the --proxy option to the isolated build environment, which fixed my particular issue. I am wondering if I should also investigate issues #8999, #8439, #7805.