pip does not respect proxy when downloading "wheel" as a dependency

[SamuraiAku]

Environment

• pip version: 18.0
• Python version: 3.5.6
• OS: linux

Platform: Xilinx Ultrascale+

Description

When using --proxy, pip normally downloads any required dependencies for the requested package without issue. But there is an edge condition when the dependency is "wheel". In this case, the attempt to download errors out because pip is not going through the proxy. The workaround is to install wheel separately first. This was observed when trying to download and build scipy 1.4.1 from source.

Expected behavior

pip downloads wheel through the proxy.

How to Reproduce

1. From Linux on ARM, verify that wheel is not installed
2. pip install scipy --proxy
3. pip errors out
4. pip install wheel --proxy
5. pip install scipy --proxy

It's ok if the scipy installation errors out because you don't have the compilers and required libraries setup. The error described here occurs before you reach that point.

Output

home# pip3 install scipy --proxy proxy1.lmco.com:80 --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org Collecting scipy
  Downloading https://files.pythonhosted.org/packages/04/ab/e2eb3e3f90b9363040a3d885ccc5c79fe20c5b8a3caa8fe3bf47ff653260/scipy-1.4.1.tar.gz (24.6MB)
    100% |################################| 24.6MB 3.8MB/s
  Installing build dependencies ... error
  Complete output from command /usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-d3vvzxn3 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org -- wheel setuptools Cython>=0.29.13 "numpy==1.13.3; python_version=='3.5' and platform_system!='AIX'" "numpy==1.13.3; python_version=='3.6' and platform_system!='AIX'" "numpy==1.14.5; python_version=='3.7' and platform_system!='AIX'" "numpy==1.17.3; python_version>='3.8' and platform_system!='AIX'" "numpy==1.16.0; python_version=='3.5' and platform_system=='AIX'" "numpy==1.16.0; python_version=='3.6' and platform_system=='AIX'" "numpy==1.16.0; python_version=='3.7' and platform_system=='AIX'" "numpy==1.17.3; python_version>='3.8' and platform_system=='AIX'" pybind11>=2.4.0:
  Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version >= "3.8" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.5" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version >= "3.8" and platform_system == "AIX"' don't match your environment
  Collecting wheel
    Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f92a75860>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/wheel/
    Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f92a75710>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/wheel/
    Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f92a75630>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/wheel/
    Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f92a757b8>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/wheel/
    Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f92a75208>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/wheel/
    Could not find a version that satisfies the requirement wheel (from versions: )
  No matching distribution found for wheel

  ----------------------------------------
Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-d3vvzxn3 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org -- wheel setuptools Cython>=0.29.13 "numpy==1.13.3; python_version=='3.5' and platform_system!='AIX'" "numpy==1.13.3; python_version=='3.6' and platform_system!='AIX'" "numpy==1.14.5; python_version=='3.7' and platform_system!='AIX'" "numpy==1.17.3; python_version>='3.8' and platform_system!='AIX'" "numpy==1.16.0; python_version=='3.5' and platform_system=='AIX'" "numpy==1.16.0; python_version=='3.6' and platform_system=='AIX'" "numpy==1.16.0; python_version=='3.7' and platform_system=='AIX'" "numpy==1.17.3; python_version>='3.8' and platform_system=='AIX'" pybind11>=2.4.0" failed with error code 1 in None

[uranusjr]

More specifically, pip does not correctly propagate options to the subprocess when installing PEP 517 dependencies. Could you help test whether this behaviour still happens on the latest version?

[SamuraiAku]

I can if I’m able. How do I upgrade pip and will it work with python 3.5?

[uranusjr]

You can test it in a virtual environment, something like

python3.5 -m venv ./test-pip-7805
./test-pip-7805/bin/python install -m pip install --upgrade pip  # To the latest version.
./test-pip-7805/bin/pip scipy --proxy  # Test with the pip inside virtual environment...

[SamuraiAku]

I tried that out and got the same error

[benfiedler]

Another pip package where I encounter this same problem is python-augeas when dependency pip package cffi is not installed. Experienced in CentOS pip ver 8.1.2 (see my duplicate isssue saltstack/salt#57957 ) but even tried bumping pip to latest 20.1.1 and error persists.

$ pip install --upgrade pip --proxy my.proxy:1234
Collecting pip
  Downloading https://files.pythonhosted.org/packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 737kB/s
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
      Successfully uninstalled pip-8.1.2
Successfully installed pip-20.1.1

$ pip install python-augeas --proxy my.proxy:1234

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting python-augeas
  Using cached python-augeas-1.1.0.tar.gz (93 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-RXZKir/python-augeas/setup.py'"'"'; __file__='"'"'/tmp/pip-install-RXZKir/python-augeas/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-Zrtk1v
         cwd: /tmp/pip-install-RXZKir/python-augeas/
    Complete output (25 lines):
    Download error on https://pypi.python.org/simple/cffi/: [Errno 101] Network is unreachable -- Some packages may not be found!
    Couldn't find index page for 'cffi' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno 101] Network is unreachable -- Some packages may not be found!
    No local packages or download links found for cffi>=1.0.0
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-RXZKir/python-augeas/setup.py", line 43, in <module>
        test_suite="test.test_augeas",
      File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
        parse_requirements(requires), installer=self.fetch_build_egg
      File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
        dist = best[req.key] = env.best_match(req, self, installer)
      File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
        return self.obtain(req, installer) # try and download/install
      File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
        return installer(requirement)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
        return cmd.easy_install(req)
      File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 617, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi>=1.0.0')
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[cristina-rosca]

Any solution for this problem ? i'm facing this issue while installing gdown transitive dependencies

[aschor]

Hi, python 3.10.4 here. Same issue, how can I overcome this ? :

pip install -v --proxy=http://127.0.0.1:3128 setuptools py7zr
Using pip 22.1.1 from C:\Applications\Anaconda3\envs\mpp2\lib\site-packages\pip (python 3.10)
Requirement already satisfied: setuptools in c:\applications\anaconda3\envs\mpp2\lib\site-packages (62.3.2)
Collecting py7zr
  Using cached py7zr-0.18.6.tar.gz (4.9 MB)
  Running command pip subprocess to install build dependencies
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'Une connexion existante a dû être fermée par l’hôte distant', None, 10054, None))': /simple/setuptools/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'Une connexion existante a dû être fermée par l’hôte distant', None, 10054, None))': /simple/setuptools/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'Une connexion existante a dû être fermée par l’hôte distant', None, 10054, None))': /simple/setuptools/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'Une connexion existante a dû être fermée par l’hôte distant', None, 10054, None))': /simple/setuptools/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'Une connexion existante a dû être fermée par l’hôte distant', None, 10054, None))': /simple/setuptools/
  ERROR: Could not find a version that satisfies the requirement setuptools>=58 (from versions: none)
  ERROR: No matching distribution found for setuptools>=58
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: 'C:\Applications\Anaconda3\envs\mpp2\python.exe' 'C:\Users\VDMC5301\AppData\Local\Temp\1\pip-standalone-pip-znf6sfv9\__env_pip__.zip\pip' install --ignore-installed --no-user --prefix 'C:\Users\VDMC5301\AppData\Local\Temp\1\pip-build-env-i2ltdhl0\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=58' wheel 'setuptools_scm[toml]>=6.0.1'
  cwd: [inherit]
  Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

OK, I found a trick to overcome the issue. Like explained in the docs, I created the directory and file %APPDATA%\pip\pip.ini
and added this content (the proxy configuration). Then my package installs.

[global]
proxy = http://127.0.0.1:3128

[kielnino]

The solution from @aschor solved the problem for me!

[giulianisanches]

Just to update that with python 3.9 and the latest version of PIP the issue persists. I have used the environment variables http_proxy and https_proxy as a workaround, but I will try to change my pip.ini to apply the proxy just for the pip command.