Which pip version does poetry use?

[timonbimon]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

I was wondering how poetry chooses which pip version it uses when installing new packages? And how can we make it choose a different one?

I am asking this question because of the following problem I encountered:

While trying to install "cryptography" in a Docker container with poetry I got the following error message:

Command ['/root/.cache/pypoetry/virtualenvs/hybrid-py3.7/bin/pip', 'install'  
, '--no-deps', 'cryptography==2.4.2'] errored with the following output:      
Collecting cryptography==2.4.2                                                
  Downloading https://files.pythonhosted.org/packages/f3/39/d3904df7c56f8654  
691c4ae1bdb270c1c9220d6da79bd3b1fbad91afd0e1/cryptography-2.4.2.tar.gz (468k  
B)                                                                            
  Could not find a version that satisfies the requirement cffi!=1.11.3,>=1.7  
 (from versions: )                                                            
No matching distribution found for cffi!=1.11.3,>=1.7                         
You are using pip version 10.0.1, however version 18.1 is available.    

I found the error a little bit weird in the first place, because it does not occur when I run poetry install locally, but after a little bit of googling it seemed that the cffi problem was related to the pip version and should be fixable, by upgrading pip => which leads to my question: My dockerfile is based on python:3.7-alpine and has pip 18.1 installed, so why does poetry still try to use pip version 10.0.1?

Thanks a lot to anyone taking the time to read through this :)

Edit: To fix my issue, I just had to do poetry run pip install --upgrade pip before poetry install, but I guess my question where poetry gets the idea to use 10.0.1 from still holds? :)

[eblume]

This is most likely related: #712

[asodeur]

I just ran into this as well and think it is a separate issue.

venv creates virtual environments using ensurepip which contain pip==10.0.1 and setuptools==39.0.1. This explains where pip==10.0.1 is comming from on Python >=3.3.

The next issue is that packages that require certain versions of pip or setuptools usually do not declare these as dependencies. Hence, IMHO cryptography should have a requirement pip>=X.Y.Z somewhere, maybe in setup.py/setup_requires or pyproject.toml/[build-system]/requires?

However, even if an explicit dependency on pip or setuptools is declared it is currently not honored by poetry. This is regardless of whether this is done in [tool.poetry.dependencies], setup.py, or [build-system]\nrequires = [...]. pip, setuptools, and distribute seem to get filtered out in two places:
puzzle.provider.Provider.UNSAFE_PACKAGES and repositories.installed_repository.InstalledRepository.load (would need to call pip freeze --all).

For now manually upgrading pip, setuptools, and distribute as decribed above seems to be the only way.

[asodeur]

Checked our packages and there are a lot with setuptools>=40.0.0 constraint (on Windows any package with src-layout). However, there are none with < or <= constraints on pip, distribute, setuptools, or wheels. I'd assume < or <= constraints on pip, distribute, setuptools, or wheels are very rare in general. Encountered no build system beyond setuptools and poetry.

As a tactical solution pip, setuptools, and wheels could be upgraded to the version found in the installation running poetry during venv creation. This is not fully deterministic as now the resulting installation depends on the interpreter running poetry at the time of virtual environment creation.

A "proper" implementation of setup requirements/build system might in a worst-case scenario require virtual environments separate from the target virtual environment to build packages. I'd suspect this would be a lot more effort and since today very few packages actually declare their setup requirements properly likely in vain.

@sdispater what do you think?

[sdispater]

@asodeur The thing is that it's a tricky subject. Poetry uses pip internally so allowing it to be manipulated as a dependency can lead to undesirable side effects.

Also, if setuptools, or pip for that matter, is treated like any other dependency, it can also lead to them being uninstalled if they are no longer present in the pyproject.toml, which would in turn make the virtualenv corrupted.

I am not sure what the best approach is to be honest.

[asodeur]

Ok, than I'd propose to simply upgrade pip and setuptools to the versions found in the base environment during venv creation for now. This is probably the least confusing behaviour, would resolve the issues encountered by @timonbimon and myself, and be fairly simple to implement (simple enough for me to create a pull request).

[nbigaouette-eai]

I am confused too by this. I wish poetry would use the latest pip version available (or simply the one already installed).

Here's the simplest reproducible case:

pyproject.toml:

[tool.poetry]
name = "piptest"
version = "0.1.0"
description = ""
authors = ["Nicolas Bigaouette <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

> poetry install -vv
Creating virtualenv piptest-py3.7 in /Users/nbigaouette/tmp/piptest/.venv
Using virtualenv: /Users/nbigaouette/tmp/piptest/.venv
Updating dependencies
Resolving dependencies...
   1: fact: piptest is 0.1.0
   1: derived: piptest
   1: selecting piptest (0.1.0)
   1: Version solving took 0.005 seconds.
   1: Tried 1 solutions.

Writing lock file

Nothing to install or update

> poetry run pip -V
pip 19.0.3 from /Users/nbigaouette/tmp/piptest/.venv/lib/python3.7/site-packages/pip (python 3.7)

But:

> ~/.python/3.7.3/bin/pip -V
pip 19.1.1 from /Users/nbigaouette/.python/3.7.3/lib/python3.7/site-packages/pip (python 3.7)

Since poetry doesn't uses pip from the initial Python install directory (the latest), the first step of every poetry package install is to actually upgrade pip, which seems to defeat poetry's goal (since it needs pip to do the install...)

[kummerer94]

The proposed solution does not seem to work anymore, at least for me. Whenever I try to upgrade pip inside of poetry, I run into the following error:

Collecting pip
Using cached https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 10.0.1
Uninstalling pip-10.0.1:
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: 'C:\Users\me\AppData\Local\Temp\pip-uninstall-32ke2pg9\users\me\appdata\local\pypoetry\cache\virtualenvs\my-package-py3.7\lib\site-packages\pip\_vendor\urllib3\packages\ssl_match_hostname\pycache\_implementation.cpython-37.pyc'

I was not able to find the root of this issue. Does this have something to do with the problems that poetry and pip have?

[brycedrennan]

As @asodeur pointed out this is an issue with ensurepip. You can see the related bug report and discussion in the python bug tracker.

It would be nice to fix this but I think any fix should maintain the ability to create virtualenvs without internet access. Perhaps it could be a command line arguement or setting?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[chdsbd]

This still seems like a valid bug.

[pavanmr94]

Hi, we are facing the same issue where we did a following on a fresh windows machine:

1. install python 3.7.5
2. downgrade pip to 18.1
3. install poetry 1.0.0b7 through get-poetry.py script
4. do a poetry install on a project that needs pip 18.1, has it as an explicit dependency
5. install fails, and it says using "pip 19.2.1"

Can you help me with this? How do i force poetry to use a specific pip version?

[pappasam]

@pavanmr94 a hacky workaround to get you through these dark times:

$ # Upgrades pip in a newly-installed virtualenv
$ poetry run pip install pip==18.1
$ # Installs your project dependencies, but with the correct version of pip
$ poetry install

[pavanmr94]

@pavanmr94 a hacky workaround to get you through these dark times:

$ # Upgrades pip in a newly-installed virtualenv
$ poetry run pip install pip==18.1
$ # Installs your project dependencies, but with the correct version of pip
$ poetry install

@pappasam Thanks, we ended up doing just that.

[TheButlah]

The inconsistencies experienced with poetry using an older version of pip than the system installed one/pyenv one are causing a lot of problems for me. For example, for an unknown reason pyqt5 only installs on the newer version of pip (tested with pip 20.0.2), but not the version of pip that comes with poetry (18.1).

Also, having an older version of pip than the system one means that someone might do pip install . to take advantage of pip 19's ability to install from a pyproject.toml file, but once inside poetry's venv, you can't do that anymore.

Additionally, no amount of workarounds in the pyproject.toml file seem to fix it:

[build-system]
requires = ["poetry>=1.0.3", "pip>=20.0.0"]
build-backend = "poetry.masonry.api"

#etc...
[tool.poetry.dependencies]
python = "^3.6"
pip = "^20.0.0"

(pip remains version 18)

I suggest as a temporary fix to upgrade poetry's vendored pip to a version at least as new as the system/pyenv one upon install

[lucasshiva]

The inconsistencies experienced with poetry using an older version of pip than the system installed one/pyenv one are causing a lot of problems for me. For example, for an unknown reason pyqt5 only installs on the newer version of pip (tested with pip 20.0.2), but not the version of pip that comes with poetry (18.1).

Also, having an older version of pip than the system one means that someone might do pip install . to take advantage of pip 19's ability to install from a pyproject.toml file, but once inside poetry's venv, you can't do that anymore.

Additionally, no amount of workarounds in the pyproject.toml file seem to fix it:

[build-system]
requires = ["poetry>=1.0.3", "pip>=20.0.0"]
build-backend = "poetry.masonry.api"

#etc...
[tool.poetry.dependencies]
python = "^3.6"
pip = "^20.0.0"

I suggest as a temporary fix to upgrade poetry's vendored pip to a version at least as new as the system/pyenv one upon install

Ran into the same issue and your solution fixed it for me.

[louwers]

I had to use pip install --upgrade pip setuptools to be able to use Poetry.

Otherwise I would get this error when installing a certian package that requires a new version of both (:

ImportError: cannot import name 'find_namespace_packages'

Poetry seems to ignore both build_requires in setup.py as well as requires = ["setuptools>=47.3.1"] in pyproject.toml.

[TheButlah]

Still running into issues with this on a daily basis - many packages fail to install in the older pip version, which causes poetry install to break

[IamGianluca]

I have the same issue. Lots of packages (e.g., opencv-contrib-python-headless) are failing to install because of the old version of pip.

[mirosval]

why was this closed? As far as I can tell this is still an issue.

I tried to install scipy on macOS, but because it does not work with the Accelerate backend as of 1.6.0, it can't be built. There is a wheel, but a newer pip is required to install it.

What is the recommended approach in this case?

[stinodego]

I am running into this as well; specifically with pyarrow version 3.0.0 which requires the latest pip to install on my MacOS. I'd be willing to look into creating a fix for this? Let me know if that's something you guys would appreciate.

[PetterS]

Maybe @abn can shed some light here?

[abn]

The pip version used by poetry for all virtual environments created after 1.1.0b1 uses the pip version bundled within the virtualenv package used by poetry. This is because since #2666 was merged poetry uses virtualenv and the bundled pip wheel when creating new virtual environenments.

All this said, depending on how you installed poetry (pip, pipx, get-poetry.py) and to what environment it was installed (system site, user site, virtualenvironment, poetry site), the version of pip that get triggerred will vary.

Further, with #2826, we are no longer installing pip into the virtual environments created. We use the pip wheels in-place from the virtualenv install. In future version's if your package requires an explicit version of pip, you need to define it as part of the development dependencies.

Hope this helps.

[austinbutler]

@abn at what point can we begin adding pip as a dev dependency?

[abn]

The change is in master and the next 1.2.0 pre-release should have it.

[vettrivikas]

Just install with ADMIN problem will solve

[AEHamrick]

On my work VPN pip installs packages successfully while poetry will not. On open internet both work just fine. Platform: Windows 10, pip 21.1, poetry 1.1.8, poetry-core 1.0.4

This is the error I get from poetry:

Updating dependencies
Resolving dependencies...

  ConnectionError

  ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

  at c:\users\[redacted]\appdata\local\programs\python\python38\lib\site-packages\requests\adapters.py:498 in send
      494¦                     low_conn.close()
      495¦                     raise
      496¦
      497¦         except (ProtocolError, socket.error) as err:
    ? 498¦             raise ConnectionError(err, request=request)
      499¦
      500¦         except MaxRetryError as e:
      501¦             if isinstance(e.reason, ConnectTimeoutError):
      502¦                 # TODO: Remove this in 3.0.0: see #2811

Could this be related to poetry using its own version of pip? I don't see this specific error searching the issues.

[ericriff]

Do we have an ETA for the 1.2 release to go live?

[drorata]

FWIW, it feels like a similar problem is still around. Certain packages (e.g. mypy) fail to install using poetry when connected to corporate network. It might be related to #6791

[neersighted]

The issue you have is unrelated -- it relates entirely to the issues I've linked you to in your previous issue.

For those spelunking, this change has been partially reverted. The present state of things is this:

• Poetry does not depend on pip being in the target environment; if it is missing, we will fall back to the embedded copy in the virtualenv module.
• However, if it is in the environment, we will use it.
• Poetry will manage pip -- you can explicitly depend on it now in your pyproject.toml.
• Poetry will always manipulate pip before managing any other packages to prevent breaking the copy of pip it uses for installer operations.

There are a lot of unrelated topics in the comments -- please start a Discussion or join Discord if you would like clarification on any of the points here.

[daviibf]

why was this closed? As far as I can tell this is still an issue.

I tried to install scipy on macOS, but because it does not work with the Accelerate backend as of 1.6.0, it can't be built. There is a wheel, but a newer pip is required to install it.

What is the recommended approach in this case?

I am having the exact same issue. Any workaround found for this ? @mirosval

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.