pip resolver looks into older versions and seems to take forever #10373
Labels
type: bug
A confirmed bug or unintended behavior
Comments
atodorov
added
S: needs triage
|
Originally reported as python-social-auth/social-app-django#350 b/c I thought that upgrading to social-auth-app-django==5.0.0 is causing the problem. However from my comments at python-social-auth/social-app-django#350 (comment) that doesn't seem the case. In particular:
I will try pinning pip to an older version to see how it goes and will post comments here. |
|
With pip 21.2.4 (default when I create my venv) and the branch above which uses social-auth-app-django==5.0.0 I get the same results - pip starts backtracking into older versions of cryptography. With pip 21.2.4 (default when I create my venv) and the master branch which uses social-auth-app-django==4.0.0 the transaction completes and it looks like we're using latest versions of all packages. In particular cryptography==3.4.7 which is the latest. |
atodorov
added a commit
to kiwitcms/enterprise
that referenced
this issue
Aug 17, 2021
atodorov
added a commit
to kiwitcms/enterprise
that referenced
this issue
Aug 17, 2021
|
Same problem for me after updating from 19.3.1 to 21.2.4 |
|
We're also seeing this with pip 21.2.4 and Python 3.8.x. The resolver now takes so long, our build system hits the 2 hour timeout 😞 |
|
Facing the same problem, Any Fixes? 😢 |
|
Hi all, I reproduced OPs issue by simplifying it to a single requirements file: I then just used the pip download command so it doesn't affect my environment: I have been experimenting with optimizations for pip in these cases that have large potential solution spaces and backtracking is required: #10201 (comment) My testing indicates it resolves this test case but I wasn't able to reproduce your environment exactly so I would appreciate if you could test yourselves. It's worth noting that it still had to do some heavy backtracking to kiwitcms 8.9, here is my full output, it may help you to know the versions I could install so you can build a better requirements file: |
It's a freshly created venv with 3.8, nothing special there. Maybe Python versions do play a little difference but I was able to reproduce both on Fedora and Ubuntu.
That's not what I want and definitely not what earlier versions of pip were doing. I want the latest version of kiwitcms, which is 10.3 at the time of writing. Why does pip start backtracking in the first place and goes all the way back to kiwitcms-8.9 ? Also what's different between the latest pip version and 21.1.3 which is proposed as workaround and seems to not suffer from this issue ? |
Earlier versions of pip installed incompatible requirements, new versions of pip (20.3 onward) no longer support installing incompatible requirements. If you want a specific version of kiwitcms you must specify it in your requirements file, e.g. But testing adding Pip therefore does not support installing these requirements and you need to fix the requirements to install kiwitcms 10.3. Or if you want to install incompatible requirements you must force install them with no dependencies one at a time and completely skip pip's resolution engine.
Between Pip 21.1 and 21.2 a new feature that was added that pip respects the order of the users requirements when attempting to pin requirements. Requirements near the top of the list are attempted to be pinned first. This theoretically gives the user more control over how it searches the total possible solution space. But unfortunately understanding the possible solution space and using this to give pip better hints on how to search is itself non-trivial. As far as I can tell though pip 21.1 still will backtrack on kiwitcms as there doesn't seem to be a recent matching solution with the rest of the requirements. That said the worst case for 21.1 and 21.2 are the same, so it just happens your use case was badly impacted, others might have been greatly improved. It's hard to develop good optimizations for the infinite possible solution space that is dependency requirements. |
Many thanks for pointing this out. I didn't see it before. We've managed to get this sorted out on our end by upgrading to pygithub 1.55 which also requires PyJWT>=2.0.0 so all fine now. Feel free to close this issue or leave it open if others have any more questions. |
1 task
|
Closing this, for the same reasons as #9187 (comment). |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
pip starts resolving dependencies and in the process starts backtracking into older versions which just takes so long that it never completes (in practice GitHub Actions just times out the job).
Expected behavior
pip completes in a reasonable time and installs the packages.
pip version
21.2.4
Python version
3.8.10 / 3.8.11
OS
Linux (Fedora/Ubuntu)
How to Reproduce
On Ubuntu system
Output
The text was updated successfully, but these errors were encountered: