when going into complex version resolving, output conflicting versions

[nadavwe]

when pip outputs this message:
INFO: pip is looking at multiple versions of webencodings to determine which version is compatible with other requirements. This could take a while.
It does not give any information on the conflicting versions.

this requires going into constraining the versions manually and trying to find the culprit.

it would be nice to have a message describing what version conflict was found such that the compatibility search was necessary.

[nadavwe]

another suggestion would be to add a flag to not try to even run the resolver, but fail at the first conflict.

[DiddiLeija]

What is your pip version? Because this could be related to the new pip's resolver.

[nadavwe]

ye, it's a feature request for the new resolver.
i'm using 21.1.3 and 21.2.1.

[uranusjr]

Could you describe what exactly do you want to see? Maybe write a fake pip install output that shows what you are looking for.

[nadavwe]

OK, I kinda got carried away. here is the important bit:

pip install --show-conflicts a b
...
Processing .../a/a-1.1.0-py3-none-any.whl
Processing .../b/b-1.0.0-py3-none-any.whl
Processing .../requests/requests-2.19.1-py2.py3-none-any.whl
INFO: Cannot install a==1.1.0 and b==1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    a 1.1.0 depends on requests<2.20
    b 1.0.0 depends on requests>2.21
pip is looking at multiple versions of a to determine which version is compatible with other requirements. This could take a while.
Processing /Users/nadavwe/localpip/repo/simple/a/a-1.0.0-py3-none-any.whl
...

if you want/need all the details, here you go:

so we have project a:

• version 1.0.0 depends on requests
• version 1.1.0 depends on requests<2.20
  we also have project b, that depends on requests>2.21.
  when running pip install a b, I get the following:

pip install a b
...
Processing .../a/a-1.1.0-py3-none-any.whl
Processing .../b/b-1.0.0-py3-none-any.whl
Processing .../requests/requests-2.19.1-py2.py3-none-any.whl
INFO: pip is looking at multiple versions of a to determine which version is compatible with other requirements. This could take a while.
Processing /Users/nadavwe/localpip/repo/simple/a/a-1.0.0-py3-none-any.whl

so apparently some collision occurred otherwise pip wouldn't go fetching other versions. (we know that b does not support requests in such version).
now if i want to find what's with the lastest versions of a and b, i have to do:

pip install 'a==1.1.0' 'b==1.0.0'
...
Processing .../a/a-1.1.0-py3-none-any.whl
Processing .../b/b-1.0.0-py3-none-any.whl
Processing .../requests/requests-2.19.1-py2.py3-none-any.whl
INFO: pip is looking at multiple versions of a to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install a==1.1.0 and b==1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    a 1.1.0 depends on requests<2.20
    b 1.0.0 depends on requests>2.21

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

what I expect is that the ERROR message appears every time pip finds a conflict (or a least a flag. or in verbose output).
e.g. something like:

pip install --show-conflicts a b
...
Processing .../a/a-1.1.0-py3-none-any.whl
Processing .../b/b-1.0.0-py3-none-any.whl
Processing .../requests/requests-2.19.1-py2.py3-none-any.whl
INFO: Cannot install a==1.1.0 and b==1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    a 1.1.0 depends on requests<2.20
    b 1.0.0 depends on requests>2.21
pip is looking at multiple versions of a to determine which version is compatible with other requirements. This could take a while.
Processing /Users/nadavwe/localpip/repo/simple/a/a-1.0.0-py3-none-any.whl
...

Thanks!

[nadavwe]

looks like the relevant part to add the logging would be just before this line:
https://github.com/pypa/pip/blob/main/src/pip/_vendor/resolvelib/resolvers.py#L371
using the causes variable.
from the looks of it, it does look like there might be a need to change the reporter interface, adding start_backtracking(self, causes). or some other idea.
If you're OK with this plan I would love to open a PR to resolvelib and to pip itself.
Thanks!

[uranusjr]

Sounds like a reasonable change. We can work out the details in the PR.

[nadavwe]

cool! i'll post a pr in the upcoming days.

[tonkolviktor]

@nadavwe
I had a same issue and after some searching I've find that pip-compile from pip-tools prints the most useful message.
https://pypi.org/project/pip-tools/

As opposed to pip install, it will do some basic stuff (which in context of pip backtracking seems magical):

• It will tell you which is the conflicting dependency
• which libraries are depending on it
• best of all it is ~fast (1-2minues) not hours like pip install: New resolver takes a very long time to complete #9187

The only thing I do not understand why cannot the pip install use simply this, specially if it's already available.

[potiuk]

Great info. I will try pip-compile in the meantime of some of our recent apache-airflow investigation with celery upgrade :)

[pradyunsg]

Bundling this into #9254, since that is the same issue but older.