pip-compile doesn't support the new pip resolver

[rouge8]

pip-compile doesn't support using the 2020 resolver.

Environment Versions

1. OS Type macOS
2. Python version: $ python -V: Python 3.7.8
3. pip version: $ pip --version: pip 20.2
4. pip-tools version: $ pip-compile --version: pip-compile, version 5.3.0

Steps to replicate

1. Create t.txt:

   --use-feature=2020-resolver
   boto3
   

2. Run pip-compile t.txt:

Traceback (most recent call last):
  File ".tox/pip-compile/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/scripts/compile.py", line 458, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/resolver.py", line 169, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/resolver.py", line 274, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/resolver.py", line 384, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/repositories/local.py", line 75, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 232, in get_dependencies
    download_dir, ireq, wheel_cache
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 184, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
AttributeError: 'Resolver' object has no attribute '_resolve_one'

Expected result

pip-compile completes successfully.

Actual result

pip-compile errors out.

[atugushev]

Hello @rouge8,

Thanks for the issue! Yes, currently, pip-tools doesn't support new pip's resolver. I think it makes sense to adopt it after the resolver is being stable so that we can use the internal API.

[rouge8]

Well pip never considers the internal API stable right? :)

For my purposes I don’t actually need the new resolver logic except to deal with pypa/pip#5780. If you don’t think the new resolver internals are stable enough for pip-tools yet, I’m sure I can come up with some hack to work around this issue. :)

[groodt]

I'm hitting this error too.

The new resolver is going to be the default from middle or end of October if I understand correctly.
pypa/pip#8937
(This will be very exciting)

It probably makes sense to ensure pip-tools works with both --use-feature=2020-resolver and --use-deprecated-feature=legacy-resolver

[AndydeCleyre]

pypa/pip#988 (comment):

We are aiming to release pip 20.3 about a week from now, on Wednesday or Thursday, Oct 28 or 29. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

@atugushev @graingert How much do you already know about what needs to be done here in pip-tools?

[ssbarnea]

This issue is critical as the new resolver is days away from becoming default. If you was one of those that added use-feature = 2020-resolver to your ~/.config/pip/pip.conf in order to benefit from extra conflict resolution in newer resolver you are unable to use pip-compile at all.

[hynek]

So weirdly the 5.4 release seems to have fixed this for Python 3. But it still breaks for Python 2 (I know, I know). Can someone confirm this behavior, or is it just me?

[atugushev]

@hynek pip-tools==5.4.0 adds only pip==20.3 support. I've tried locally py27, but works fine. Could you create an issue with a traceback?

[hynek]

I…uh…played a bit with the settings hence and forth and suddenly it started working and I can't break it anymore. 😳 It was 100% 5.4.0/20.3 before. I guess my computer is just haunted.

[webknjaz]

Well pip never considers the internal API stable right? :)

Can't we just use resolvelib directly instead of the pip's vendored copy? This way pip-tools wouldn't need to rely on the internal pip's resolver wrapper but use it directly. Its API is quite simple (I'm in the progress of integrating resolvelib into ansible-galaxy CLI, for example) — using it would mean having a stable public API to rely on but the behavior may differ from what a given version of pip does, of course.

[ssbarnea]

This bus is still a big PITA because even latest version will choke if you happen to define:

export PIP_USE_FEATURE=2020-resolver

This used to be a popular method to enable the new resolver before it was released, so we would detect issues at CI level.

[hynek]

I believe you can work around that for now using pip check?

[ssbarnea]

Well,.. I removed that line from that CI pipeline that was running pip-compile, but having to do this on multiple projects is no joy. We should still have to fix it. Sadly lots of people interested but nobody willing to do it.

[brycedrennan]

Is anyone working on this? Thinking about taking a look but don't want to duplicate work.

[groodt]

This should actually be working now. I think this issue could be closed.

Which version of pip are you seeing errors with? I use pip==21.0.1 (new resolver enabled by default) with the latest pip-compile and things work correctly.

[brycedrennan]

okay well I have something that works with pip install but not pip compile. I'll see if I can boil it down to the base case."

python 3.7.10
pip 21.0.1
pip-tools 6.0.1

[pohmelie]

I have simplest example:
req.txt:

asyncpg ~= 0.15.0
asyncpgsa >= 0.22.0

pip-compile fails with:

Could not find a version that matches asyncpg>=0.22.0,~=0.15.0 (from -r req.txt (line 1))
Tried: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.1, 0.6.3, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.20.0, 0.20.0, 0.20.1, 0.20.1, 0.21.0, 0.21.0, 0.22.0, 0.22.0
There are incompatible versions in the resolved dependencies:
  asyncpg~=0.15.0 (from -r req.txt (line 1))
  asyncpg>=0.22.0 (from asyncpgsa==0.27.1->-r req.txt (line 2))

but pip install succeed with:

...
Successfully installed asyncpg-0.15.0 asyncpgsa-0.27.0 greenlet-1.0.0 sqlalchemy-1.4.6

Tested on both pypi and github-master versions of pip-tools. Latest pip (since pip-tools required it).

If I change req.txt as:

asyncpg ~= 0.15.0
asyncpgsa >= 0.22.0, < 0.27.1

Then I get pip-compile succeed with:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile req.txt
#
asyncpg==0.15.0
    # via
    #   -r req.txt
    #   asyncpgsa
asyncpgsa==0.27.0
    # via -r req.txt
greenlet==1.0.0
    # via sqlalchemy
sqlalchemy==1.4.6
    # via asyncpgsa

[brycedrennan]

confirmed pip-compile can't handle,

boto3
botocore==1.20.20

but pip finds valid solution:

boto3==1.17.20
botocore==1.20.20

So sounds like no ones working on this?

[Eric-Arellano]

Hi, I'm a maintainer at https://github.com/pantsbuild/pants. We're working on a revamp of our 3rd-party support with lockfiles at the crux of the changes. We've been using pip-compile so far and are really happy with it so far, great UX!

This issue is a blocker for us to call the project complete. We're interested in taking on this issue, but are trying to scope it out: how much work would you anticipate this change will take?

Also, would anyone be available to help, like giving some tips or pair programming?

Finally, Pants still supports Python 2.7, so we'd ideally backport this feature to pip-tools 5, in addition to pip-tools 6. How feasible do you expect a backport like that would be?

[graingert]

@Eric-Arellano there's a new lock file spec here too https://www.python.org/dev/peps/pep-0665/

[ssbarnea]

Ohh yeah, it only took me to read the header to guess what this pep will propose: more TOML :( ... few pages later, I got my guess confirmed.

[atugushev]

#1539 adds support for new resolver and fixes the issue:

Details

❯ pip-compile --resolver backtracking
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --resolver=backtracking
#
boto3==1.21.4
    # via -r requirements.in
botocore==1.24.5
    # via
    #   boto3
    #   s3transfer
jmespath==0.10.0
    # via
    #   boto3
    #   botocore
python-dateutil==2.8.2
    # via botocore
s3transfer==0.5.1
    # via boto3
six==1.16.0
    # via python-dateutil
urllib3==1.26.8
    # via botocore

[atugushev]

The resolver is released as part of pip-tools v6.8.0 🎉

[matthewwritter]

@atugushev I can't say how much this saved the day for me - thank you for getting this in there when you did!