Add pip>=20.3 support

[atugushev]

1. Support pip=>20.3
2. Introduce new tox env: tox -e pipprevious
3. Update CI to run latest and previous versions of pip

Changelog-friendly one-liner: Add pip>=20.3 support.

Contributor checklist

• Provided the tests for the changes.
• Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[codecov]

Codecov Report

Merging #1216 (3b79e9d) into master (05c9ec0) will decrease coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1216      +/-   ##
==========================================
- Coverage   99.50%   99.44%   -0.07%     
==========================================
  Files          36       36              
  Lines        2855     2863       +8     
  Branches      332      336       +4     
==========================================
+ Hits         2841     2847       +6     
- Misses          8        9       +1     
- Partials        6        7       +1     

Impacted Files	Coverage Δ
piptools/repositories/pypi.py	94.86% <100.00%> (-0.65%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05c9ec0...3b79e9d. Read the comment docs.

[AndydeCleyre]

@atugushev Looks like we've got to add compatibility for the new merging of download_dir and wheel_download_dir. Please see my added and reverted commit, which I think would be fine if we were only supporting >=20.3.

I'll try to get back to this very soon to make it compatible with older releases as well.

EDIT: Please see the new commit.

[AndydeCleyre]

Ideally we can push a release out before the pip 20.3 release, which is imminent.

[groodt]

The 2.3 beta 1 is now on pypi https://pypi.org/project/pip/20.3b1/

[AndydeCleyre]

OK, we've got at least one more recent change in pip that require attention here:

• parse_requirements drops its comes_from argument Remove unused comes_from argument of parse_requirements pypa/pip#8551 -- but I don't think we use that. comes_from is used elsewhere for other things, which should be fine.
• _resolve_one is not present in the new resolver -- this needs addressing. I'm having a look now.

[AndydeCleyre]

Notes on the _resolve_one issue:

• that method remains in the legacy resolver, which may be used by users of pip>=20.3, so a version check isn't going to cut it.
• I don't know how our support for choosing the resolver will really be, but my priority is keeping this usable at all with the new pip and new resolver.

[AndydeCleyre]

I'm trying to understand the use and purpose of _resolve_one, by pip-tools. It was introduced in 1db139c, which says:

Incorporate changes from #666 (comment)

I welcome any extra notes from @techalchemy

Link to that comment

I don't understand yet. Obviously it resolves dependencies. It partially may be about filtering direct requirements from the results. It also looks like it's used to "prepare" the reqs.

For now I'm trying using pip's resolver's resolve method instead while working it out.

I also now realize that the pip version checks we already have in this PR are not going to cut it when the user has newer pip, but passes --pip-args --use-deprecated=legacy-resolver. I don't know a good way to detect which resolver is active -- some isinstance check, or attribute check, or something? For now I'm catching AttributeErrors to guess, but hopefully we can work out something better.

We may want to modify most or all of the compile cli tests to run again with that option, if the pip version is new enough...

[groodt]

Looks like it's going GA very soon: pypa/pip#8936 (comment)

[pradyunsg]

Ask me anything you want, regarding the new resolver in pip and the internal model/design of the various components (from pip's PoV)! :)

[atugushev]

@AndydeCleyre thank you very much for helping me with pip-20.3 🙏

I don't know a good way to detect which resolver is active -- some isinstance check, or attribute check, or something? For now I'm catching AttributeErrors to guess, but hopefully we can work out something better.

It seems for now we should pass --use-deprecated=legacy-resolver option to the InstallComamnd to force pip to use legacy resolver until pip-tools would support new resolver. See a1e34a7.

---

@pradyunsg thank you! Your assistance would help a lot when we start to work on the new resolver!

[AndydeCleyre]

I'm sorry I haven't been more helpful on this.

[atugushev]

It's ready for review, and yeah it's better to merge and release this before pip-20.3 being released (which is today 😰)

[vphilippon]

LGTM 👍

[atugushev]

@vphilippon thanks for reviewing this! I'll merge this then eagerly if nobody minds.

[atugushev]

Ah, no I can't. Travis is still pending since yesterday :(

[vphilippon]

@atugushev Yeah I saw that. I cancelled and retried the build in hope to get it unstuck, but to no avail.
I was going to ask if that was something you knew about.

[atugushev]

@vphilippon not that I'm aware of. Maybe we have reached free credit for free usage? See also jazzband/help#206 (comment)

[pradyunsg]

I'd suggest turning Travis CI off, because they have an open source plan "in progress" and not actually rolled out to any project AFAIK. The one-time credit limit they announced is like 16 hours of CI time, which just isn't enough.

[pradyunsg]

#1229 is the relevant PR. :)

[hugovk]

I'd suggest turning Travis CI off, because they have an open source plan "in progress" and not actually rolled out to any project AFAIK. The one-time credit limit they announced is like 16 hours of CI time, which just isn't enough.

Raku is the only project I've seen which has been granted something:

• https://twitter.com/jjmerelo/status/1324272244349173760

I've applied but not got anything yet. But, yeah, #1229 :)

[groodt]

Thanks for the quick turnaround all!