Modernizing the codebase after Python 2 #8802

pradyunsg · 2020-08-25T14:34:59Z

Looking forward into the post-Python-2-support world (i.e. post pip 20.3), we're going to want to modernize pip's codebase. That's going to involve a lot of things.

Consider this issue an attempt to try to create a singular location, to create the list of all of the things we'd want to free from the chains of maintaining Python 2 support. :)

Cleanup the mypy configuration.
Enum usage.
Drop __ne__ when not needed.
Drop Python-2 compatibility-related vendored modules.

The text was updated successfully, but these errors were encountered:

pradyunsg · 2020-08-25T14:36:59Z

Searching for sys.version_info gives 115 results in 58 files.
Searching for Py(thon)? ?2 gives 607 results in 152 files.

This is gonna be a lot of work. :)

hugovk · 2020-09-04T13:51:22Z

Run https://github.com/asottile/pyupgrade --py36
Check try/excepts, especially ImportErrors

obi1kenobi · 2021-01-24T05:35:48Z

While looking into the "cleaning up the mypy config" task mentioned here, I happened to come across an issue with how mypy is being used, so I opened #9502. In particular, I found that issue by attempting to improve the mypy config via typing_copilot: https://github.com/obi1kenobi/typing-copilot

It is a tool designed to help projects gradually move toward strict mypy compliance: it auto-detects the strictest passing mypy config for the current state of the codebase, and can then set up a "ratcheting" mechanism for type correctness where type quality can only improve in the future. With typing_copilot, you can set up a CI check enforcing that any time the mypy configuration can be made stricter, it is done so — tightening the ratchet whenever possible.

I am the maintainer of typing_copilot, and while it is a relatively young tool, it is already being used across more than 100kLOC of Python across a few projects. While supporting pip's vendorized dependencies would require some additional configuration functionality in typing_copilot, I'd be happy to add that functionality if it would be useful to the pip project or other core Python ecosystem tools.

I'd also be happy to help improve the mypy config used in pip, including helping resolve #9502 itself.

Thanks for all your hard work building and maintaining pip!

215: Update pip to 21.0.1 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.3.3** to **21.0.1**. <details> <summary>Changelog</summary> ### 21.0.1 ``` =================== Bug Fixes --------- - commands: debug: Use packaging.version.parse to compare between versions. (`9461 <https://github.com/pypa/pip/issues/9461>`_) - New resolver: Download and prepare a distribution only at the last possible moment to avoid unnecessary network access when the same version is already installed locally. (`9516 <https://github.com/pypa/pip/issues/9516>`_) Vendored Libraries ------------------ - Upgrade packaging to 20.9 ``` ### 21.0 ``` ================= Deprecations and Removals ------------------------- - Drop support for Python 2. (`6148 <https://github.com/pypa/pip/issues/6148>`_) - Remove support for legacy wheel cache entries that were created with pip versions older than 20.0. (`7502 <https://github.com/pypa/pip/issues/7502>`_) - Remove support for VCS pseudo URLs editable requirements. It was emitting deprecation warning since version 20.0. (`7554 <https://github.com/pypa/pip/issues/7554>`_) - Modernise the codebase after Python 2. (`8802 <https://github.com/pypa/pip/issues/8802>`_) - Drop support for Python 3.5. (`9189 <https://github.com/pypa/pip/issues/9189>`_) - Remove the VCS export feature that was used only with editable VCS requirements and had correctness issues. (`9338 <https://github.com/pypa/pip/issues/9338>`_) Features -------- - Add ``--ignore-requires-python`` support to pip download. (`1884 <https://github.com/pypa/pip/issues/1884>`_) - New resolver: Error message shown when a wheel contains inconsistent metadata is made more helpful by including both values from the file name and internal metadata. (`9186 <https://github.com/pypa/pip/issues/9186>`_) Bug Fixes --------- - Fix a regression that made ``pip wheel`` do a VCS export instead of a VCS clone for editable requirements. This broke VCS requirements that need the VCS information to build correctly. (`9273 <https://github.com/pypa/pip/issues/9273>`_) - Fix ``pip download`` of editable VCS requirements that need VCS information to build correctly. (`9337 <https://github.com/pypa/pip/issues/9337>`_) Vendored Libraries ------------------ - Upgrade msgpack to 1.0.2. - Upgrade requests to 2.25.1. Improved Documentation ---------------------- - Render the unreleased pip version change notes on the news page in docs. (`9172 <https://github.com/pypa/pip/issues/9172>`_) - Fix broken email link in docs feedback banners. (`9343 <https://github.com/pypa/pip/issues/9343>`_) .. note You should *NOT* be adding new change log entries to this file, this file is managed by towncrier. You *may* edit previous change logs to fix problems like typo corrections or such. To add a new change log entry, please see https://pip.pypa.io/en/latest/development/contributing/#news-entries .. towncrier release notes start ``` ### 20.3.4 ``` =================== Features -------- - ``pip wheel`` now verifies the built wheel contains valid metadata, and can be installed by a subsequent ``pip install``. This can be disabled with ``--no-verify``. (`9206 <https://github.com/pypa/pip/issues/9206>`_) - Improve presentation of XMLRPC errors in pip search. (`9315 <https://github.com/pypa/pip/issues/9315>`_) Bug Fixes --------- - Fixed hanging VCS subprocess calls when the VCS outputs a large amount of data on stderr. Restored logging of VCS errors that was inadvertently removed in pip 20.2. (`8876 <https://github.com/pypa/pip/issues/8876>`_) - Fix error when an existing incompatibility is unable to be applied to a backtracked state. (`9180 <https://github.com/pypa/pip/issues/9180>`_) - New resolver: Discard a faulty distribution, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9203 <https://github.com/pypa/pip/issues/9203>`_) - New resolver: Discard a source distribution if it fails to generate metadata, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9246 <https://github.com/pypa/pip/issues/9246>`_) Vendored Libraries ------------------ - Upgrade resolvelib to 0.5.4. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <[email protected]>

rth · 2021-07-29T08:50:31Z

Another thing could be to move type annotations as comments to actual type annotations for Python 3.6+. For instance,

pip/src/pip/_internal/utils/wheel.py

Lines 86 to 87 in 58ca200

    
           def wheel_dist_info_dir(source, name): 
        
               # type: (ZipFile, str) -> str

Unless there are reasons to avoid doing this migration yet?

pradyunsg · 2021-07-29T08:59:00Z

That's in-progress. :)

https://github.com/pypa/pip/pulls?q=is%3Apr+sort%3Aupdated-desc+complete+type+annotations

@pombredanne

WARNING: fetchcode's vcs will not work on this commit. This is result of separating pip's code from changes made in scope of this repository. This should make it easier to track potentially replicated issues from pip when taking their vcs pkg. It also made cleaning up easier, due to some maintenance activities done in pip: - dropping Python 2 & 3.5 support pypa/pip#9189 - modernized code after above - partially done, tracked in: pypa/pip#8802 - added py3.9 support - updated vendored libraries (e.g. fixing CVE-2021-28363) multiple PRs pip._internal.vcs (and related code) changes: - Fetch resources that are missing locally: pypa/pip#8817 - Improve SVN version parser (Windows) pypa/pip#8665 - Always close stderr after subprocess completion: pypa/pip#9156 - Remove vcs export feature: pypa/pip#9713 - Remove support for git+ssh@ scheme in favour of git+ssh:// pypa/pip#9436 - Security fix in git tags parsing (CVE-2021-3572): pypa/pip#9827 - Reimplement Git version parsing: pypa/pip#10117 In next commits, most of pip's internals will be removed from fetchcode, leaving only vcs module with supporting code (like utils functions, tests (which will be added & submitted with this change)) This will allow for changes such as ability to add return codes (probably via futures) from long running downloads and other features. Switching to having own vcs module might also be a good call due to pip._internal.vcs close integration with pip's cli in vcs module (some pip code has been commented out in commit mentioned below) While generally copy-pasting code without history is bad idea, this commit follows precedence set in this repo by: 8046215 with exception that all changes to pip's code will be submitted as separate commits. It has been agreed with @pombredanne & @TG1999 that history from pip will be rebased on fetchcode by @pombredanne (thanks!). It will be done only for the files that are of concern for fetchcode to limit noise in git history. I'm leaving this commit without SoB intentionally, as this is not my work, but that of the many pip's authors: https://github.com/pypa/pip/blob/21.2.4/AUTHORS.txt License of pip: MIT (https://pypi.org/project/pip/)

@pombredanne

Please note that fetchcode's vcs might not work on this branch. Initially pip was commited without it's history and with few changes applied. This update approaches this differently- by commiting pip code in a single commit & applying changes on top of it in separate commits. While much of pip's code will be stripped from this repository, the goal of this is to make it easier to take changes from upstream, even after the code will be modified. While git-subtree could be used it brings it's own set of issues. Update should make it easier to track potentially replicated issues from pip when taking their vcs pkg. It also made cleaning up easier, due to some maintenance activities done in pip: - dropping Python 2 & 3.5 support pypa/pip#9189 - modernized code after above - partially done, tracked in: pypa/pip#8802 - added py3.9 support - updated vendored libraries (e.g. fixing CVE-2021-28363) multiple PRs pip._internal.vcs (and related code) changes between 20.1.1 and 21.2.4 - Fetch resources that are missing locally: pypa/pip#8817 - Improve SVN version parser (Windows) pypa/pip#8665 - Always close stderr after subprocess completion: pypa/pip#9156 - Remove vcs export feature: pypa/pip#9713 - Remove support for git+ssh@ scheme in favour of git+ssh:// pypa/pip#9436 - Security fix in git tags parsing (CVE-2021-3572): pypa/pip#9827 - Reimplement Git version parsing: pypa/pip#10117 In next commits, most of pip's internals will be removed from fetchcode, leaving only vcs module with supporting code (like utils functions, tests (which will be submitted alongside this change)) This will allow for changes such as ability to add return codes (probably via futures) from long running downloads and other features. Switching to having own vcs module might also be a good call due to pip._internal.vcs integration with pip's cli in vcs module (some pip code has been commented out in commit mentioned below) While generally copy-pasting code (rather than using submodules/subtrees etc) makes it harder to track, my git-foo is not great enough for me to attempt regrafting subset of pips history that is of note from fetchcode perspective. It has been agreed with @pombredanne & @TG1999 that history from pip will be regrafted on fetchcode by @pombredanne (thanks!). It will be done only for the files that are of concern for fetchcode to limit noise in git history. The code submitted in scope of this commit is work of many pip's authors that can bee seen here: https://github.com/pypa/pip/blob/21.2.4/AUTHORS.txt Pip is licensed under MIT (https://pypi.org/project/pip/) Signed-off-by: Alexander Mazuruk <[email protected]>

@pombredanne

Please note that fetchcode's vcs might not work on this branch. Initially pip was commited without it's history and with few changes applied. This update approaches this differently- by commiting pip code in a single commit & applying changes on top of it in separate commits. While much of pip's code will be stripped from this repository, the goal of this is to make it easier to take changes from upstream, even after the code will be modified. While git-subtree could be used it brings it's own set of issues. Update should make it easier to track potentially replicated issues from pip when taking their vcs pkg. It also made cleaning up easier, due to some maintenance activities done in pip: - dropping Python 2 & 3.5 support pypa/pip#9189 - modernized code after above - partially done, tracked in: pypa/pip#8802 - added py3.9 support - updated vendored libraries (e.g. fixing CVE-2021-28363) multiple PRs pip._internal.vcs (and related code) changes between 20.1.1 and 21.2.4 - Fetch resources that are missing locally: pypa/pip#8817 - Improve SVN version parser (Windows) pypa/pip#8665 - Always close stderr after subprocess completion: pypa/pip#9156 - Remove vcs export feature: pypa/pip#9713 - Remove support for git+ssh@ scheme in favour of git+ssh:// pypa/pip#9436 - Security fix in git tags parsing (CVE-2021-3572): pypa/pip#9827 - Reimplement Git version parsing: pypa/pip#10117 In next commits, most of pip's internals will be removed from fetchcode, leaving only vcs module with supporting code (like utils functions, tests (which will be submitted alongside this change)) This will allow for changes such as ability to add return codes (probably via futures) from long running downloads and other features. Switching to having own vcs module might also be a good call due to pip._internal.vcs integration with pip's cli in vcs module (some pip code has been commented out in commit mentioned below) While generally copy-pasting code (rather than using submodules/subtrees etc) makes it harder to track, my git-foo is not great enough for me to attempt regrafting subset of pips history that is of note from fetchcode perspective. It has been agreed with @pombredanne & @TG1999 that history from pip will be regrafted on fetchcode by @pombredanne (thanks!). It will be done only for the files that are of concern for fetchcode to limit noise in git history. The code submitted in scope of this commit is work of many pip's authors that can bee seen here: https://github.com/pypa/pip/blob/21.2.4/AUTHORS.txt Pip is licensed under MIT (https://pypi.org/project/pip/) Signed-off-by: Alexander Mazuruk <[email protected]>

hugovk · 2022-05-05T16:00:39Z

There's been a good number of PRs for this :)

I guess the bulk has been done, and of course modernisation can continue, should this tracking issue be closed now?

uranusjr · 2022-05-30T06:03:00Z

I’m just going to do it anyway.

pradyunsg added the type: maintenance Related to Development and Maintenance Processes label Aug 25, 2020

pradyunsg mentioned this issue Dec 12, 2020

Remove dependency on the external module mock #9266

Closed

pradyunsg mentioned this issue Dec 20, 2020

Drop support for Python 2.7 and Python 3.5 #9189

Merged

This was referenced Dec 22, 2020

Remove redundant Python 2.7 code #9335

Merged

Use Formatter.default_time_format after dropping PY2 #9367

Merged

Clean up urllib imports #9371

Merged

Replace Python 2 tmpdir removal workarounds #9373

Merged

The future is now #9388

Merged

hexagonrecursion mentioned this issue Feb 10, 2021

Drop python2 related stuff in tests #9579

Merged

uranusjr closed this as completed May 30, 2022

github-actions bot locked as resolved and limited conversation to collaborators Jun 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modernizing the codebase after Python 2 #8802

Modernizing the codebase after Python 2 #8802

pradyunsg commented Aug 25, 2020

pradyunsg commented Aug 25, 2020

hugovk commented Sep 4, 2020

obi1kenobi commented Jan 24, 2021

rth commented Jul 29, 2021

pradyunsg commented Jul 29, 2021

hugovk commented May 5, 2022

uranusjr commented May 30, 2022

Modernizing the codebase after Python 2 #8802

Modernizing the codebase after Python 2 #8802

Comments

pradyunsg commented Aug 25, 2020

pradyunsg commented Aug 25, 2020

hugovk commented Sep 4, 2020

obi1kenobi commented Jan 24, 2021

rth commented Jul 29, 2021

pradyunsg commented Jul 29, 2021

hugovk commented May 5, 2022

uranusjr commented May 30, 2022