gh-95299: Stop installing setuptools as a part of ensurepip and venv #101039

pradyunsg · 2023-01-14T17:50:08Z

This PR removes the bundled setuptools wheel from ensurepip, and stops installing setuptools in environments created by venv.

I based this PR off of my understanding of venv and ensurepip, and using the following search to locate all the relevant bits of code + ensuring that tests are happy locally.

In this PR, I've intentionally not touched the example in https://docs.python.org/3.12/library/venv.html#an-example-of-extending-envbuilder. That example is largely outdated and is likely better served being updated in a dedicated PR for it.

Issue: Remove bundled setuptools #95299

pradyunsg · 2023-01-14T18:52:01Z

Hmm... @vstinner @encukou any opinions on what I should do with test_cppext (added in #91321) here?

That test has an implicit assumption that setuptools will be present in the virtual environment created by venv (added in #92639, as part of removing reliance on distutils).

Notably, some leading questions that affect what direction I could take for updating that test:

Does this test need to use packaging tooling? (otherwise, I guess we can hand-craft building the extension and rely on that)
If it should use setuptools, would it be OK for this test to require reaching out over the network to fetch setuptools via pip?
Would it be OK to build the extension in pip-installable package, rather than using the deprecated approach of directly invoking setup.py?

pradyunsg · 2023-01-14T22:40:17Z

Since it's cheap to revert commits/remove them from a PR, and nice to have a code change to discuss details over, I've gone ahead and pushed a change that assumes that the answer to the questions I raised above is yes, yes and yes. :)

Mentioning a GitHub UI thing that I find very useful (and non-initutive): you can click "force-pushed" on the event above to see what changed in the force pushes between the hashes that GitHub lists in the from ... to ... at the end of the event.

.github/workflows/verify-ensurepip-wheels.yml

vsajip

The changes seem reasonable to me, though I'm only the code owner for the venv and test_venv bits.

encukou · 2023-01-16T13:51:03Z

I guess we can hand-craft building the extension and rely on that

IMO, that would be best. Relatedly, I've been thinking how to handle distutils removal in the extension tutorial -- it should definitely mention setuptools, but a simple but working example command line, because setuptools is not always available.
Having that in the tests -- even in a way that only satisfies our CI on select platforms -- would help :)
CPython tests depending on setuptools isn't great in general, and neither is downloading code as part of the test run.

Anyway, if that sounds like more than you signed up for, go ahead with Setuptools.
I don't have time for a full PR review now, sorry! Whoever gets around to it, please test-with-buildbots before merging.

pradyunsg · 2023-01-16T21:03:55Z

Anyway, if that sounds like more than you signed up for, go ahead with Setuptools.

I spent some time today looking into this after my work day, and I've not figured out the details in a portable manner. I'd prefer if this could be tackled in a follow up instead. :)

pradyunsg · 2023-01-16T21:07:10Z

/cc @dstufft @ncoghlan as other experts on ensurepip.

I'm technically listed as an expert on ensurepip already and I'm ~sure that the changes I've made to it are correct; but it certainly won't hurt if y'all can review this. :)

bedevere-bot · 2023-01-31T13:31:39Z

🤖 New build scheduled with the buildbot fleet by @pradyunsg for commit 7457fd3 🤖

If you want to schedule another build, you need to add the :hammer: test-with-buildbots label again.

Lib/test/test_cppext.py

pfmoore · 2023-01-31T14:18:13Z

I added a review comment that should fix the buildbot errors.

Lib/ensurepip/__init__.py

Lib/test/test_cppext.py

hugovk · 2023-02-02T15:15:19Z

Should this also go in What's New, under Removed?

pfmoore · 2023-02-02T16:02:25Z

Should this also go in What's New, under Removed?

The inclusion of setuptools was always explicitly documented as an internal implementation detail, and as such, I don't think we have any obligation to report its removal. It could be mentioned in the same way that we might document the removal of an internal API - I did a quick check and the 3.11 What's New included "Removed the undocumented private float.__set_format__() method". So I guess there's a precedent.

encukou · 2023-02-03T09:44:57Z

Please add it as a convenience to users, even if there's technically no obligation. It's been very easy to end up depending on setuptools accidentally.

IMO, for removals/breaks, it's best to write What's New entries for the target audience of a poor soul who isn't too well-versed in Python but inherited some complex system with a “temporary” hack. Point them to a solution, don't apologize, but don't berate them either.
With that in mind, please mention pkg_resources, as that shows up in ImportErrors but the connection to setuptools may not be clear.

(cc @python/proofreaders -- hope the above is good general advice)

pradyunsg · 2023-04-17T23:26:04Z

All the buildbot failures seem like false negatives, with test_asyncio, test_eintr, test_toolsaand test_typing failing for unrelated reasons.

pradyunsg · 2023-04-18T04:40:56Z

Given that this has multiple reviews, that the CI and buildbots are happy (all the non-flaky ones), that all outstanding comments are addressed and the multiple approvals at various points, I'm going to go ahead and merge this. If there's more needed here, we can handle that in a follow up!

pradyunsg · 2023-04-18T04:44:02Z

Thanks all! Holler on the issue, if there's any concerns! :)

* main: (24 commits) pythongh-98040: Move the Single-Phase Init Tests Out of test_imp (pythongh-102561) pythongh-83861: Fix datetime.astimezone() method (pythonGH-101545) pythongh-102856: Clean some of the PEP 701 tokenizer implementation (python#103634) pythongh-102856: Skip test_mismatched_parens in WASI builds (python#103633) pythongh-102856: Initial implementation of PEP 701 (python#102855) pythongh-103583: Add ref. dependency between multibytecodec modules (python#103589) pythongh-83004: Harden msvcrt further (python#103420) pythonGH-88342: clarify that `asyncio.as_completed` accepts generators yielding tasks (python#103626) pythongh-102778: IDLE - make sys.last_exc available in Shell after traceback (python#103314) pythongh-103582: Remove last references to `argparse.REMAINDER` from docs (python#103586) pythongh-103583: Always pass multibyte codec structs as const (python#103588) pythongh-103617: Fix compiler warning in _iomodule.c (python#103618) pythongh-103596: [Enum] do not shadow mixed-in methods/attributes (pythonGH-103600) pythonGH-100530: Change the error message for non-class class patterns (pythonGH-103576) pythongh-95299: Remove lingering setuptools reference in installer scripts (pythonGH-103613) [Doc] Fix a typo in optparse.rst (python#103504) pythongh-101100: Fix broken reference `__format__` in `string.rst` (python#103531) pythongh-95299: Stop installing setuptools as a part of ensurepip and venv (python#101039) pythonGH-103484: Docs: add linkcheck allowed redirects entries for most cases (python#103569) pythongh-67230: update whatsnew note for csv changes (python#103598) ...

This is a refactoring change that aims to simplify ``ensurepip``. Before it, this module had legacy infrastructure that made an assumption that ``ensurepip`` would be provisioning more then just a single wheel. That assumption is no longer true since [[1]][[2]][[3]]. In this change, the improvement is done around removing unnecessary loops and supporting structures to change the assumptions to expect only the bundled or replacement ``pip`` wheel. [1]: python@ece20db [2]: python#101039 [2]: python#95299

This is a refactoring change that aims to simplify ``ensurepip``. Before it, this module had legacy infrastructure that made an assumption that ``ensurepip`` would be provisioning more then just a single wheel. That assumption is no longer true since [[1]][[2]][[3]]. In this change, the improvement is done around removing unnecessary loops and supporting structures to change the assumptions to expect only the bundled or replacement ``pip`` wheel. [1]: python@ece20db [2]: python#101039 [3]: python#95299

Currently `get-pip.py` installs not only pip, but also setuptools and wheel by default, unless the `--no-setuptools` / `--no-wheel` (or `PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used. This has historically been necessary, however, modern versions of pip will now fallback to `pyproject.toml` (PEP 517: [1]) based builds (which will default to a setuptools backend, and thus automatically install setuptools and wheel in the isolated build environment) if either setuptools is not installed (as of pip 22.1: [2]), or if wheel is not installed (as of pip 23.1: [3]). In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv` modules no longer install setuptools, and only install pip ([4]). As such, it is now time for `get-pip.py` to stop installing setuptools and wheel by default on Python 3.12+, in order to: - Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+. - Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip. Closes pypa#200. [1]: https://peps.python.org/pep-0517/ [2]: pypa/pip#10717 [3]: pypa/pip#11871 [4]: python/cpython#101039

Currently the buildpack performs a system site-packages install of not only pip, but also setuptools and wheel. This has historically been necessary for pip to be able to build source distributions (sdists) for packages that don't ship with compatible wheels. However: - Thanks to PEP 518, packages can now (and many already do) specify an explicit build backend using `[build-system]` in their `pyproject.toml`. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions. - As of pip v22.1, pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend), if the setuptools package isn't installed globally. This means that packages that haven't yet migrated to a PEP 518 `pyproject.toml` build backend config can still build even if setuptools isn't installed globally. There are a small number of rarely used packages in the wild that aren't compatible with build isolation mode, however, these typically require more build dependencies than just setuptools, which means they wouldn't have worked with this buildpack anyway. As such, it's no longer necessary for us to install setuptools and wheel globally. This matches the behaviour of the `venv` and `ensurepip` modules in Python 3.12+, where setuptools and wheel installation has also been removed. And it also matches the default behaviour of Poetry too, whose `install --sync` command removes any implicitly installed packages in the current environment (other than pip). See: https://peps.python.org/pep-0518/ https://pip.pypa.io/en/stable/reference/build-system/ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation pypa/pip#10717 python/cpython#101039 pypa/get-pip#218 astral-sh/uv#2252 GUS-W-16437776.

It is necessary for building the package when running the integration tests, but more recent Python versions no longer include it by default: python/cpython#101039.

pradyunsg added stdlib Python modules in the Lib dir 3.12 bugs and security fixes labels Jan 14, 2023

pradyunsg requested review from vsajip and ezio-melotti as code owners January 14, 2023 17:50

bedevere-bot added the awaiting review label Jan 14, 2023

bedevere-bot mentioned this pull request Jan 14, 2023

Remove bundled setuptools #95299

Closed

pradyunsg force-pushed the remove-setuptools-ensurepip branch from 17a121e to 32b3ccc Compare January 14, 2023 22:37

pradyunsg force-pushed the remove-setuptools-ensurepip branch from 32b3ccc to 6939d48 Compare January 14, 2023 22:42

hugovk reviewed Jan 14, 2023

View reviewed changes

.github/workflows/verify-ensurepip-wheels.yml Outdated Show resolved Hide resolved

vsajip approved these changes Jan 15, 2023

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Jan 15, 2023

pradyunsg mentioned this pull request Jan 16, 2023

Stop installing setuptools and wheel by default on Python 3.12+ environments pypa/virtualenv#2487

Closed

jherland mentioned this pull request Jan 24, 2023

How to treat non-stdlib packages that are available by default in [all] venvs ? tweag/FawltyDeps#80

Closed

pradyunsg added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 31, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 31, 2023

pfmoore approved these changes Jan 31, 2023

View reviewed changes

pfmoore reviewed Jan 31, 2023

View reviewed changes

Lib/test/test_cppext.py Outdated Show resolved Hide resolved

merwok reviewed Jan 31, 2023

View reviewed changes

Lib/ensurepip/__init__.py Show resolved Hide resolved

merwok reviewed Jan 31, 2023

View reviewed changes

Lib/test/test_cppext.py Outdated Show resolved Hide resolved

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 17, 2023

pradyunsg merged commit ece20db into python:main Apr 18, 2023

bedevere-bot removed the awaiting merge label Apr 18, 2023

pradyunsg deleted the remove-setuptools-ensurepip branch April 18, 2023 04:43

hugovk mentioned this pull request Apr 27, 2023

Update bundled setuptools provided by ensurepip in current 3.8.x through 3.11.x to include fix for CVE-2022-40897? #102202

Closed

henryiii mentioned this pull request May 4, 2023

Project setup.cfg causes errors in cibuildwheel installing other packages pypa/cibuildwheel#1487

Closed

webknjaz mentioned this pull request Sep 10, 2023

GH-80789: Get rid of the ensurepip infra for many wheels #109245

Merged

edmorley mentioned this pull request Nov 23, 2023

Stop installing setuptools and wheel by default pypa/get-pip#200

Closed

albertvaka mentioned this pull request Jan 30, 2024

Missing dependency on setuptools sinricpro/python-sdk#66

Closed

pwang347 mentioned this pull request May 29, 2024

Missing packages message when trying to open data viewer microsoft/vscode-data-wrangler#191

Closed

edmorley mentioned this pull request Jul 7, 2024

Stop installing setuptools and wheel on Python 3.12+ pypa/get-pip#218

Merged

edmorley mentioned this pull request Aug 7, 2024

Stop installing setuptools and wheel heroku/buildpacks-python#243

Merged

edmorley mentioned this pull request Aug 11, 2024

remove no-setuptools config option python-poetry/poetry#9331

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-95299: Stop installing setuptools as a part of ensurepip and venv #101039

gh-95299: Stop installing setuptools as a part of ensurepip and venv #101039

pradyunsg commented Jan 14, 2023 •

edited

Loading

pradyunsg commented Jan 14, 2023 •

edited

Loading

pradyunsg commented Jan 14, 2023 •

edited

Loading

vsajip left a comment

encukou commented Jan 16, 2023

pradyunsg commented Jan 16, 2023

pradyunsg commented Jan 16, 2023

bedevere-bot commented Jan 31, 2023

pfmoore commented Jan 31, 2023

hugovk commented Feb 2, 2023

pfmoore commented Feb 2, 2023

encukou commented Feb 3, 2023

pradyunsg commented Apr 17, 2023

pradyunsg commented Apr 18, 2023

pradyunsg commented Apr 18, 2023

gh-95299: Stop installing setuptools as a part of ensurepip and venv #101039

gh-95299: Stop installing setuptools as a part of ensurepip and venv #101039

Conversation

pradyunsg commented Jan 14, 2023 • edited Loading

pradyunsg commented Jan 14, 2023 • edited Loading

pradyunsg commented Jan 14, 2023 • edited Loading

vsajip left a comment

Choose a reason for hiding this comment

encukou commented Jan 16, 2023

pradyunsg commented Jan 16, 2023

pradyunsg commented Jan 16, 2023

bedevere-bot commented Jan 31, 2023

pfmoore commented Jan 31, 2023

hugovk commented Feb 2, 2023

pfmoore commented Feb 2, 2023

encukou commented Feb 3, 2023

pradyunsg commented Apr 17, 2023

pradyunsg commented Apr 18, 2023

pradyunsg commented Apr 18, 2023

pradyunsg commented Jan 14, 2023 •

edited

Loading

pradyunsg commented Jan 14, 2023 •

edited

Loading

pradyunsg commented Jan 14, 2023 •

edited

Loading