Rationalise and improve the --target, --prefix and --root options

[pfmoore]

What's the problem this feature will solve?

At present, pip allows the user to select alternative install locations using --target, --prefix and --root. However, these are install-only options, and so do not work for other pip commands, like list or uninstall.

Describe the solution you'd like

Make the "target install scheme" options global, so that they apply to all commands.

In particular, be able to manage directories populated using pip install --target the same as a site-packages directory (with support for upgrades, uninstalls, list and list --outdated, etc).

Alternative Solutions

Stick with what we have. Continue requiring users to delete and recreate directories populated via --target whenever they want to change things.

Additional context

In terms of implementation, experiments suggest that this can be achieved just by modifying get_scheme in pip._internal.locations, and get_(default)_environment in pip._internal.metadata. In practice, the bulk of the work will be removing the existing code in favour of the new approach.

The proposed new approach will not work with the "legacy" setup.py install mechanism. We will need to decide whether to continue to support the existing options in that case. If we do, care will be needed to leave "just enough" of the existing code to support that route. The alternative would be to desupport using these options with the legacy install route, either by failing with an error suggesting use of --use-pep517, or by simply switching that option on automatically. Or we could delay implementing this feature until the legacy install route is finally removed 🙂

A feasible alternative/transition method would be to add one or more new global options that replicated the behaviour of the existing options using the new approach. The old options could then be removed at a later date. The big issues here are (1) thinking of good names for the "new" option(s), and (2) replicating all of the existing tests to check the new approach as well as the old one.

One further point to consider is that --target currently appears to install into a temporary directory and move the files into the target location once the install is complete. It's not clear why this is done, and I could find no documentation that either explained or guaranteed this behaviour. Would it be acceptable to remove that as part of the reorganisation? (My best guess is that it's to avoid a failed install damaging the target location - if we make --target do a "first class" install, with upgrade and uninstall support, our existing protections against breaking environments should be a superior replacement for the "temporary location and copy" approach).

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Or we could delay implementing this feature until the legacy install route is finally removed 🙂

That’s what I’d prefer.

[pfmoore]

Do we have any feel for how long it's likely to be before we get to that point? I know @sbidoul has been working on this, but I don't have a good feel for timescales.

[sbidoul]

I've restarted work on deprecating all setup.py install code paths. And I'm making good progress. #11331 is ready to merge. And I think I finally found the elusive good path to disentangle --no-binary (the first step is ready to review in #11359).

So if all goes well, by the end of this year, the deprecation of setup.py install will be explicit for all use cases, and there will be alternate mechanisms for users (possibly depending on pypa/setuptools#2491 and #11325).

After that, it will be a matter of getting a feel of user impact to decide when to switch to --use-deprecated and finally pull the plug. That is harder to predict.

Regarding this proposal: +100. On the how, I'd be fine if pip aborts when hitting setup.py install with the new scheme.

[septatrix]

So which one of these should one currently use to install packages into another root directory, e.g. a chroot? --root "$BUILDROOT" --prefix /usr? Should --target also be used? And what exactly is the difference between python3.11 -m pip and python3.12 -m pip --python python3.11?