vendor setuptools/pkg_resources?

[dstufft]

sorry, donald, I hijacked your issue, loading all this into one issue, since the decision is interconnected --qwcode

3 problems to be solved:

1. making pip a self-sufficient wheel installer
2. removing the headache of users being responsible for managing (and possibly breaking/uninstalling) the setuptools build dependency themselves (most notably a pain in non-virtualenv environments),
3. the burden/fragility of supporting any pip version with any setuptools version.

solutions for #1:

1. vendor pkg_resources
   • pro: it's safer than using distlib
2. emulate pkg_resources using the already vendored distlib (Use distlib to emulate pkg_resources #909 )
   • con: why emulate?, it's weird, just use directly
3. use distlib directly to replace our uses of pkg_resources

solutions for #2:

( "setuptools-core"/"pkg_resources"/"setuptools" is a fictional restructure)

1. vendor "setuptools-core" (by including a .zip/.egg that you add to sys.path for install subprocess) and have pip install "pkg_resources" for run-time entry point use
   • pro: no way a user can break pip's setuptools support
   • con: time drain of getting setuptools distributed differently
   • con: its anti-MEBS
   • "con": a user can't upgrade setuptools themselves for use within pip
2. have get-pip.py install setuptools from wheel (assuming a solution to problem No instructions for how to install pip #1 is implemented)
   • pro: it's less work than other solutions (although the dynamic install solution is pretty easy too)
3. dynamic install of setuptools from wheel when pip needs to install sdists or when installing something from wheel that uses pkg_resources entry points and declares no setuptools dependency (assuming a solution to problem No instructions for how to install pip #1 is implemented)
   • pro: it's closer to the long-term MEBs model
   • pro: pretty easy to do
   • con: can fail offline trying to get setuptools, when a user is expecting it should work
4. just be happy that the pip bootstrap/bundle efforts will alleviate the pain in new versions of python (by pre-installing setuptools?)
   • con: it may not happen, or take longer than we want

solutions for #3:

• vendor a specific version (No instructions for how to install pip #1 above)
• for 2 above, use >= (or ==?) for setuptools in pip's install_requires
  • con: can result in a DistributionNotFound exception that users will be confused by
• for 3 above, have pip always check for a specific requirement of setuptools

[dstufft]

This is a related ticket that can probably go away once this is done #650.

[qwcode]

related discussions:
https://groups.google.com/forum/#!topic/pypa-dev/YjtGVk2UnXA
https://groups.google.com/forum/#!topic/pypa-dev/SIdju4xXbRc

[dstufft]

Pip is in a class of tools where a broken dependency can completely cripple it with no good way of fixing it.

Ideally in my eyes pip should not depend on anything existing or being a particular version except for what it includes in it's own code, or what is available in the standard library. As we've recently seen the dependency on setuptools can cause a tremendous headache for users when something goes south. And when it does go south pip is rendered useless and users need to fix it manually. We've also seen that new setuptools versions can cause regressions on older versions of pip.

I do not believe we want to try to pin versions because that still has a ton of areas for failure actually increases the failure points becasue the setuptools console scripts will throw an exception when it can't find the proper version of setuptools.

Dynamically installing setuptools could work as long as pip had no internal dependency on setuptools. I do still believe that if we do that we should try and push for pkg_resources to be split out so that installing from Wheel doesn't require getting setuptools but only pkg_resources.

So I guess in my eyes either way pips internal use of pkg_resources needs to be vendored. If we are already going to vendor setuptools I think it makes sense to use that instead of an emulation layer with distlib. If we're going to use distlib I think I'd rather us just use it than use some sort of a shim.

The biggest thing that worries me about dynamically installing setuptools is that it monkeypatches things and we might end up with bugs depending on if it is or isn't installed.

tl;dr
- Vendor pkg_resources or use distlib wholesale, no shim layer
- Dynamically install setuptools or vendor it for sdists
- Hopefully Install just pkg_resources for entry points if needed, otherwise all of setuptools

[pfmoore]

+1 to all 3 points @dstufft makes.

• I prefer using the already vendored distlib over vendoring pkg_resources
• I'm not a fan of either dynamically installing or vendoring for sdists. I don't have a good answer here. Although it's only an issue for non-virtualenv builds (as we have setuptools preinstalled in virtualenv).
• I'm strongly in favour of seeing pkg_resources split out as a separate distribution from setuptools, so that we can depend only on pkg_resources for entry points. That's very much the setuptools team's call, though.

Longer term, I think we should look very closely at the mechanism by which distil does its installs. As far as I understand it, distil manages to build a significant proportion of PyPI packages without needing setuptools at all (even packages which explicitly use setuptools in the setup.py). Given that it's possible, I strongly believe that our goal should be something similar - provide a viable, transparent migration path from "executable stup.py with setuptools" to a zero-dependency install from sdist process.

[dstufft]

I don't think it's only an issue for non virtualenvs. Just because setuptools is installed in a virtualenv doesn't mean soemthing won't cause it to be uninstalled (such as the bug we're having now). Now dynamically installing in that case doesn't help but vendoring does. I do think long term virtualenvs shouldn't bundle setuptools as it primary does it to make pip work afaict.

I don't think we should try to be clever like distil is trying. Clever is the gateway to horribly broken in tons of edge cases. The path for no longer using setup.py and such is via Wheel and sdist 2.0.

[dstufft]

Also I don't have the distil source code available but I can easily point out several distributions where it's completely impossible to determine the actual set of installation dependencies without processing setup.py in some way likely executing it. I would be very surprised if distil has actually managed to handle that case. My gut instinct is it's handled the 80% case while completely breaking or giving a degraded install experience for the other 20%.

[pfmoore]

Good point regarding virtualenv. I'm fighting as hard as I can against vendoring setuptools (because once we do, I am concerned that we'll never be able to get rid of it), but I'm finding it harder and harder to do so :-(

You're probably right about distil. I'd like to see some specifics about how sdist 2.0 will help, though. And in particular, I'd like to look at how we remove the need for setuptools even when building distributions that haven't been updated to sdist 2.0. Or to put it another way, I don't want to have to deal with setuptools any longer than I have to as part of a "legacy" support solution. (I didn't think that was possible, but distil - even if it's being "clever" - shows that there is a way of doing it). I did deliberately say "look at it" - it may not be a suitable approach, but I don't want to ignore the possibility that there are ideas in there that we could use.

As far as how distil does it, I believe that it uses a set of extra metadata that Vinay maintains based on a one-off exercise of scanning the setup scripts somehow - he'd need to explain how. Obviously, relying on 3rd party metadata isn't viable for pip, but I assume that what Vinay has could be used to populate PyPI's metadata 2.0 information for existing packages (we need some automated way of extracting that data for older packages, or the whole exercise of migrating to metadata 2.0 is going to take so long that we may as well resign ourselves to keeping setup.py till Python 4 comes out...)

[dstufft]

We're going to have to support a setup.py for a long time. There's no good way around that. There's so many different things that can be done in one that either we break things in some circumstances or we support it. Personally I lean towards breaking as little things as possible.

At the moment i'm going back and forth on vendoring setuptools. If we remove the dependency (runtime or otherwise) for setuptools in pip and it's only needed for old style sdists then I think dynamically installing it is ok (and paves the way forward for the eventual MEB system).

In my mind then the biggest reason to vendor it vs dynamically installing it (assuming we remove runtime dependecy in pip) is less moving pieces and a more robust build system, especially in the wake of this distribute/setuptools merger. The biggest reason for not vendoring is it's closer to our eventual goal of setuptools being just another possible builder and it allows pip to never have to argue for getting rid of it from inside of pip.

[dstufft]

Oh the other big reason not to vendor it is because vendoring it is going to require a pretty decent engineering effort in setuptools to move imports into a system that pip can reasonable vendor it.

[qwcode]

not [...] pin versions because that still has a ton of areas for failure
increases the failure points [...] console scripts will throw an exception
when it can't find the proper version of setuptools.

yes, getting DistributionNotFound for setuptools==PIN_VERSION would be annoying, but pkg_resources would be doing it's job. as for fixing things, it could just involve running get-pip.py --repair (see below)

get-pip.py [<options>] [<pip-version>]

  install/upgrade to latest pip (and the setuptools it requires)

  args:
     pip-version  specify a specific pip version

  options:
     --repair     use existing pip version if present, but fill in appropriate setuptools

I think I'd rather us just use it than use some sort of a shim.

I said the same in vinay's PR, but I'm inclined towards the safe route of vendoring the existing pkg_resources right now.

The biggest thing that worries me about dynamically installing setuptools is
that it monkeypatches things and we might end up with bugs depending on if it is
or isn't installed.

not sure what you mean, but "dynamic install" is not meant to imply a special kind of "dynamic" install. It's just a normal install that happens only if you need to build an sdist. It should be set to fail if an existing install is != to the version we want. (what "we want" would have to be set somewhere, but not in install_requires)

[dstufft]

I hate the get-pip.py thing and consider it a unpleasant necessity. I also think it's generally better to never (or rarely) break vs needing to "repair" it.

I'm fine with vendoring pkg_resources.

Not sure how to be clearer, just stating that setuptools monkeypatches things so there might be bugs that only exist based on if setuptools monkeypatching is there or not. I don't mean to say that's a problem per say, just a potential pain point

[qwcode]

I hate the get-pip.py thing and consider it a unpleasant necessity.

you hate it compared to what? pip has to get off the ground?
if python really does "include" pip, would it just be py3.4? or quickly back-ported?
I guess I'm skeptical it will happen on schedule, and making the get-pip.py changes is pretty easy.

I also think it's generally better to never (or rarely) break vs needing to "repair" it.

the dynamic install solution can end up "broken" too in a sense, if we're strict about what has to be installed.
a user could upgrade, and then pip will have no choice except to stop and ask what should be done? proceed at risk, or re-install to the desired version, but it's true it could all happen in-process, and with human terms that a user can understand.

[qwcode]

added all the pro/con bullets I can think of from what people have said.
again, I don't have a horse in this, just want to see it all laid out.

[qwcode]

just stating that setuptools monkeypatches things so there might be bugs
that only exist based on if setuptools monkeypatching is there or not

ok, pip does use distutils directly in a few limited cases, so it's possible they behave differently depending on setuptool's patch presence.

[dstufft]

I don't really consider that a major blocker fwiw. Just something to weigh when deciding between bundled or unbundled setuptools.

[dstufft]

s/bundled/vendored/

[dholth]

If you did want to vendor setuptools you would do it by including a .zip or .egg that you added to sys.path in the build subprocess. Not by trying to "import pip.setuptools".

[qwcode]

ok, so I'll remove this con from the description: "headache of rewriting or overriding setuptools imports for use in pip"

[pfmoore]

Someone should confirm this works. When I last tried to use setuptools from a zip or similar (I can't recall what precisely I did) it didn't work as expected because the .pth hacks present in setuptools weren't activated.

[dstufft]

I think this should probably get bumped from the 1.5 timeline, any objections?

[dholth]

I think "vendor pkg_resources" is very close. What's missing?

[pfmoore]

I'd like to see it go in, but I don't think it should be a blocker for 1.5.

I'm still not entirely clear whether, if it goes in, we would expect to stop bundling setuptools in ensurepip and/or virtualenv (it would no longer be needed for wheel installs, but sdist installs will still be the most common type for a long time yet).

[dstufft]

My thinking was it was a decent enough change that this itself should probably happen early on in a release cycle to make sure we don't introduce something strange because of some assumption in pkg_resources.

[pfmoore]

Fair point.

[qwcode]

closing this issue. think it's outlived it's usefulness for analysis.
the direct action for now is to vendor pkg_resources via #1422