Use distlib to emulate pkg_resources

[vsajip]

I've merged the latest changes. I got some test failures, but all of these appear to be because the test virtualenv created in tests/tests_cache/test_ws/setuptools/.virtualenv had both pip-1.3.1 and pip-1.4.dev1 in its site-packages. I can't see any reason why the uninstallation of pip during the test environment creation fails, and there's no check for failure in the env creation process.

I fixed the failures by adding code to ensure that any old pip directory is removed after the uninstallation.

Note: There are some occurrences of pkg_resources in string literals, which I haven't changed. These might need looking at.

[pfmoore]

I'd be much happier with this if vendor/distlib/pkg_resources.txt was somewhere else - it's not part of upstream distlib, so shouldn't be under distlib here. Actually it's not a "vendor" file at all as it isn't an unchanged copy of an upstream file.

Also, the vendor/distlib changes are an issue, as they will presumably be broken if anyone re-vendors distlib. They seem to be mostly around the metadata 1.3 vs metadata 2.0 issue, so maybe it's a temporary problem.But of course, this is why wheel support was waiting on acceptance of the standards in the first place, and why the fact that metadata 2.0 is still in flux is hurting us :-(

[vsajip]

Fair point about the location of the pkg_resources shim - I'll move it somewhere else. Re. the 1.3/2.0 issue, I believe it is only temporary (I only changed the 2.0 references to 1.3, no other changes were needed).

[vsajip]

I can't think of a better place to move the shim to than directly in the pip package, i.e. it would become pip.pkg_resources. Does that seem acceptable? If not, can any one suggest anything better?

[pfmoore]

I couldn't think of an ideal location either. pip/pkg_resources sounds fine. Or maybe pip/compat/pkg_resources. But the extra level doesn't really add much other than complexity...

[vsajip]

I think I will move the shim to pip.compat - the extra level isn't needed now, but it will avoid another move if I ever have to add another module relating to shim functionality.

[qwcode]

I want to make sure I understand why you think this pkg_resources adapter help's the migration.

There's 4 things here to distinguish:

1. not changing very much core pip code, for fear we might want to easily revert out of this later
2. changing less code, because it decreases chances for bugs
3. pip having classes/functions on top of distlib so as not to repeat routines over and over.
4. pip continuing to use the pkg_resources interface, because that's perceived as a good thing for some reason?

what is your rationale for doing it this way in terms of 1-4?

unless we're really motivated by 1, not sure why we'd do this.
we're creating an adapter that we'll want to replace later with a more direct distlib layer.
we don't have that much code that consumes pkg_resources interfaces to replace.
in any case, the risk is mostly about switching to distlib, not so much having more code changes.

[qwcode]

I want to make sure I understand why you think the pkg_resources adapter help's the migration.

There's 4 things here to distinguish:

1. not changing very much core pip code, for fear we might want to easily revert out of this later
2. changing less code, because it decreases chances for bugs
3. pip having classes/functions on top of distlib so as not to repeat routines over and over.
4. pip continuing to use the pkg_resources interface, because that's perceived as a good thing for some reason?

what is your rationale for doing it this way in terms of 1-4?

unless we're really motivated by 1, not sure why we'd do this.
we're creating an adapter that we'll want to replace later with a more direct distlib layer.
we don't have that much code that consumes pkg_resources interfaces to replace.
in any case, the risk is mostly about switching to distlib, not so much having more code changes.

[vsajip]

I want to make sure I understand why you think the pkg_resources adapter help's the migration.

I don't know yet to what extent it can help, but it certainly demonstrates that for a tool like pip, distlib can provide equivalent functionality to pkg_resources.

1. not changing very much core pip code, for fear we might want to easily revert out of this later

The advantage of minimising code changes means that the changes are easier to reason about and review. It's also important to be able to easily revert these changes if we do find problems.

2. changing less code, because it decreases chances for bugs

Not sure how you're distinguishing between point 1) and point 2) here. I could make the changes even more minimal by e.g. having all the other modules import pkg_resources from backwardcompat and in there, importing pkg_resources from either pip.compat or the real pkg_resources.

3. pip having classes/functions on top of distlib so as not to repeat routines over and over.

That's part of a wider design consideration about pip. As I see it, it's useful to be able to demonstrate that distlib can supplant pkg_resources with the relatively small amount of work that I've put in.

4. pip continuing to use the pkg_resources interface, because that's perceived as a good thing for some reason?

I don't see the pkg_resources interface as especially great, but doing it the way I've done it makes it easier to review because the code changes are small. I was also able to migrate individual modules over from the real pkg_resources to the shim, which has made the overall process of getting to here somewhat smoother than it otherwise might have been.

unless we're really motivated by 1, not sure why we'd do this.

A larger change would take longer to review, and entail more risks (because any change could introduce a bug). It could also involve a fair amount of changes to the test code, which I've managed to avoid by using the strategy I did.

we're creating an adapter that we'll want to replace later with a more direct distlib layer.

True, but the adapter should show that distlib can provide the raw functionality needed, which should give confidence about making more wide-ranging changes.

we don't have that much code that consumes pkg_resources interfaces to replace.

Well, there has been some talk (I think by Daniel) of vendoring pkg_resources, so the dependency is not entirely trivial.

in any case, the risk is mostly about switching to distlib, not so much having more code changes.

I'm not opposed to wider code changes calling more directly into distlib, but I think that by doing it that way, it would have taken somewhat longer to show meaningful progress. I've been more focused on ensuring that distlib has the relevant functionality to support the needs of pip - e.g. in the course of this work I made some slight improvements to distlib,database to make it easier to subclass distributions, which could benefit other clients of distlib.

[qwcode]

useful to be able to demonstrate that distlib can supplant pkg_resources

a "direct" non-adapter migration would demonstrate that too.
my pt of view, is to try to sort out what approach is best for pip long term.

It's also important to be able to easily revert these changes if we do find problems.

so that is part of your rationale, i.e. that we might end up doing some post-release revert of all this.

Not sure how you're distinguishing between point 1) and point 2) here.

I was distinguishing:

• making the code more easily revertable
• v.s. making less changes for the sake of reducing the chance for bugs

A larger change would take longer to review, and entail more risks (because any change could introduce a bug).

whether a direct approach would be that much larger and harder, is what I'm not sure about.
The advantage, is not having to redo the pkg_resources adapter later. I fear it would hang around for way too long. To speak with more confidence though, I would need to actually try what I'm talking about : )

[qwcode]

I'll look at this. regardless of this pull, I'm curious what might be going on here.

[vsajip]

Indeed - there wasn't even a non-zero returncode to indicate something might be wrong.

[vsajip]

a "direct" non-adapter migration would demonstrate that too.
my pt of view, is to try to sort out what approach is best for pip long term.

A non-adapter migration would entail more work and more risk, but I agree that longer term a different approach would be better. However, this approach allows that to be done module by module, rather than a "big bang" which from my experience would likely take longer to achieve, because "everything would stop working" while bugs were ironed out.

so that is part of your rationale, i.e. that we might end up doing some post-release revert of all this.

The likelihood of having to revert is dependent on the confidence level in the coverage of the pip tests. I don't see a show-stopper problem which would require a revert (any bugs that show up can be fixed just as I've been doing during this development) but I want to make it easy to revert, should it be necessary.

making the code more easily revertable
v.s. making less changes for the sake of reducing the chance for bugs

Making fewer changes makes it easier to make decisions about reverting, because it's easier to see what's going on.

whether a direct approach would be that much larger and harder, is what I'm not sure about.
The advantage, is not having to redo the pkg_resources adapter later. I fear it would hang around for way too long.
To speak with more confidence though, I would need to actually try what I'm talking about : )

Well, the adapter is done now, so there's no work there apart from reviewing it (easier than reviewing a direct-to-distlib approach) and deciding what to do about it. When it comes to a more direct approach, it can be approached a module at a time, which makes the development flow more smoothly IMO - which is one of the main reasons why I went the adapter route (the other main reason being ease of review).

[qwcode]

non-adapter migration would entail [...] more risk,
this approach [...] rather than a "big bang"

my point is that the "big bang" is moving over to distlib, which is where most of the risk is at.
yes, a direct migration would require more plumbing work to rewire to different interfaces, but stuff like that generally fails immediately if it's wrong and is not where most of the risk is at I think.

Well, the adapter is done now,

this argument has the most power practically : )

[qwcode]

just included this pull as an option to fast track getting pip PEP439 compliant (see #863)

[dstufft]

I'm closing this as we've vendored pkg_resources now, and any move to replace pkg_resources with distlib will likely be done without a shim layer.