Deprecate --egg

[dstufft]

I would like to deprecate and ultimately remove the --egg option. This will be a backwards incompatible change. This option was originally added in order to work around an issue with namespace packages (#3).

However using this option subtly breaks a number of things, namely:

• Installation from Wheels.
  • This will be a much bigger deal if we move to always installing from Wheel in the future.
• Loss of Control over dependency resolution/satisfaction.
  • If this option is selected than all of it's dependencies will be installed using setuptools instead of pip.
  • Thus, --index-url, --no-index, --find-links, --cert, and --no-deps flags will not have any affect.
  • PEP440 versioning and --pre support will not have any affect.
  • External/Internal and Verifiable/Unverifiable links protection will no longer have affect, thus reverting things back to the old rules.

Most of this are fundamental problems with attempting to use --egg since the only way to actually get an egg is to pass control back to setuptools and re-enable it's own built in support for these things. A few of them might possibly be able to be worked around but ultimately I think this option is a footgun we provide to our users.

So given all of the above, I want to deprecate this in 1.6 for removal in 1.8.

[dstufft]

Thoughts @pypa/pip-developers ?

[dstufft]

For reference, the PR that added --egg was #541.

[pfmoore]

I'm +1 in principle (I don't think we should be supporting the egg installation format any more) and the issues you mention with falling back to setuptools' build support are compelling. I don't really understand the issue #3 that triggered the inclusion of --egg but now that core Python has better support for namespace packages via PEP 420, it may be that this is less relevant now.

[jezdez]

+1

[qwcode]

someone should first determine if there's another workaround to #3, or what the current status of that is given PEP420.

[dstufft]

AFAIK there is no other work around, we have to pick which set of broken we want.

1. Leave the option in as a footgun to people who attempt to use it
   • Some people who do seem to think it means pip will install from .egg instead of install as egg
   • All of the things I listed originally
2. Take the option out and remove a work around to the namespace issue.
   • Even with the work around I've seen very few successful uses of namespace packages, sooner or later they always seem to bite someone in the ass.
   • I don't think a good work around is one that breaks half our features in subtle and confusing ways.

[carljm]

I think Donald's summary is correct. For anyone using Python 2, PEP 420 is useless. So that is not a useful workaround for #3 for most people.

If I had realized all the forms of brokenness that the implementation of --egg introduced (that Donald listed above), I would never have merged it in the first place. Sorry :-)

I actually think many of those forms of brokenness of --egg might be fixable by switching the implementation to use setup.py install rather than setup.py easy_install (I think the PR submitter switched to the latter so as to be able to use --always-unzip, so that might involve resigning to install zipped eggs).

I'm not going to weigh in in terms of whether it's better to fix --egg or remove it.

[qwcode]

the option help does have a warning: "WARNING: Because this option overrides pip's normal install logic, requirements files may not behave as expected." I added that 6 months ago when this came up.

let's post a notice in #3 that this might happen, and see if there's a reaction. maybe someone would get motivated to make it less broken.

[dstufft]

Generally I think if an option basically implies "use of this option will break half of the functionality of this program" then that option is not a good fit for that program.

[k4ml]

I actually think many of those forms of brokenness of --egg might be fixable by switching the implementation to use setup.py install rather than setup.py easy_install (I think the PR submitter switched to the latter so as to be able to use --always-unzip, so that might involve resigning to install zipped eggs).

It's the other way around actually. It currently using setup.py install just like the rest of the use case except that when --egg option is used, --single-version-externally-managed is omitted. The dependencies problem can be fixed by using setup.py easy_install since that one has --no-deps option but missing other options such as --root or --compile. Setuptools in this regard I think completely broken since install supposed to be a wrapper to easy_install but we left with a wrapper that missing an important piece of the original command that it wrap.

Unless setuptools get fixed I think there's no other way to fix --egg.

[jaraco]

install supposed to be a wrapper to easy_install

I'm not sure that's true. My understanding is install is a wrapper for the distutils command of the same name, adding support for resolving dependencies (among other things). The easy_install command is meant to be more like an installer (similar to what pip does). It doesn't derive from the distutils install command. It doesn't install the current project, but instead installs the packages indicated on the command-line.

I think part of the problem here is there's an underlying philosophical difference between what pip wants to do and what the users expect.

What pip does:

• Insist on coping all files of a package into a single directory, removing namespace support.

What users expect:

• Some files of a package should be located separately in some cases so they can be edited in place (editable) or so they can be included/excluded at run-time (multi-version).

Setuptools has gone to great pains to maintain that expectation. Prior to PEP420, setuptools was the only package that implements support of editable packages in a namespace packages environment. I personally contributed to the PEP420 implementation because I feel passionately about this feature.

Even with the work around I've seen very few successful uses of namespace packages, sooner or later they always seem to bite someone in the ass.

Namespace packages are a very powerful idea. They're a natural progression of package management. Just like DNS went from a flat namespace to a nested one and just like Java did to manage the proliferation of names, it's an imperative for Python to support this model. If pip can't support developing namespace packages, then it is pip that is flawed, not the developers' desire to use such packages. Is it really the case that namespace packages are too hard?

I agree --egg is a bad paradigm, but I would be strongly opposed to its removal. I've referred several developers who have encountered issues using pip -e. Our company uses namespace packages extensively. I personally work and develop in several namespaces:

• Zope and pyramid
• backports
• jaraco
• svg
• ore
• private namespaces

Namespaces allow packages to encapsulate names, maintain branding, share effort, but most importantly, scale.

If pip has painted itself into a corner by its installation technique (removing package files), then it should consider extending that installation technique to support this powerful model. I believe it would be a travesty for pip to declare defeat against this proven functionality. Pip should add this support before it removes the only viable workaround. Failing to do so will certainly lead to extra consternation within the community.

Unless setuptools get fixed...

If there's anything that setuptools can do to help in this regard, then please do file tickets and submit pull requests. Describe what the defects are and how enhanced behavior would help. Changes that enable pip to function better would be highly valued and given priority.

[dstufft]

What pip does:

• Insist on coping all files of a package into a single directory, removing namespace support.

What users expect:

• Some files of a package should be located separately in some cases so they can be edited in place (editable) or so they can be included/excluded at run-time (multi-version).

@jaraco Assuming that by package here you're refering the import-able thing and not the thing hosted on PyPI. Pip doesn't really insist on that, Python itself does. Setuptools works around that by relying on runtime support via pkg_resources. We don't support multi-version and I think doing so is likely to be a bad idea.

Even with the work around I've seen very few successful uses of namespace packages, sooner or later they always seem to bite someone in the ass.

Namespace packages are a very powerful idea. They're a natural progression of package management. Just like DNS went from a flat namespace to a nested one and just like Java did to manage the proliferation of names, it's an imperative for Python to support this model. If pip can't support developing namespace packages, then it is pip that is flawed, not the developers' desire to use such packages. Is it really the case that namespace packages are too hard?

I agree that namespace packages are a useful concept, what I meant by them being a bad idea is that the implementation that is available in pkgutil and setuptools is extremely fragile. pkgutil basically doesn't work at all in the concept of editables, and the setuptools version doesn't work unless you use .egg dirs or put them in the same directory. This means that in order to use editable you have to use --egg which means you're relying on setuptools to do the version resolution for all of the dependencies of your project.

This is what I mean by sooner or later they come to regret it.

I don't like supporting a feature that uses the non standard .egg and path munging, but I could live with that. I think it's actively confusing for users to have two different tools handling the dependency resolution and I think pip needs to do what it can to ensure that it "owns" that whenever someone is invoking pip. We've had a lot of confusion over the fact that setuptools handles setup_requires and --egg is in the same boat.

If pip has painted itself into a corner by its installation technique (removing package files), then it should consider extending that installation technique to support this powerful model. I believe it would be a travesty for pip to declare defeat against this proven functionality. Pip should add this support before it removes the only viable workaround. Failing to do so will certainly lead to extra consternation within the community.

Unless setuptools get fixed...

If there's anything that setuptools can do to help in this regard, then please do file tickets and submit pull requests. Describe what the defects are and how enhanced behavior would help. Changes that enable pip to function better would be highly valued and given priority.

I honestly think a better way forward is to figure out how to backport PEP420 to earlier Pythons. I think that is going to be a much more robust mechanism than relying on pkg_resources's runtime path munging. In the meantime however if setuptools gave us more granular control I would feel a lot less bad about the --egg option. Currently we only have one real tool, --single-version-externally-managed to make setuptools not do the dependenecy resolution. If we could do setup.py install --always-unzp --no-deps that would be useful I think.

[jaraco]

If we could do setup.py install --always-unzp --no-deps that would be useful I think.

That's a good consideration. I'm not immediately aware what that implies, but I'm happy to explore the possibility.

I honestly think a better way forward is to figure out how to backport PEP420 to earlier Pythons.

The corollary to that is that until PEP420 is available in all supported Python versions, setuptools-installed (or similarly installed) eggs in the only sane way to deal with namespace packages in all but the most simple of installation scenarios (which are, admittedly, the norm).

I personally would enthusiastically support the backporting of PEP420 to Python 3.2 and 2.7, but given the subtle, perhaps surprising behavior that it could impose, we might be hard-pressed to convince the stakeholders.

[dstufft]

I'd like to push for this again.

I've just been made aware that Eric Snow backported the entire Python 3.x importlib machinery and it is available on PyPI (https://warehouse.python.org/project/importlib2/). This includes the ability to do implicit namespace packages i'm told.

This should hopefully allow for implicit namespace package support and make it possible to have namepsace packages that work across all methods of install.