Existing numpy installations are not getting cleaned up correctly

[njsmith]

The latest version of numpy breaks if you install it over a previous version of numpy, without properly uninstalling the previous version. So we're getting to re-discover all the cases where packages get installed without the metadata that pip needs to remove them again properly: numpy/numpy#12736

One case in particular seems particular surprising:

1. Take a package that install_requires=["numpy==old_version"]
2. Run pip install -e <this package>
3. Now run pip install numpy==latest_version

Now you have a corrupted numpy install, that's a mixture of the old and new versions.

I assume what's happening is that pip install -e <this package> is calling setup.py develop, which is then using easy_install to satisfy the numpy==old_version requirement, and easy_install doesn't record the metadata that pip needs to uninstall the old version.

For regular installs, pip has historically avoided this by first running setup.py egg_info, then installing all the install_requires packages, and then invoking setup.py install. I think it could do the exact same thing for editable installs: first run setup.py egg_info, install the requirements, and then run setup.py develop. Maybe it should?

[pradyunsg]

So we're getting to re-discover all the cases where packages get installed without the metadata that pip needs to remove them again properly

Hehe. Bug Reports are welcome. :)

---

This seems reasonable to me. I can't think of "sane" setups, that are not-broken currently and would break if we do this.

PS: I rephrased the title, to be shorter. Hopefully, I got it right. :)

[cjerdonek]

I assume what's happening is that pip install -e is calling setup.py develop, which is then using easy_install to satisfy the numpy==old_version requirement,

pip's InstallRequirement.install_editable() method calls setup.py develop with the --no-deps option (see the following), so I'm wondering if the explanation above is still a possible explanation:

pip/src/pip/_internal/req/req_install.py

Lines 755 to 763 in 0637aad

	call_subprocess(
	[
	sys.executable,
	'-c',
	SETUPTOOLS_SHIM % self.setup_py
	] +
	list(global_options) +
	['develop', '--no-deps'] +
	list(install_options),

This leads me to think that easy_install isn't (or at least shouldn't be) doing anything with the numpy dependency. Or am I misunderstanding your explanation?

The pip docs also say this:

To install a package in “editable” mode (pip install -e), setup.py must implement the following command:
setup.py develop --no-deps
This should implement the complete process of installing the package in “editable” mode.

[pradyunsg]

This leads me to think that easy_install isn't (or at least shouldn't be) doing anything with the numpy dependency.

Yeah, pip handles dependencies itself.

$ tree
.
├── setup.py
└── trial
    └── __init__.py

1 directory, 2 files
$ cat setup.py
from setuptools import setup, find_packages

setup(
    name="trial",
    version="0.1.0",
    packages=find_packages(),
    install_requires=["six"]
)
$ pip install -e .
Obtaining file:///Users/pradyunsg/.virtualenvs/tmp-74d363ab9317367/trial
Collecting six (from trial==0.1.0)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, trial
  Running setup.py develop for trial
Successfully installed six-1.12.0 trial

Seems like the explanation is OP is not correct. It'll still be useful to investigate why there are leftover files.

[cjerdonek]

Thanks for confirming, @pradyunsg. The title should probably be changed too, then, to describe the problem that occurred rather than the conjectured explanation / proposed solution.

[cjerdonek]

@njsmith It would be good to have log output of one of the confirmed problem cases to get a better understanding of the issue. (You're probably more familiar with which cases in the linked issue best illustrate the issue you're posting.)

[pradyunsg]

The title should probably be changed to

I guess this should do. I'll avoid touching the title again though.

[njsmith]

It would be good to have log output of one of the confirmed problem cases to get a better understanding of the issue. (You're probably more familiar with which cases in the linked issue best illustrate the issue you're posting.)

Unfortunately I have no real idea how these users ended up with messed up installs. I filed this based on one specific comment that claimed to have tested a minimal reproducer:

numpy/numpy#12736 (comment)

But then it turns out that the very next comment claims to have failed to reproduce the problem, just like Pradyun:

numpy/numpy#12736 (comment)

So... I dunno! I suppose it could be dependent on setuptools or pip version, or just a mistake somewhere. If anyone wants to jump in that thread to try to get more details out of the reporters then it might be useful? But I'm not sure there's any real pip bug here at all.

[tzickel]

Here is a reproduction of the bug on an empty ubuntu 18.04 docker image:
numpy/numpy#12736 (comment)

[tzickel]

I'm updating my findings on that issue, strange:
numpy/numpy#12736 (comment)

[tzickel]

Ok, found the bug it's in debian modification of the original pip source code:
numpy/numpy#12736 (comment)

[pfmoore]

Duplicate of #4222

[pradyunsg]

Thanks for some really nice investigation @tzickel!

I'll go ahead and close this issue then. :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.