Cannot downgrade setuptools with pip when using the flags "-IU"

[ale-rt]

If I have the last version of setuptools in my virtualenv, then I am not able anymore to downgrade it when I specify the -IU flags:

$ python3 -m venv /tmp/testenv
$ /tmp/testenv/bin/pip install -IU setuptools==36.2.3
Collecting setuptools==36.2.3
  Using cached setuptools-36.2.3-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-36.2.3
$ /tmp/testenv/bin/pip install -IU setuptools==36.2.2
Collecting setuptools==36.2.2
  Using cached setuptools-36.2.2-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-36.2.3

With other packages, it works as I would expect, i.e. the setuptools version is the one I pin through the command line or with the requirements file.

A workaround is to run the command twice, e.g.:

./bin/pip install -Ur requirements.txt
./bin/pip install -IUr requirements.txt

[benoit-pierre]

I think the problem is your use of the -I (ignore installed) flag, as it prevents pip from uninstalling the existing version of setuptools before installing another one, and you end up with multiple dist-info entries in your site-packages directory:

> python -m venv env
> ./env/bin/pip freeze --all
pip==9.0.1
setuptools==28.8.0
> ./env/bin/pip install -U setuptools
Collecting setuptools
  Using cached setuptools-36.2.3-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-36.2.3
> ./env/bin/pip install -I 'setuptools==36.2.2'
Collecting setuptools==36.2.2
  Using cached setuptools-36.2.2-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-36.2.3
> ./env/bin/pip freeze --all
pip==9.0.1
setuptools==36.2.3
> print -l env/lib/python3.6/site-packages/setuptools*
env/lib/python3.6/site-packages/setuptools
env/lib/python3.6/site-packages/setuptools-36.2.2.dist-info
env/lib/python3.6/site-packages/setuptools-36.2.3.dist-info

[benoit-pierre]

And the behavior is the same with other packages.

[ale-rt]

Thanks @benoit-pierre for the attention and the explanation.
So the -I flag will work for upgrades (leaving stale entries in the site-packages folder, but will not for downgrades because the most recent version will be picked.

[benoit-pierre]

Not quite, even for upgrades I don't think it's a good idea to use it:

> unzip -l dist/foo-0.1-py3-none-any.whl
Archive:  dist/foo-0.1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-07-26 13:11   foo/__init__.py
        0  2017-07-26 13:11   foo/mod1/__init__.py
       39  2017-07-26 13:13   foo-0.1.dist-info/DESCRIPTION.rst
      227  2017-07-26 13:13   foo-0.1.dist-info/metadata.json
        4  2017-07-26 13:13   foo-0.1.dist-info/top_level.txt
       92  2017-07-26 13:13   foo-0.1.dist-info/WHEEL
      172  2017-07-26 13:13   foo-0.1.dist-info/METADATA
      598  2017-07-26 13:13   foo-0.1.dist-info/RECORD
---------                     -------
     1132                     8 files
> unzip -l dist/foo-0.2-py3-none-any.whl
Archive:  dist/foo-0.2-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-07-26 13:11   foo/__init__.py
        0  2017-07-26 13:12   foo/mod2/__init__.py
       39  2017-07-26 13:12   foo-0.2.dist-info/DESCRIPTION.rst
      227  2017-07-26 13:12   foo-0.2.dist-info/metadata.json
        4  2017-07-26 13:12   foo-0.2.dist-info/top_level.txt
       92  2017-07-26 13:12   foo-0.2.dist-info/WHEEL
      172  2017-07-26 13:12   foo-0.2.dist-info/METADATA
      598  2017-07-26 13:12   foo-0.2.dist-info/RECORD
---------                     -------
     1132                     8 files
> ./env/bin/pip install dist/foo-0.1-py3-none-any.whl
Processing ./dist/foo-0.1-py3-none-any.whl
Installing collected packages: foo
Successfully installed foo-0.1
> ./env/bin/pip install -I dist/foo-0.2-py3-none-any.whl
Processing ./dist/foo-0.2-py3-none-any.whl
Installing collected packages: foo
Successfully installed foo-0.2
> find env/lib/python3.6/site-packages/foo
env/lib/python3.6/site-packages/foo
env/lib/python3.6/site-packages/foo/mod2
env/lib/python3.6/site-packages/foo/mod2/__pycache__
env/lib/python3.6/site-packages/foo/mod2/__pycache__/__init__.cpython-36.pyc
env/lib/python3.6/site-packages/foo/mod2/__init__.py
env/lib/python3.6/site-packages/foo/mod1
env/lib/python3.6/site-packages/foo/mod1/__pycache__
env/lib/python3.6/site-packages/foo/mod1/__pycache__/__init__.cpython-36.pyc
env/lib/python3.6/site-packages/foo/mod1/__init__.py
env/lib/python3.6/site-packages/foo/__pycache__
env/lib/python3.6/site-packages/foo/__pycache__/__init__.cpython-36.pyc
env/lib/python3.6/site-packages/foo/__init__.py

[ale-rt]

I see your point and thanks again for the explanation.

The issue I was trying to solve with the -I flag is that in one of the builds I am working with, for some reason if the build breaks, the bin/buildout script is removed.

Running pip install -Ur requirements.txt proved to not be enough to recreate the script in the bin folder, because the package is already in the site packages folder.
To recreate the script I added the -I flag.

Now I understand that, if I want the build to be as much convergent as possible, I have to run first pip install -Ur requirements.txt && pip install -Ir requirements.txt.
In this way I will end up with the right stuff in the site-packages folder and I will be sure I have the scripts freshly rebuilt.

[benoit-pierre]

What about with --force-reinstall?

[ale-rt]

Thanks I missed that option!

[ale-rt]

Unluckily --force-reinstallis not working the way I expected:

$ rm /tmp/testenv/bin/pep8 
rm: remove regular file '/tmp/testenv/bin/pep8'? y
$ /tmp/testenv/bin/pip install --force-reinstall pep8
Requirement already satisfied: pep8 in /tmp/testenv/lib/python3.5/site-packages
$ ls /tmp/testenv/bin/pep8
ls: cannot access '/tmp/testenv/bin/pep8': No such file or directory

[benoit-pierre]

You need to combine it with upgrade:

> pip install -h | grep force-reinstall
  --force-reinstall           When upgrading, reinstall all packages even if they are already up-to-date.

[ale-rt]

That works great! Thanks :)