pip install --user ignores system-installed dependencies, installs older versions

[akkana]

• Pip version: 9.0.1
• Python version: 2.7.13
• Operating System: Debian testing (stretch)

Description:

pip install --user, on a package that has depedencies, ignores the debian-installed dependencies and installs a different (perhaps quite a bit older) version from pypi.

In particular: In https://github.com/matplotlib/mpl_finance I run python setup.py sdist
That setup.py includes: install_requires=['matplotlib']
I have 2.0.0~rc2-1 installed as part of the python-matplotlib debian package.

pip show matplotlib sees it:
Name: matplotlib
Version: 2.0.0rc2
Summary: Python plotting package
Home-page: http://matplotlib.org
Author: John D. Hunter, Michael Droettboom
Author-email: [email protected]
License: BSD
Location: /usr/lib/python2.7/dist-packages
Requires:

But if I run: pip install --user dist/mpl_finance-0.10.0.tar.gz
pip will download matplotlib version 1.5.3 from pypi and install it in ~/.local/lib. (This causes everything to break, because that version of matplotlib has a bug that makes it not work on linux 32-bit.) It also downloads a bunch of matplotlib dependencies that are also already installed in the system.

This is wrong according to https://pip.pypa.io/en/stable/user_guide/#user-installs :
"When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied (similar to how global packages can satisfy requirements when installing packages in a --system-site-packages virtualenv)."

It's not because the system matplot lib is a release candidate: if I add --pre to the pip install line, it will download matplotlib 2.0.0rc2 (same as my system version) from pypi.

If I install (without --user) in a virtualenv --system-site-packages, things work right, it sees the system matplotlib and doesn't install the older one or any other packages. It's just with --user that it breaks.

[msjyoo]

I am also suffering from this issue. Very annoying since it'll install pip 10 at every possible opportunity and break the system. A workaround would be great.

[rwb27]

I've run into this one with matplotlib (and numpy et al) on a Raspberry Pi (latest Raspbian) as well. It's possible to work around it by first looking in the package's setup.py and manually installing any missing dependencies, then installing my package, but telling pip not to install the dependencies (pip install --no-deps mypackagename).

Obviously that's a pretty poor workaround, but it has at least got my code running this morning! Hope it helps someone.

[benoit-pierre]

Looking at the patches for the Debian version of pip-9.0.1, I found this gem:

From f392f69c8f813779b3ad3a384f57dc1f805890ef Mon Sep 17 00:00:00 2001
From: Barry Warsaw <[email protected]>
Date: Wed, 10 Feb 2016 11:18:37 -0500
Subject: Default to --user in non-virtual environments.

When running as a normal user in a non-virtual environment, default to
--user and --ignore-installed.  When inside virtual environments or when
running as root, keep the default behavior.

Author: Didier Roche <[email protected]>,
        Barry Warsaw <[email protected]>
Bug: https://github.com/pypa/pip/issues/1668
Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725848
Bug-Ubuntu: https://bugs.launchpad.net/ubuntu/+source/pip/+bug/1419695

Patch-Name: set_user_default.patch
---
 docs/user_guide.rst     |  6 ++++--
 pip/commands/install.py | 20 +++++++++++++++++++-
 2 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/docs/user_guide.rst b/docs/user_guide.rst
index 7d7fb86..83d6a7c 100644
--- a/docs/user_guide.rst
+++ b/docs/user_guide.rst
@@ -513,8 +513,10 @@ which means that all Python distributions support an alternative install
 location that is specific to a user.  The default location for each OS is
 explained in the python documentation for the `site.USER_BASE
 <http://docs.python.org/library/site.html#site.USER_BASE>`_ variable.  This mode
-of installation can be turned on by specifying the :ref:`--user
-<install_--user>` option to ``pip install``.
+of installation is the default on Debian and derivative systems (--user has no
+effect) when inside non-virtual environments, and when the script is run as
+non-root. --ignore-installed is then used.  This behavior can be turned off by
+specifying the :ref:`--system <install_--system>` option to ``pip install``.
 
 Moreover, the "user scheme" can be customized by setting the
 ``PYTHONUSERBASE`` environment variable, which updates the value of ``site.USER_BASE``.
diff --git a/pip/commands/install.py b/pip/commands/install.py
index 227c526..39292b1 100644
--- a/pip/commands/install.py
+++ b/pip/commands/install.py
@@ -24,6 +24,7 @@ from pip.utils.deprecation import RemovedInPip10Warning
 from pip.utils.filesystem import check_path_owner
 from pip.wheel import WheelCache, WheelBuilder
 
+from pip.locations import running_under_virtualenv
 
 logger = logging.getLogger(__name__)
 
@@ -54,6 +55,12 @@ class InstallCommand(RequirementCommand):
     def __init__(self, *args, **kw):
         super(InstallCommand, self).__init__(*args, **kw)
 
+        default_user = True
+        if running_under_virtualenv():
+            default_user = False
+        if os.geteuid() == 0:
+            default_user = False
+
         cmd_opts = self.cmd_opts
 
         cmd_opts.add_option(cmdoptions.constraints())
@@ -116,6 +123,7 @@ class InstallCommand(RequirementCommand):
             '-I', '--ignore-installed',
             dest='ignore_installed',
             action='store_true',
+            default=default_user,
             help='Ignore the installed packages (reinstalling instead).')
 
         cmd_opts.add_option(cmdoptions.ignore_requires_python())
@@ -128,10 +136,20 @@ class InstallCommand(RequirementCommand):
             '--user',
             dest='use_user_site',
             action='store_true',
+            default=default_user,
             help="Install to the Python user install directory for your "
                  "platform. Typically ~/.local/, or %APPDATA%\Python on "
                  "Windows. (See the Python documentation for site.USER_BASE "
-                 "for full details.)")
+                 "for full details.)  On Debian systems, this is the "
+                 "default when running outside of a virtual environment "
+                 "and not as root.")
+
+        cmd_opts.add_option(
+            '--system',
+            dest='use_user_site',
+            action='store_false',
+            help="Install using the system scheme (overrides --user on "
+                 "Debian systems)")
 
         cmd_opts.add_option(
             '--egg',

Which means that the Debian pip runs with --ignore-installed (unless executed as root or using a virtualenv)!

[benoit-pierre]

Of course, this patch is also present on Ubuntu since Xenial...

[benoit-pierre]

@warsaw: enabling --ignore-installed is a very bad idea! See pypa/setuptools#1104 (comment) and pypa/setuptools#1104 (comment) for (other) examples of why.

[rwb27]

good spot! I was going to flag that to the package maintainers, but it's been flagged already.

Installing system-wide would presumably not have this problem, but (unless there's a flag I've missed), there's no way to install a package for the user without ignoring installed packages. That is frustrating! I'll update the install instructions for my package, as I suspect folk will start having problems as soon as they're using the latest OS on their Raspberry Pis...

[benoit-pierre]

Just use get-pip.py --user to get a version of pip that actually works, and use that instead of the broken system version...

[rwb27]

Sure - though as my package is mostly used by beginners on Raspberry Pis, I was hoping to minimise the number of install steps and not mess too much with their Python environment. Installing a non-broken pip seems like the best option for now, though!

[benoit-pierre]

Actually, setting PIP_IGNORE_INSTALLED=0 in your environment might also work.

[benoit-pierre]

It does when tested locally with stock pip-9.0.1 + part of the above Debian patch.

[rwb27]

oh, that would be nice! I can confirm that running

export PIP_IGNORE_INSTALLED=0
pip install <my package>

also works nicely on Raspbian Stretch with the system pip 9.0.1. Thanks a million :)

[benoit-pierre]

This can be closed: it's a Debian/Ubuntu bug.

[chrahunt]

Closing as a downstream bug, as mentioned by @benoit-pierre.