Conditional requirements are not checked when loaded from cache

[ekohilas]

• Pip version: 9.0.1
• Python version: 3.6.1/3.6.2
• Operating system: Linux Mint

Description:

I was trying to download a package which had a python_version = "2.7" requirement marker.
I ran python3.6 -m pip install fbchat --upgrade
and got

Collecting fbchat
  Downloading fbchat-1.0.23.tar.gz
  Ignoring enum34: markers 'python_version == "2.7"' don't match your environment
Requirement already up-to-date: requests in /home/user/.local/lib/python3.6/site-packages (from fbchat)
Requirement already up-to-date: lxml in /home/user/.local/lib/python3.6/site-packages (from fbchat)
Requirement already up-to-date: beautifulsoup4 in /home/user/.local/lib/python3.6/site-packages (from fbchat)
Requirement already up-to-date: idna<2.7,>=2.5 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Requirement already up-to-date: certifi>=2017.4.17 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Building wheels for collected packages: fbchat
  Running setup.py bdist_wheel for fbchat ... done
  Stored in directory: /home/user/.cache/pip/wheels/e0/1a/94/54cf9fc7001ca3288e6d9cc19e3ca067e22bf8ac8aa2070ce0
Successfully built fbchat
Installing collected packages: fbchat
  Found existing installation: fbchat 1.0.21
    Uninstalling fbchat-1.0.21:
      Successfully uninstalled fbchat-1.0.21
  Rolling back uninstall of fbchat
Exception:
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/user/.local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/home/user/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/home/user/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/home/user/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/home/user/.local/lib/python3.6/site-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/home/user/.local/lib/python3.6/site-packages/pip/wheel.py", line 316, in clobber
    ensure_dir(destdir)
  File "/home/user/.local/lib/python3.6/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)
  File "/usr/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/fbchat'

Note that enum34 was ignored.
It then failed because I forgot to specify the --user flag, and I didn't run the command in sudo.
Then, when I ran the command again with --user the conditional requirement wasn't checked, so enum34 was installed anyway, when it shouldn't have.

$ python3.6 -m pip install fbchat --upgrade --user
Collecting fbchat
Requirement already up-to-date: requests in /home/user/.local/lib/python3.6/site-packages (from fbchat)
Requirement already up-to-date: lxml in /home/user/.local/lib/python3.6/site-packages (from fbchat)
Collecting enum34 (from fbchat)
  Using cached enum34-1.1.6-py3-none-any.whl
Requirement already up-to-date: beautifulsoup4 in /home/user/.local/lib/python3.6/site-packages (from fbchat)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Requirement already up-to-date: certifi>=2017.4.17 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Requirement already up-to-date: idna<2.7,>=2.5 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in /home/user/.local/lib/python3.6/site-packages (from requests->fbchat)
Installing collected packages: enum34, fbchat
  Found existing installation: fbchat 1.0.21
    Uninstalling fbchat-1.0.21:
      Successfully uninstalled fbchat-1.0.21
Successfully installed enum34-1.1.6 fbchat-1.0.23

Running the second command on my other laptop worked fine.
After uninstalling both fbchat and enum34, only running the command with --no-cache-dir would follow the conditional requirement.
I believe this is because on crash, the conditional requirements aren't extracted to the cache correctly.

I may be able to fix this problem, but I'm not sure where to look.

[benoit-pierre]

Does it work if you use --no-binary :all:?

[ekohilas]

@benoit-pierre Did you want me to uninstall both, simulate another crash and then add that flag?

[benoit-pierre]

No, no need to simulate another crash. I think I can reproduce it: the problem is with the cached wheel. Try:

• pip uninstall fbchat
• pip install fbchat: you should see the bug again
• pip uninstall fbchat
• pip install --no-binary :all: fbchat: this should show that enum34 is being ignored

[ekohilas]

@benoit-pierre correct

[benoit-pierre]

The problem is the setuptools version that was used to create the (cached) wheel. If you purge pip's cache, update setuptools to at least 36.2.7, and retry, you should get the same (correct) behavior wether or not the cache is hit.

[benoit-pierre]

For reference, see:

• https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3627
• https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3620
• Fix environment markers handling setuptools#1081

[ekohilas]

@benoit-pierre to clarify, is this something that the package owner should be doing, or the installee?

[benoit-pierre]

• the installee can use a more recent version of setuptools, so environment markers in install_requires are properly supported
• the package owner could fix its setup.py so older setuptools versions are still supported:

 setup.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git i/setup.py w/setup.py
index 2b8560a..80e5559 100644
--- i/setup.py
+++ w/setup.py
@@ -20,8 +20,12 @@ requirements = [
     'requests',
     'lxml',
     'beautifulsoup4',
-    "enum34; python_version == '2.7'"
 ]
+extras_requirements = {
+    ':python_version == "2.7"': [
+        'enum34',
+    ]
+}
 
 version = None
 author = None
@@ -77,6 +81,7 @@ setup(
     include_package_data=True,
     packages=['fbchat'],
     install_requires=requirements,
+    extras_require=extras_requirements,
     url=source,
     version=version,
     zip_safe=True,

[ekohilas]

Thank you, I will forward this on :)

[pradyunsg]

Thanks @benoit-pierre for looking into this! ^>^

[ekohilas]

@benoit-pierre
The : in

extras_requirements = {
    ':python_version == "2.7"'

shouldn't be there right?

[benoit-pierre]

The : is important, see the documentation.

[ekohilas]

Oh I see. Thanks!

[pradyunsg]

Can this issue be closed?

[benoit-pierre]

I think it can.