pip 19.0 fails to install packages that import to-be-installed package from CWD

[jeanlaroche]

Environment

• pip version: 19.0
• Python version: 3.6
• OS: MacOS

Description
When running pip install pyinstaller==3.4 with pip 19.0 we are getting an install error. ModuleNotFoundError: No module named 'PyInstaller'

Expected behavior
Expect pyinstall to be installed, as it is with pip 18.1

How to Reproduce
Using python3:
pip install pyinstaller=3.4

Output

pip install pyinstaller==3.4
Collecting pip
  Using cached https://files.pythonhosted.org/packages/60/64/73b729587b6b0d13e690a7c3acd2231ee561e8dd28a58ae1b0409a5a2b20/pip-19.0-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-19.0
(BuildVEnv) jlaroche-mbp:TrackSense$ pip install pyinstaller
Collecting pyinstaller
  Using cached https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  Complete output from command /Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/bin/python3 /Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/tmps3z6flnv:
  Traceback (most recent call last):
    File "/Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/pip-build-env-lo_ir5_f/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 115, in get_requires_for_build_wheel
      return _get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/pip-build-env-lo_ir5_f/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 101, in _get_build_requires
      _run_setup()
    File "/private/var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/pip-build-env-lo_ir5_f/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 85, in _run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 20, in <module>
      from PyInstaller import __version__ as version, HOMEPATH, PLATFORM
  ModuleNotFoundError: No module named 'PyInstaller'

---

Maintainer note on timeline: See #6163 (comment)

[pradyunsg]

This seems to be an issue with either how pyinstaller is importing itself for installation.

It might be a good idea to file an issue over at the PyInstaller folks.

[rt121212121]

We currently use 18.1, and upgrading to 19.0 causes this problem for us as well. There is a related issue on the PyInstaller repo, where it is because pip '' is not in sys.path.

pyinstaller/pyinstaller#2730

[lithammer]

I think this is a pretty common workflow. You put __version__ = "1.2.3" in foo/__init__.py and then do import foo in setup.py so that you don't have to specify the version in two places. And any user of the the library can inspect the version according to PEP 396.

# foo/__init__.py
__version__ = "1.2.3"

# setup.py
from setuptools import setup

import foo

setup(..., version=foo.__version__)

[lithammer]

Also, this only happens if you have a pyproject.toml file (and setup.py). Remove it and the installation works fine. So there seems to be some differences in behaviour there. Maybe the traditional way modifies sys.path/PYTHONPATH?

[lithammer]

Ah, I think I get what is happening. Since by using a pyproject.toml file, you're basically telling pip you want to use PEP 517/518.

# pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]

The above tells pip that it needs setuptools and wheel to build PyInstaller. But in the case of PyInstaller, it's also got this in its setup.py:

# setup.py
from PyInstaller import __version__

From a PEP 517 perspective, aside from setuptools and wheel, it means it needs itself to build. Which is of course a bit weird.

# pyproject.toml
[build-system]
requires = ["setuptools", "wheel", "PyInstaller"]

[dstufft]

As @cjerdonek mentioned in #6175 (comment), could someone please check if --no-use-pep517 fixes this for them?

I suspect the cause of this issue is that build isolation or the PEP 517 code isn't making sure that the root of the package directory is on the sys.path, because pandas has a versioneer.py sitting next to setup.py. I recall this coming up at some point, but I don't remember off the top of my head what that discussion was. This might be considered an issue with the setuptools build backend instead of pip, or it might be the fault of pip's isolation mechanism.

[lithammer]

[...] could someone please check if --no-use-pep517 fixes this for them?

PyInstaller installs fine with --no-use-pep517.

[dstufft]

Ok, then that's certainly an issue with the new PEP 517 code and I'm pretty sure the issue is just that the directory containing the project root hasn't been added to sys.path. Maybe @pfmoore will have a better sense of if that should be pip's responsbility or setuptools.

[ashb]

If it helps another example of this (via apache-airflow is pip install pendulum==1.4.4 fails, but pip install --no-use-pep517 pendulum==1.4.4 works.

The stack trace we get is similar:

Collecting pendulum==1.4.4
  Using cached https://files.pythonhosted.org/packages/85/a5/9fc15751f9725923b170ad37d6c61031fc9e941bafd5288ca6ee51233284/pendulum-1.4.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  Complete output from command /Users/ash/.virtualenvs/clean-airflow/bin/python3.7 /Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/tmprosed3kj:
  Traceback (most recent call last):
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/pip-build-env-g__m0jh6/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 115, in get_requires_for_build_wheel
      return _get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/pip-build-env-g__m0jh6/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 101, in _get_build_requires
      _run_setup()
    File "/private/var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/pip-build-env-g__m0jh6/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 85, in _run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 47, in <module>
      from build import *
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/build.py", line 7, in <module>
      from pip._vendor import pytoml
  ModuleNotFoundError: No module named 'pip'

[dlamblin]

Also the installation of the following also doesn't work with pip v 19.0 but would if using the --no-use-pep5517:
pendulum==1.5.0 (AttributeError: module 'enum' has no attribute 'IntFlag')
pendulum==1.5.1 (AttributeError: module 'enum' has no attribute 'IntFlag')
pendulum==2.0.0 (AttributeError: module 'enum' has no attribute 'IntFlag')
pendulum==2.0.1 (AttributeError: module 'enum' has no attribute 'IntFlag')
pendulum==2.0.2 (AttributeError: module 'enum' has no attribute 'IntFlag')

Whereas 2.0.3 and 2.0.4 install fine.

[TimoRoth]

cartopy(At least their latest release) also fails to install since 19.0, failing to import its versioneer.py that's next to setup.py

[fchorney]

This is also an issue with some projects that I deal with. We use a pyproject.toml to define our python black parameters, and do a similar from project.version import __version__ in our setup.py.

At the very least I feel like being able to define no project isolation in the pyproject.toml would be sufficient. It seems unreasonable to me to make anybody wanting to install the project to use --no-buid-isolation or --no-use-pep517

[pfmoore]

The failure appears to be in get_requires_for_build_wheel, and the setuptools backend runs setup.py to do some sort of introspection to determine build requirements (the specific code is here). That code appears weird to me, and I don't understand why it's required. My initial instinct is that this is a bug in the setuptools backend that should be addressed by them.

PEP 517 does not state that frontends should run hooks in an environment that adds the build directory to sys.path, and there's a potential concern that if we did that, it could break isolation (if the build directory contained a copy of some required but not specified package, for instance). So my preference would be to not add the build directory to sys.path. But it may be expedient to do so if that offers a quick fix for this regression. I don't think projects should rely on this, though.

Summary:

1. This should be reported to setuptools for review as a backend issue. I'd consider fixing it in the setuptools backend (possibly just by them adding the build directory to sys.path) as the ideal resolution.
2. If setuptools doesn't do it, pip could add the build directory to sys.path, but I don't think that PEP 517 views that as the frontend's responsibility..
3. Requiring the build directory to be visible to hooks on sys.path would require at least a PEP clarification.

I don't think this scenario was considered when PEP 517 was being developed. Maybe because it's setuptools-specific (or rather, specific to backends that run arbitrary Python code as part of the build).

[dstufft]

I think it's fairly common for people to import something from the current directory into a setup.py, and just generally treat things as if setup.py is in $PWD.

I think it's reasonable to push this responsibility onto setuptools, since that's probably the only project that really needs it.

[pfmoore]

Yep, thinking about this some more, I'm certain it's a setuptools backend responsibility. Pre-PEP 517, pip ran setup.py as a script, so standard Python rules put the script directory onto sys.path. Under PEP 517, invocation of setup.py is replaced with calls to the backend hooks, so those hooks need to preserve the semantics. Because setuptools runs setup.py in-process from the hooks, it needs to manage sys.path itself. Hopefully, it's not a big fix for them. @jeanlaroche (or someone else hitting this issue) could you raise an issue on the setuptools tracker, referring back to this thread?

[adembo]

[...] could someone please check if --no-use-pep517 fixes this for them?

I can confirm that --no-use-pep517 allows pip install pandas to succeed.

[Overdrivr]

Any ETA on the new release that will integrate the fix ?

[pradyunsg]

In a few hours. :)

See the pinned issue.

[pradyunsg]

pip 19.0.2 has been released with a fix for this.

[ghost]

I am unable to install pyinstaller with the lastest version of pip, even when using --no-use-pep517

pip install pyinstaller --no-cache-dir --no-use-pep517
Collecting pyinstaller
  Downloading https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz (3.5MB)
     |████████████████████████████████| 3.5MB 273kB/s
  Installing build dependencies ... done
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Raffaele\AppData\Local\Temp\pip-install-5e9w2p2c\pyinstaller\setup.py", line 20, in <module>
        from PyInstaller import __version__ as version, HOMEPATH, PLATFORM
    ModuleNotFoundError: No module named 'PyInstaller'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in C:\Users\Raffaele\AppData\Local\Temp\pip-install-5e9w2p2c\pyinstaller\

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.