RuntimeError: Python version >= 3.5 required.

[tysonclugg]

Environment

• pip version: 19.1.1
• Python version: 2.7.12
• OS: Ubuntu 16.04

Description
Fails to install packages from source where install_requires includes versions have incompatible python_requires.

Expected behavior
Source package should be installed with compatible versions of packages listed in install_requires from setup.py. Eg: Installing pandas in Python 2.7 should result in numpy version 1.16.4 being installed (the last 2.7 compatible release before 1.17.0rc1).

How to Reproduce

1. Run pip install --no-binary=pandas pandas in a Python 2.7 environment.
2. An error occurs.

Output

$ mkvirtualenv -ppython2.7 pip27
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /home/tclugg/.virtualenvs/pip27/bin/python2.7
Also creating executable in /home/tclugg/.virtualenvs/pip27/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/tclugg/.virtualenvs/pip27/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/tclugg/.virtualenvs/pip27/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/tclugg/.virtualenvs/pip27/bin/preactivate
virtualenvwrapper.user_scripts creating /home/tclugg/.virtualenvs/pip27/bin/postactivate
virtualenvwrapper.user_scripts creating /home/tclugg/.virtualenvs/pip27/bin/get_env_details
Python 2.7.12
(pip27) $ pip install -U pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: pip in /home/tclugg/.virtualenvs/pip27/lib/python2.7/site-packages (19.1.1)
(pip27) $ pip install --no-binary=pandas pandas
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting pandas
  Using cached https://files.pythonhosted.org/packages/b2/4c/b6f966ac91c5670ba4ef0b0b5613b5379e3c7abdfad4e7b89a87d73bae13/pandas-0.24.2.tar.gz
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-rC8dFo/pandas/setup.py", line 746, in <module>
        **setuptools_kwargs)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/dist.py", line 717, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/dist.py", line 784, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/home/tclugg/.virtualenvs/pip27/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-sfO_WV/numpy-1.17.0rc1/setup.py", line 31, in <module>
    
    RuntimeError: Python version >= 3.5 required.
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-rC8dFo/pandas/

Paste the output of the steps above, including the commands themselves and
pip's output/traceback etc.

[xavfernandez]

pandas is still using setup_requires mechanism ( cf https://github.com/pandas-dev/pandas/blob/v0.24.2/setup.py#L40) which invokes easy_install behind pip's back, and apparently easy_install does not honor python_requires declarations.

You should install numpy first with pip that will honor python_requires, then install pandas.
And to definitely fix this kind of situation, pandas should start using PEP-518 build-system.requires to declare its build dependencies.

[tysonclugg]

@xavfernandez I understand what you're saying in that easy_install is effectively an escape hatch out from pip into easy_install.

I invoked pip, and was presented with a cryptic traceback. I am fortunate enough that I could understand enough to file this bug report. You've been kind enough to give me both a valid work-around, and made a good suggestion as to what the pandas team can do for future releases. Thanks!

However, you've failed to convince me this is someone else's problem. At some point, it was pip that delegated the setup_requires task to easy_install (or perhaps to some other intermediary). I didn't invoke easy_install, and nor did pandas.

I feel that pip should do one of the following:

1. Stop delegating to external tools (ie: the buck stops here).
2. Present a big warning before and/or after delegating to external tools, so that package developers (eg: the pandas team) are aware of issues with their packages (ie: pass the buck).

Not doing either of the above will mean more pip users are left staring at cryptic tracebacks, and further bug reports being logged.

[tysonclugg]

You should install numpy first with pip that will honor python_requires, then install pandas.

This works as expected. Thanks. :-)

And to definitely fix this kind of situation, pandas should start using PEP-518 build-system.requires to declare its build dependencies.

FYI: A pull request to introduce (or reintroduce?) pyproject.toml in pandas was closed only 4 days ago. The comments cited various issues with pip, and the final verdict was that they didn't feel using pyproject.toml was necessary. Given the issues pandas experienced resulting from pip bugs, I'm not surprised they're holding back for the moment.

[TomAugspurger]

the final verdict was that they didn't feel using pyproject.toml was necessary.

Just to clarify, pandas wants to adopt pyproject.toml, but it's not a blocker for our next release.

[xavfernandez]

However, you've failed to convince me this is someone else's problem. At some point, it was pip that delegated the setup_requires task to easy_install (or perhaps to some other intermediary). I didn't invoke easy_install, and nor did pandas.

Well, pip has no concept of setup_requires (that's a big part of PEP-518 rationale ).

Since you are specifying --no-binary=pandas, you're basically saying to pip: "go ahead, execute pandas' setup.py".
That's where setuptools (pandas' build tool) checks setup_requires, sees that numpy isn't available and decides to use easy_install to provide it.
So I'd say that, technically, pandas did invoke easy_install.

[pradyunsg]

Yea, there's nothing actionable here from pip's end -- in an ideal world, pandas would to switch over to PEP 518.

Until that happens, it'd be best for pandas to improve error messaging that's printed when there's some sort of incompatibility.

[TomAugspurger]

@tysonclugg speaking as a maintainer (not a pip maintainer), those kinds of comments are disheartening to read :/

[tysonclugg]

@TomAugspurger I was miffed that the response had been "technically, pandas ..." and "nothing actionable by pip".

I feel there is a reasonable case to update the docs, or the output of pip install --help, or to emit a warning when installing source packages. Instead, I feel my concerns have been brushed aside.

You're point is heeded though, the frustrated (and abrasive) tone of my previous comment was unnecessary. Apologies to all involved.

[pradyunsg]

Uhhhhh. I've also hidden your comment since I don't want to be reading that again.

I understand that this can get a little frustrating. Let me clarify what @xavfernandez and I have been saying, since I don't think we've been super verbose/clear with our reasoning here. Apologies for that.

---

This error is originating from setuptools's logic for handling setup_requires. This was a mechanism that existed to help ensure dependencies for running setup.py are available. If you have a keen eye, you'll notice that this means running setup.py to get information for what's needed to run setup.py. That's why setup_requires is not a good thing. Further, setuptools does not expose the information about setup_requires in any way for pip to satisfy that requirement before executing the setup.py script.

pip developers, setuptools developers and a lot more people, worked together on PEP 518 to help with exactly this situation: specifying build dependencies. That's why one actionable item here is that pandas adopt the use of PEP 518 (i.e. use pyproject.toml). I understand that there were growing pains to adopting this new standard. None the less, we're almost across the bridge on that; and @TomAugspurger has stated that pandas is open to doing so. :)

---

I feel that pip should do one of the following:

I appreciate your suggestions but I think neither of them are actionable suggestions for pip.

• Stop delegating to external tools (ie: the buck stops here).

pip does not have any logic for converting a source distribution to an installable artifact. It depends on external "build backends" for doing that. So, well, this is out of scope for this tool. :)

• Present a big warning before and/or after delegating to external tools, so that package developers (eg: the pandas team) are aware of issues with their packages (ie: pass the buck).

That's status quo and what we do already.

The fact that pip depends on build backends is known to packager since they have to use the backend and tell it how to package things. Further, pip already states when it invokes a build backend. From your output in the first post:

ERROR: Complete output from command python setup.py egg_info:
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-rC8dFo/pandas/

To reiterate, pip's delegating to and printing the output from pandas's setup.py. Because of that, pip can't really do much here. The most I can think that pip can do is, change the error message to say "hey, talk to the maintainers of pandas about this failure" but that's not exactly a thing we want to do without considering the implications of that.

---

Further, both pandas and setuptools are in a position to improve the output for this scenario:

• pandas can either handle this case explicitly in their setup.py and print a friendlier message; or adopt pyproject.toml so that pip handles this for them.
  • If, for some reason, the new standard does not fit their use case, they can poke us at https://discuss.python.org/c/packaging and we can figure out the details there.
• setuptools can improve error messaging when a setup_requires; though I'm not sure how setuptools wants to handle that option, in a world where pyproject.toml is a thing.

---

I'm going to step back from this issue now. Don't @ me. :)

[chrahunt]

I've extracted the suggestion above (change the error message to say "hey, talk to the maintainers of pandas about this failure") into issue #6959. I don't see anything else that needs to be addressed here and it's been awhile, so I will close this issue.