Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pip-compile fails if pandas is in requirements.in #586

Closed
yuvallanger opened this issue Oct 24, 2017 · 2 comments
Closed

pip-compile fails if pandas is in requirements.in #586

yuvallanger opened this issue Oct 24, 2017 · 2 comments

Comments

@yuvallanger
Copy link

After adding the package pandas to the requirement.in file, running pip-compile results in an exception and a failure to produce a requirements.txt file.

Environment Versions
  1. Ubuntu 17.04
  2. Python version: Python 3.6.1
  3. pip version: pip 9.0.1 from /home/yuval/.virtualenvs/a/lib/python3.6/site-packages (python 3.6)
  4. pip-tools version: pip-compile, version 1.10.1
Steps to replicate
  1. Add pandas to your requirements.in
  2. Run pip-compile requirements.in
Expected result

pandas installed.

Actual result
Using indexes:
  https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  bokeh
  Cython
  dask[complete]
  graphviz
  h5py
  holoviews[all]
  jupyter_contrib_nbextensions
  matplotlib
  numpy
  pandas
  pip-tools
  scipy
  shapely[vectorized]
  tables

Finding the best candidates:
  found candidate bokeh==0.12.10 (constraint was <any>)
  found candidate cython==0.27.2 (constraint was <any>)
  found candidate dask[complete]==0.15.4 (constraint was <any>)
  found candidate graphviz==0.8.1 (constraint was <any>)
  found candidate h5py==2.7.1 (constraint was <any>)
  found candidate holoviews[all]==1.8.4 (constraint was <any>)
  found candidate jupyter-contrib-nbextensions==0.3.3 (constraint was <any>)
  found candidate matplotlib==2.1.0 (constraint was <any>)
  found candidate numpy==1.13.3 (constraint was <any>)
  found candidate pandas==0.20.3 (constraint was <any>)
  found candidate pip-tools==1.10.1 (constraint was <any>)
  found candidate scipy==0.19.1 (constraint was <any>)
  found candidate shapely[vectorized]==1.6.1 (constraint was <any>)
  found candidate tables==3.4.2 (constraint was <any>)

Finding secondary dependencies:
  jupyter-contrib-nbextensions==0.3.3 requires ipython-genutils, jupyter-contrib-core>=0.3.3, jupyter-core, jupyter-highlight-selected-word>=0.0.10, jupyter-latex-envs>=1.3.8, jupyter-nbextensions-configurator>=0.2.8, lxml>=3.8.0, nbconvert>=4.2, notebook>=4.0, pyyaml, tornado, traitlets>=4.1
  numpy==1.13.3             requires -
  pip-tools==1.10.1         requires click>=6, first, setuptools, six
  bokeh==0.12.10            requires Jinja2>=2.7, numpy>=1.7.1, python-dateutil>=2.1, PyYAML>=3.10, six>=1.5.2, tornado>=4.3
  matplotlib==2.1.0         requires backports.functools-lru-cache, cycler>=0.10, numpy>=1.7.1, pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1, python-dateutil>=2.0, pytz, six>=1.10, subprocess32
  graphviz==0.8.1           requires -
  cython==0.27.2            requires -
  holoviews[all]==1.8.4     requires bokeh>=0.12.6, cyordereddict, ipython, jinja2, jsonschema, lancet-ioam, matplotlib, nose, notebook, numpy>=1.0, pandas, param<2.0,>=1.5.1, pygments, pyzmq, seaborn, tornado
  scipy==0.19.1             requires numpy>=1.8.2
  h5py==2.7.1               requires numpy>=1.7, six
  dask[complete]==0.15.4    requires cloudpickle>=0.2.1; extra == "complete", distributed>=1.16; extra == "complete", numpy; extra == "complete", pandas>=0.19.0; extra == "complete", partd>=0.3.8; extra == "complete", toolz>=0.7.3; extra == "complete"
  pandas==0.20.3            requires numpy>=1.7.0, python-dateutil, pytz>=2011k
  tables==3.4.2             requires numexpr>=2.5.2, numpy>=1.8.0, six>=1.9.0
  shapely[vectorized]==1.6.1 requires numpy; extra == "vectorized"

New dependencies found in this round:
  adding ['backports.functools-lru-cache', '', '[]']
  adding ['bokeh', '>=0.12.6', '[]']
  adding ['click', '>=6', '[]']
  adding ['cloudpickle', '>=0.2.1', '[]']
  adding ['cycler', '>=0.10', '[]']
  adding ['cyordereddict', '', '[]']
  adding ['distributed', '>=1.16', '[]']
  adding ['first', '', '[]']
  adding ['ipython', '', '[]']
  adding ['ipython-genutils', '', '[]']
  adding ['jinja2', '>=2.7', '[]']
  adding ['jsonschema', '', '[]']
  adding ['jupyter-contrib-core', '>=0.3.3', '[]']
  adding ['jupyter-core', '', '[]']
  adding ['jupyter-highlight-selected-word', '>=0.0.10', '[]']
  adding ['jupyter-latex-envs', '>=1.3.8', '[]']
  adding ['jupyter-nbextensions-configurator', '>=0.2.8', '[]']
  adding ['lancet-ioam', '', '[]']
  adding ['lxml', '>=3.8.0', '[]']
  adding ['matplotlib', '', '[]']
  adding ['nbconvert', '>=4.2', '[]']
  adding ['nose', '', '[]']
  adding ['notebook', '>=4.0', '[]']
  adding ['numexpr', '>=2.5.2', '[]']
  adding ['numpy', '>=1.0,>=1.7,>=1.7.0,>=1.7.1,>=1.8.0,>=1.8.2', '[]']
  adding ['pandas', '>=0.19.0', '[]']
  adding ['param', '<2.0,>=1.5.1', '[]']
  adding ['partd', '>=0.3.8', '[]']
  adding ['pygments', '', '[]']
  adding ['pyparsing', '!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1', '[]']
  adding ['python-dateutil', '>=2.0,>=2.1', '[]']
  adding ['pytz', '>=2011k', '[]']
  adding ['pyyaml', '>=3.10', '[]']
  adding ['pyzmq', '', '[]']
  adding ['seaborn', '', '[]']
  adding ['six', '>=1.10,>=1.5.2,>=1.9.0', '[]']
  adding ['subprocess32', '', '[]']
  adding ['toolz', '>=0.7.3', '[]']
  adding ['tornado', '>=4.3', '[]']
  adding ['traitlets', '>=4.1', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  backports.functools-lru-cache
  bokeh>=0.12.6
  click>=6
  cloudpickle>=0.2.1
  cycler>=0.10
  cyordereddict
  Cython
  dask[complete]
  distributed>=1.16
  first
  graphviz
  h5py
  holoviews[all]
  ipython
  ipython-genutils
  Jinja2>=2.7
  jsonschema
  jupyter-contrib-core>=0.3.3
  jupyter_contrib_nbextensions
  jupyter-core
  jupyter-highlight-selected-word>=0.0.10
  jupyter-latex-envs>=1.3.8
  jupyter-nbextensions-configurator>=0.2.8
  lancet-ioam
  lxml>=3.8.0
  matplotlib
  nbconvert>=4.2
  nose
  notebook>=4.0
  numexpr>=2.5.2
  numpy>=1.0,>=1.7,>=1.7.0,>=1.7.1,>=1.8.0,>=1.8.2
  pandas>=0.19.0
  param<2.0,>=1.5.1
  partd>=0.3.8
  pip-tools
  pygments
  pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1
  python-dateutil>=2.0,>=2.1
  pytz>=2011k
  pyyaml>=3.10
  pyzmq
  scipy
  seaborn
  shapely[vectorized]
  six>=1.10,>=1.5.2,>=1.9.0
  subprocess32
  tables
  toolz>=0.7.3
  tornado>=4.3
  traitlets>=4.1

Finding the best candidates:
  found candidate backports.functools-lru-cache==1.4 (constraint was <any>)
  found candidate bokeh==0.12.10 (constraint was >=0.12.6)
  found candidate click==6.7 (constraint was >=6)
  found candidate cloudpickle==0.4.0 (constraint was >=0.2.1)
  found candidate cycler==0.10.0 (constraint was >=0.10)
  found candidate cyordereddict==1.0.0 (constraint was <any>)
  found candidate cython==0.27.2 (constraint was <any>)
  found candidate dask[complete]==0.15.4 (constraint was <any>)
  found candidate distributed==1.19.3 (constraint was >=1.16)
  found candidate first==2.0.1 (constraint was <any>)
  found candidate graphviz==0.8.1 (constraint was <any>)
  found candidate h5py==2.7.1 (constraint was <any>)
  found candidate holoviews[all]==1.8.4 (constraint was <any>)
  found candidate ipython==6.2.1 (constraint was <any>)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate jinja2==2.9.6 (constraint was >=2.7)
  found candidate jsonschema==2.6.0 (constraint was <any>)
  found candidate jupyter-contrib-core==0.3.3 (constraint was >=0.3.3)
  found candidate jupyter-contrib-nbextensions==0.3.3 (constraint was <any>)
  found candidate jupyter-core==4.3.0 (constraint was <any>)
  found candidate jupyter-highlight-selected-word==0.1.0 (constraint was >=0.0.10)
  found candidate jupyter-latex-envs==1.3.8.4 (constraint was >=1.3.8)
  found candidate jupyter-nbextensions-configurator==0.2.8 (constraint was >=0.2.8)
  found candidate lancet-ioam==0.9.0 (constraint was <any>)
  found candidate lxml==4.1.0 (constraint was >=3.8.0)
  found candidate matplotlib==2.1.0 (constraint was <any>)
  found candidate nbconvert==5.3.1 (constraint was >=4.2)
  found candidate nose==1.3.7 (constraint was <any>)
  found candidate notebook==5.2.0 (constraint was >=4.0)
  found candidate numexpr==2.6.4 (constraint was >=2.5.2)
  found candidate numpy==1.13.3 (constraint was >=1.0,>=1.7,>=1.7.0,>=1.7.1,>=1.8.0,>=1.8.2)
  found candidate pandas==0.20.3 (constraint was >=0.19.0)
  found candidate param==1.5.1 (constraint was >=1.5.1,<2.0)
  found candidate partd==0.3.8 (constraint was >=0.3.8)
  found candidate pip-tools==1.10.1 (constraint was <any>)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate pyparsing==2.2.0 (constraint was >=2.0.1,!=2.0.4,!=2.1.2,!=2.1.6)
  found candidate python-dateutil==2.6.1 (constraint was >=2.0,>=2.1)
  found candidate pytz==2017.2 (constraint was >=2011k)
  found candidate pyyaml==3.12 (constraint was >=3.10)
  found candidate pyzmq==16.0.2 (constraint was <any>)
  found candidate scipy==0.19.1 (constraint was <any>)
  found candidate seaborn==0.8.1 (constraint was <any>)
  found candidate shapely[vectorized]==1.6.1 (constraint was <any>)
  found candidate six==1.11.0 (constraint was >=1.10,>=1.5.2,>=1.9.0)
  found candidate subprocess32==3.2.7 (constraint was <any>)
  found candidate tables==3.4.2 (constraint was <any>)
  found candidate toolz==0.8.2 (constraint was >=0.7.3)
  found candidate tornado==4.5.2 (constraint was >=4.3)
  found candidate traitlets==4.3.2 (constraint was >=4.1)

Finding secondary dependencies:
  cycler==0.10.0            requires six
  jupyter-latex-envs==1.3.8.4 requires ipython, jupyter_core, nbconvert, notebook>=4.0, traitlets>=4.1
  pip-tools==1.10.1         requires click>=6, first, setuptools, six
  jsonschema==2.6.0         requires -
  dask[complete]==0.15.4    requires cloudpickle>=0.2.1; extra == "complete", distributed>=1.16; extra == "complete", numpy; extra == "complete", pandas>=0.19.0; extra == "complete", partd>=0.3.8; extra == "complete", toolz>=0.7.3; extra == "complete"
  subprocess32==3.2.7 not in cache, need to check index
Traceback (most recent call last):
  File "/home/yuval/.local/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/scripts/compile.py", line 184, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/resolver.py", line 101, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/resolver.py", line 198, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/resolver.py", line 284, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/repositories/pypi.py", line 170, in get_dependencies
    self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmp2dsm6oyqbuild/subprocess32/
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "/home/yuval/.local/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/yuval/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/scripts/compile.py", line 184, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/resolver.py", line 101, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/resolver.py", line 198, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/resolver.py", line 284, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/yuval/.local/lib/python3.6/site-packages/piptools/repositories/pypi.py", line 170, in get_dependencies
    self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/yuval/.local/lib/python3.6/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmp2dsm6oyqbuild/subprocess32/
@yuvallanger yuvallanger changed the title Problem with pip-compile and pandas pip-compile fails if pandas is in requirements.in Oct 24, 2017
@vphilippon
Copy link
Member

vphilippon commented Oct 24, 2017 •
edited
Loading

That's an issue fixed by PR #571. pip-tools is fetching a Python 2 dependency (subprocess32) even if its running in a Python 3 environment. It fixed on master, but not released yet (see #531, we can't make new releases at the moment).

Try it out on master, an confirm if it works.
If it doesn't work, run pip-compile --rebuild to clear the dependency cache that could be corrupted.

(P.S. Thanks for the well done report, it makes spotting those kind of things so much faster!)

@vphilippon
Copy link
Member

This is now fixed in 1.10.2
Make sure to clear you dependency cache: pip-compile --rebuild

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuvallanger @vphilippon