Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build uninstalls setuptools even if needed by a dependency #231

Closed
astrojuanlu opened this issue Feb 12, 2021 · 12 comments · Fixed by #232
Closed

build uninstalls setuptools even if needed by a dependency #231

astrojuanlu opened this issue Feb 12, 2021 · 12 comments · Fixed by #232

Comments

@astrojuanlu
Copy link

astrojuanlu commented Feb 12, 2021

This is a continuation of #109, although slightly different. Not really, setuptools is still installed in my venv after this happens. So, build isolation is working.

I have a project that uses flit as build backend, however some of its dependencies use setuptools as build backend. When I call python -m build ., what I observe is:

  1. setuptools gets uninstalled
  2. Build dependencies are installed
  3. Runtime dependencies start being downloaded
  4. When it reaches a dependency that needs setuptools, it fails with No module named 'setuptools'

Example:

$ python -m build .                                                                                                                                                                                                                                                    
Found existing installation: setuptools 49.2.1                                                                                                                                                                                                                                                                               
Uninstalling setuptools-49.2.1:                                                                                                                                                                                                                                                                                              
  Successfully uninstalled setuptools-49.2.1                                                                                                                                                                                                                                                                                 
Collecting oldest-supported-numpy                                                                                                                                                                                                                                                                                            
  Using cached oldest_supported_numpy-0.8-py3-none-any.whl (3.7 kB)                                                                                                                                                                                                                                                          
Collecting flit_core<3,>=2.0                                                                                                                                                                                                                                                                                                 
  Using cached flit_core-2.3.0-py2.py3-none-any.whl (40 kB)                                                                                                                                                                                                                                                                  
Collecting wheel                                                                                                                                                                                                                                                                                                             
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB) 
Collecting numpy==1.19.3; python_version == "3.9" and platform_python_implementation != "PyPy"                                                                                                                                                                                                                       [79/560]
  Using cached numpy-1.19.3-cp39-cp39-manylinux2010_x86_64.whl (14.9 MB)                                                                                                                                                                                                                                                     
Collecting pytoml                                                                                                                                                                                                                                                                                                            
  Using cached pytoml-0.1.21-py2.py3-none-any.whl (8.5 kB)                                                                                                                                                                                                                                                                   
Installing collected packages: numpy, oldest-supported-numpy, pytoml, flit-core, wheel                                                                                                                                                                                                                                       
Successfully installed flit-core-2.3.0 numpy-1.19.3 oldest-supported-numpy-0.8 pytoml-0.1.21 wheel-0.36.2                                                                                                                                                                                                                    
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.                                                                                                                                                                                                                                             
You should consider upgrading via the '/tmp/build-env-yoyx_wof/bin/python -m pip install --upgrade pip' command.                                                                                                                                                                                                             
Ignoring pycodestyle: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                  
Ignoring hypothesis: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                   
Ignoring tox: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                          
Ignoring sphinx: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                       
Ignoring coverage: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                     
Ignoring myst-parser: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                  
Ignoring sphinx-gallery: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                               
Ignoring notebook: markers 'extra == "jupyter"' don't match your environment                                                                                                                                                                                                                                                 
Ignoring isort: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                        
Ignoring nbconvert: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                    
Ignoring sphinx-notfound-page: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                         
Ignoring pytest-mypy: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                  
Ignoring sphinx-autoapi: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                               
Ignoring sphinx-rtd-theme: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                             
Ignoring jupytext: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                     
Ignoring pytest-remotedata: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                            
Ignoring nbsphinx: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                     
Ignoring ipywidgets: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                                                                   
Ignoring mypy: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                            
Ignoring pytest-cov: markers 'extra == "dev"' don't match your environment                                                                                                                                                                    
Ignoring black: markers 'extra == "dev"' don't match your environment                                                                                                                                                                         
Ignoring czml3: markers 'extra == "cesium"' don't match your environment                                               
Ignoring pytest: markers 'extra == "dev"' don't match your environment                                                                                                                                                                        
Ignoring pytest-mpl: markers 'extra == "dev"' don't match your environment                                             
Ignoring jupyter-client: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                  
Ignoring ipython: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                                         
Ignoring pytest-doctestplus: markers 'extra == "dev"' don't match your environment                                                                                                                                                                                              
Ignoring sphinx-hoverxref: markers 'extra == "dev"' don't match your environment                                       
Ignoring cached-property: markers 'python_version < "3.8"' don't match your environment                                                                                                                                                                                                                                      
Ignoring ipykernel: markers 'extra == "dev"' don't match your environment                                              
Collecting matplotlib!=3.0.1,>=2.0                                                                                                                                                                                                                                                                                           
  Downloading matplotlib-3.3.4-cp39-cp39-manylinux1_x86_64.whl (11.5 MB)                                               
     |████████████████████████████████| 11.5 MB 1.8 MB/s                                                                                                                                                                                                                                                                     
Collecting astropy<5,>=3.2                                                                                             
  Downloading astropy-4.2-cp39-cp39-manylinux1_x86_64.whl (9.6 MB)                                                     
     |████████████████████████████████| 9.6 MB 1.5 MB/s                                                                                                                                                                                                                         
Collecting numba>=0.46                                                                                                                                                                                                                        
  Using cached numba-0.51.2.tar.gz (2.1 MB)                                                                                                                                                                                                                                                                                  
    ERROR: Command errored out with exit status 1:                                                                                                                                                                                                                                                                           
     command: /tmp/build-env-yoyx_wof/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vpxvqzau/numba/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vpxvqzau/numba/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');
f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-5n_1l6e9                                                                                                                                   
         cwd: /tmp/pip-install-vpxvqzau/numba/                                                                                                                
    Complete output (3 lines):                                                                                         
    Traceback (most recent call last):                 
      File "<string>", line 1, in <module>                                                                             
    ModuleNotFoundError: No module named 'setuptools'                                                                                   
    ----------------------------------------                                                                                            
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                                                              
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.                                       
You should consider upgrading via the '/tmp/build-env-yoyx_wof/bin/python -m pip install --upgrade pip' command.                                                                                                                                                                
Traceback (most recent call last):                                                                                                      
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main                                                 
    return _run_code(code, main_globals, None,                                                                         
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code                                                            
    exec(code, run_globals)                                                                                            
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 214, in <module>                                                                                                               
    main(sys.argv[1:], 'python -m build')                                                                              
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 206, in main                                                                                                                   
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)  
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 94, in build_package                                                                                                           
    _build_in_isolated_env(builder, outdir, distributions, config_settings)                   
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/__main__.py", line 58, in _build_in_isolated_env                                                                                                  
    env.install(builder.get_dependencies(distribution))                                                                
  File "/home/juanlu/Personal/poliastro/poliastro/.venv39/lib/python3.9/site-packages/build/env.py", line 143, in install                                                                                                                     
    subprocess.check_call(cmd)                                                                                         
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call                                                     
    raise CalledProcessError(retcode, cmd)                                                                             
subprocess.CalledProcessError: Command '['/tmp/build-env-yoyx_wof/bin/python', '-Im', 'pip', 'install', '--no-warn-script-location', '-r', '/tmp/build-reqs-s3r1fubv.txt']' returned non-zero exit status 1.
@astrojuanlu astrojuanlu changed the title build uninstalls setuptools in a venv even if needed by a dependency build uninstalls setuptools even if needed by a dependency Feb 12, 2021
@astrojuanlu
Copy link
Author

(Edit: Misleading reference to old issue, or so I think)

@FFY00
Copy link
Member

FFY00 commented Feb 12, 2021

Aghh, seems like pip is not building the dependencies in isolation. The dependencies should not be build in the build environment. They should be built separately and then installed onto the build environment. I think this is probably due to an old pip version.

@gaborbernat do you know when did pip started building in isolation?
We should bump pip to that version.

@uranusjr
Copy link
Member

I believe build isolation was introduced to pip with the PEP 517 implementation.

@gaborbernat
Copy link
Contributor

Which project are you trying to build? Do you have a link?

@astrojuanlu
Copy link
Author

@astrojuanlu
Copy link
Author

astrojuanlu commented Feb 12, 2021

About the old pip version: the logs say WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available., so I tried upgrading pip and still failed with the same error.

We should bump pip to that version.

Sorry, you're referring to pypa/build itself, now I get it

@FFY00
Copy link
Member

FFY00 commented Feb 12, 2021

Looks like that pip should be recent enough 🤔

@layday
Copy link
Member

layday commented Feb 13, 2021

pip does not use isolation for pyproject.toml-less packages. What I'm really curious about is why we're installing astroquery which is a runtime dependency - not a build dependency!

@layday
Copy link
Member

layday commented Feb 13, 2021

-> env.install(builder.get_dependencies(distribution))
(Pdb) builder.build_dependencies
{'flit_core >=2.0,<3', 'oldest-supported-numpy', 'wheel'}
(Pdb) builder.get_dependencies('sdist')
{'jupyter-client ; extra == "dev"', 'astroquery >=0.3.9', 'pycodestyle ; extra == "dev"', 'scipy >=1.4.0', 'sphinx-hoverxref==0.5b1 ; extra == "dev"', 'notebook ; extra == "jupyter"', 'pyerfa', 'jplephem', 'jupytext ; extra == "dev"', 'nbconvert>=5.5 ; extra == "dev"', 'coverage ; extra == "dev"', 'sphinx ; extra == "dev"', 'czml3 ~=0.5.3 ; extra == "cesium"', 'sphinx-notfound-page ; extra == "dev"', 'ipykernel ; extra == "dev"', 'ipywidgets>=7.0 ; extra == "dev"', 'isort ; extra == "dev"', 'pytest-remotedata ; extra == "dev"', 'nbsphinx>=0.5.0 ; extra == "dev"', 'ipython>=5.0 ; extra == "dev"', 'plotly ~=4.0', 'pytest-cov<2.6.0 ; extra == "dev"', 'black==20.8b1 ; extra == "dev"', 'matplotlib >=2.0,!=3.0.1', 'sphinx_rtd_theme==0.5.0 ; extra == "dev"', 'pandas', "cached_property ; python_version<'3.8'", 'astropy >=3.2,<5', 'pytest-doctestplus>=0.8 ; extra == "dev"', 'numba >=0.46', 'myst-parser>=0.13.1 ; extra == "dev"', 'tox ; extra == "dev"', 'pytest-mypy ; extra == "dev"', 'pytest>=3.2 ; extra == "dev"', 'mypy>=0.740 ; extra == "dev"', 'sphinx-autoapi ; extra == "dev"', 'sphinx-gallery ; extra == "dev"', 'numpy', 'pytest-mpl ; extra == "dev"', 'hypothesis ; extra == "dev"'}

I, uh... what?


I've traced that back to flit 2.0 - I suggest requiring flit 3.0 at a minimum if you don't want to have all of your runtime dependencies installed at build time.

As for the build issue, I assume it's only surfaced now because we're no longer inheriting pip from the environment build is installed in. One option would be to enforce build isolation in pip with --use-pep517.

@FFY00
Copy link
Member

FFY00 commented Feb 13, 2021

One option would be to enforce build isolation in pip with --use-pep517.

+1 on this, should be backwards compatible.

FFY00 added a commit to FFY00/python-build that referenced this issue Feb 13, 2021
In pypa#231 we discovered pip is not building some dependencies in
isolation, it is using the build environment, which can become an issue
since we remove setuptools from the environment.

As suggested by @layday, this patch passes --use-pep517 to pip so that
it builds in isolation, as recomended by PEP 517.

Fixes pypa#231

Signed-off-by: Filipe Laíns <[email protected]>
FFY00 added a commit to FFY00/python-build that referenced this issue Feb 13, 2021
In pypa#231 we discovered pip is not building some dependencies in
isolation, it is using the build environment, which can become an issue
since we remove setuptools from the environment.

As suggested by @layday, this patch passes --use-pep517 to pip so that
it builds in isolation, as recomended by PEP 517.

Fixes pypa#231

Signed-off-by: Filipe Laíns <[email protected]>
gaborbernat pushed a commit that referenced this issue Feb 13, 2021
In #231 we discovered pip is not building some dependencies in
isolation, it is using the build environment, which can become an issue
since we remove setuptools from the environment.

As suggested by @layday, this patch passes --use-pep517 to pip so that
it builds in isolation, as recomended by PEP 517.

Fixes #231

Signed-off-by: Filipe Laíns <[email protected]>
@astrojuanlu
Copy link
Author

I've traced that back to flit 2.0

You were totally right @layday , thanks!

@FFY00
Copy link
Member

FFY00 commented Feb 13, 2021

#232 should have fixed this anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants