[Question] Editable install fails after introducing pyproject.toml

[jcwchen]

setuptools version

65.3.0

Python version

3.9

OS

Windows

Additional environment information

No response

Description

Thank you for maintaining setuptools! Recently we saw pip install -e . started to fail after adding pyproject.toml in the package. Not sure whether it is a bug or we need to set something else properly in pyproject.toml. Provide some findings here:

• pip install -e . will fail due to missing .pyd file (FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\jacky\AppData\Local\Temp\tmpba9gls20.build-lib\onnx\onnx_cpp2py_export.cp39-win_amd64.pyd')
• set SETUPTOOLS_ENABLE_FEATURES="legacy-editable" and then pip install -e . works fine
• pip install -e . without added pyproject.toml works fine

I tried to find any related existing issues, but I didn't find a solution except adding SETUPTOOLS_ENABLE_FEATURES="legacy-editable". I am not sure whether this issue is related: #3557. Thus, I would like to confirm if it is a known issue that will be fixed from setuptools side, or something we should fix in our package. Thank you for looking into it.

Related issue: onnx/onnx#4539.

Expected behavior

pip install -e . should build without errors.

How to Reproduce

Build onnx from source with editable mode:

conda install -y -c conda-forge libprotobuf=3.16.0  # protoc (Protobuf compiler) is required to build onnx from source
git clone https://github.com/onnx/onnx.git
cd onnx
git submodule update --init --recursive
set CMAKE_ARGS=-DONNX_USE_PROTOBUF_SHARED_LIBS=ON  # use export if POSIX
pip install -e .

Output

      -- ******** Summary ********                                                                                                                                                                     
      --   CMake version             : 3.19.0-rc3                                                                                                                                                      
      --   CMake command             : C:/Program Files/CMake/bin/cmake.exe                                                                                                                            
      --   System                    : Windows                                                                                                                                                         
      --   C++ compiler              : D:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe                                                  
      --   C++ compiler version      : 19.29.30141.0                                                                                                                                                   
      --   CXX flags                 : /DWIN32 /D_WINDOWS /W3 /GR /EHsc /std:c++17 /EHsc /wd26812                                                                                                      
      --   Build type                : Debug                                                                                                                                                           
      --   Compile definitions       : __STDC_FORMAT_MACROS                                                                                                                                            
      --   CMAKE_PREFIX_PATH         :                                                                                                                                                                 
      --   CMAKE_INSTALL_PREFIX      : C:/Program Files/onnx                                                                                                                                           
      --   CMAKE_MODULE_PATH         :                                                                                                                                                                 
      --                                                                                                                                                                                               
      --   ONNX version              : 1.12.0                                                                                                                                                          
      --   ONNX NAMESPACE            : onnx                                                                                                                                                            
      --   ONNX_USE_LITE_PROTO       : OFF                                                                                                                                                             
      --   USE_PROTOBUF_SHARED_LIBS  : OFF                                                                                                                                                             
      --   Protobuf_USE_STATIC_LIBS  : ON                                                                                                                                                              
      --   ONNX_DISABLE_EXCEPTIONS   : OFF                                                                                                                                                             
      --   ONNX_WERROR               : OFF                                                                                                                                                             
      --   ONNX_BUILD_TESTS          : OFF                                                                                                                                                             
      --   ONNX_BUILD_BENCHMARKS     : OFF                                                                                                                                                             
      --                                                                                                                                                                                               
      --   Protobuf compiler         : D:/protobuf/install/bin/protoc.exe                                                                                                                              
      --   Protobuf includes         : D:/protobuf/install/include                                                                                                                                     
      --   Protobuf libraries        : D:/protobuf/install/lib/libprotobufd.lib                                                                                                                        
      --   BUILD_ONNX_PYTHON         : ON                                                                                                                                                              
      --     Python version        : 3.9                                                                                                                                                               
      --     Python executable     : C:/Users/jacky/AppData/Local/Programs/Python/Python39/python.exe                                                                                                  
      --     Python includes       : C:/Users/jacky/AppData/Local/Programs/Python/Python39/include                                                                                                     
      -- Configuring done                                                                                                                                                                              
      -- Generating done                                                                                                                                                                               
      -- Build files have been written to: D:/github/onnx/.setuptools-cmake-build                                                                                                                      
      Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework                                                                                                                          
      Copyright (C) Microsoft Corporation. All rights reserved.                                                                                                                                        
                                                                                                                                                                                                       
        onnx_proto.vcxproj -> D:\github\onnx\.setuptools-cmake-build\Debug\onnx_proto.lib                                                                                                              
        onnx.vcxproj -> D:\github\onnx\.setuptools-cmake-build\Debug\onnx.lib                                                                                                                          
        onnx_cpp2py_export.vcxproj -> D:\github\onnx\.setuptools-cmake-build\Debug\onnx_cpp2py_export.cp39-win_amd64.pyd                                                                               
      running build_ext                                                                                                                                                                                
      copying D:\github\onnx\.setuptools-cmake-build\Debug\onnx_cpp2py_export.cp39-win_amd64.pyd -> C:\Users\jacky\AppData\Local\Temp\tmpba9gls20.build-lib\onnx                                       
      Traceback (most recent call last):                                                                                                                                                               
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\_distutils\file_util.py", line 40, in _copy_file_contents                                  
          fdst = open(dst, 'wb')                                                                                                                                                                       
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\jacky\\AppData\\Local\\Temp\\tmpba9gls20.build-lib\\onnx\\onnx_cpp2py_export.cp39-win_amd64.pyd'                             
                                                                                                                                                                                                       
      During handling of the above exception, another exception occurred:                                                                                                                              
                                                                                                                                                                                                       
      Traceback (most recent call last):                                                                                                                                                               
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 140, in run                                               
          self._create_wheel_file(bdist_wheel)                                                                                                                                                         
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 330, in _create_wheel_file                                
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)                                                                                                                     
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 261, in _run_build_commands                               
          self._run_build_subcommands()                                                                                                                                                                
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\command\editable_wheel.py", line 288, in _run_build_subcommands                            
          self.run_command(name)                                                                                                                                                                       
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 319, in run_command                                               
          self.distribution.run_command(command)                                                                                                                                                       
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\dist.py", line 1217, in run_command                                                        
          super().run_command(command)                                                                                                                                                                 
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 992, in run_command                                              
          cmd_obj.run()                                                                                                                                                                                
        File "<string>", line 255, in run                                                                                                                                                              
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\command\build_ext.py", line 84, in run                                                     
          _build_ext.run(self)                                                                                                                                                                         
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 346, in run                                         
          self.build_extensions()                                                                                                                                                                      
        File "<string>", line 272, in build_extensions                                                                                                                                                 
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 351, in copy_file                                                 
          return file_util.copy_file(                                                                                                                                                                  
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\_distutils\file_util.py", line 163, in copy_file                                           
          _copy_file_contents(src, dst)                                                                                                                                                                
        File "C:\Users\jacky\AppData\Local\Temp\pip-build-env-onoe7scg\overlay\Lib\site-packages\setuptools\_distutils\file_util.py", line 42, in _copy_file_contents                                  
          raise DistutilsFileError(                                                                                                                                                                    
      distutils.errors.DistutilsFileError: could not create 'C:\Users\jacky\AppData\Local\Temp\tmpba9gls20.build-lib\onnx\onnx_cpp2py_export.cp39-win_amd64.pyd': No such file or directory            
      error: Support for editable installs via PEP 660 was recently introduced                                                                                                                         
      in `setuptools`. If you are seeing this error, please report to:                                                                                                                                 
                                                                                                                                                                                                       
      https://github.com/pypa/setuptools/issues                                                                                                                                                        
                                                                                                                                                                                                       
      Meanwhile you can try the legacy behavior by setting an                                                                                                                                          
      environment variable and trying to install again:                                                                                                                                                
                                                                                                                                                                                                       
      SETUPTOOLS_ENABLE_FEATURES="legacy-editable"                                                                                                                                                     
      [end of output]                                                                                                                                                                                  
                                                                                                                                                                                                       
  note: This error originates from a subprocess, and is likely not a problem with pip.                                                                                                                 
  ERROR: Failed building editable for onnx                                                                                                                                                             
Failed to build onnx                                                                                                                                                                                   
ERROR: Could not build wheels for onnx, which is required to install pyproject.toml-based projects                                                                                                     

[abravalheri]

Hi @jcwchen thank you very much for reporting this.

I believe the following is happening:

• By default, setuptools don't run build_py during an editable install. The assumption is that build_py copies Python files to the directory that is bundled into a wheel, but that is not necessary in an editable installation.
  • onnx seems to acknowledge this behaviour, but since it does other stuff with build_py, it manually changes the develop command to force build_py to run.
• PEP 660 uses a different implementation which is not related to develop, so this command cannot be used to overwrite the build_py behaviour.
• The new PEP660-compatible command in setuptools (editable_wheel), uses a different mechanism to interact with the build subcommands (e.g. build_py or build_ext).
  • Previously, develop would process build_py and build_ext in an ad-hoc manner, which means that develop could not be integrated with custom build sub-commands.
  • editable_wheel implementation levels the playing field. Each subcommand receives information about whether setuptools is running an editable installation or not and then it has the chance to decide to run (or simply perform an earlier return). More information can be found in the docs
  • By default build_py performs an earlier return.
• When build_ext has the chance to run, it tries to create a file in a directory that does not exist (onnx seems to rely on the fact build_py runs first and creates directories, but the customization on develop is not compatible with the PEP 660 implementation). This is very likely the origin of the error.

I did a quick test here with the following modification in setup.py seems to "work" (at least on Linux, sorry it is not very easy for me to replicate things on Windows):

diff --git i/setup.py w/setup.py
index 1ce70a4d..c23cdbb0 100644
--- i/setup.py
+++ w/setup.py
@@ -240,6 +240,7 @@ class build_py(setuptools.command.build_py.build_py):
             dst = os.path.join(TOP_DIR, os.path.relpath(src, CMAKE_BUILD_DIR))
             self.copy_file(src, dst)

+        self.editable_mode = False
         return setuptools.command.build_py.build_py.run(self)

This is a quick-and-dirty way of convincing build_py to run normally instead of doing an early return in the case of editable installs.

Please note however that this may copy files to temporary folders that are simply ignored (the files added to build_lib should make no difference in the case of an editable install: files are imported directly from the project directory).

Thus, I would like to confirm if it is a known issue that will be fixed from setuptools side, or something we should fix in our package.

I don't think backwards compatibility between the editable_wheel and develop commands are possible: the best we can do right now for the ecosystem is to keep these 2 commands separated. PEP 660 imposes a new way of doing things to setuptools, unfortunately some of that burden will fan out in the case of projects that rely on heavy setup.py customisations.

I believe the best step moving forward would be for onnx to review the command customisations having in mind the information shared in:

• https://setuptools.pypa.io/en/latest/userguide/development_mode.html and
• https://setuptools.pypa.io/en/latest/userguide/extension.html#supporting-sdists-and-editable-installs-in-build-sub-commands.

---

Please let me know if you are able to trace this error to a different part of setuptools and create a different minimal reproducible example that does not touch the points I previously mentioned... If my assumptions are wrong here, we might want to solve this problem in a different way.

[jcwchen]

Thank you @abravalheri so much for the quick response and detailed explanation. Much appreciated. I can confirm that the "quick-and-dirty way" (self.editable_mode = False) does help for now. pip install -e . can success with that.

I don't think backwards compatibility between the editable_wheel and develop commands are possible: the best we can do right now for the ecosystem is to keep these 2 commands separated.

Perhaps a naive question: what's difference between editable_wheel and develop command? Is there any drawback if a project only works with develop command?

Please let me know if you are able to trace this error to a different part of setuptools and create a different minimal reproducible example that does not touch the points I previously mentioned.

I am happy to dig deeper into this issue, but how can I help? Are there concrete action items?

I have a question: why having pyproject.toml matters here? (Adding pyproject.toml causes this error) Thank you for answering.

[abravalheri]

I am happy to dig deeper into this issue, but how can I help? Are there concrete action items?

The motivation behind my initial words is that I am happy to review everything I have said if anyone finds out a different reason/root cause for the problem. But to be sincere, I don't believe there is anything else...

I have a question: why having pyproject.toml matters here? (Adding pyproject.toml causes this error)

This behaviour is motivated by PEP 517/PEP 660. Although these PEP are not 100% prescriptive, I believe that pip's interpretation of the standard is very good.

The TL;DR is: after PEP 517/PEP 660 all projects that want to be built/installed in editable mode SHOULD have a pyproject.toml and the installer should use a set of build APIs that is different from the classic python setup.py install/build_wheel/develop.

What pip is doing is try to be as much as backward compatible by assuming that if users don't have a pyproject.toml the "legacy" behaviour is preferred.