"AttributeError: cython_sources" with Cython 3.0.0a10

[The-Compiler]

When attempting to install PyYAML from sources with Cython 3.0.0a10, e.g. like so:

pip install --pre -v git+https://github.com/yaml/pyyaml.git

the install fails with:

Using pip 21.3.1 from /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting git+https://github.com/yaml/pyyaml.git
  Cloning https://github.com/yaml/pyyaml.git to /tmp/pip-req-build-rdv2sw86
  Running command git version
  git version 2.34.1
  Running command git clone --filter=blob:none -q https://github.com/yaml/pyyaml.git /tmp/pip-req-build-rdv2sw86
  Running command git rev-parse HEAD
  8cdff2c80573b8be8e8ad28929264a913a63aa33
  Resolved https://github.com/yaml/pyyaml.git to commit 8cdff2c80573b8be8e8ad28929264a913a63aa33
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /tmp/pip-standalone-pip-7sp3xayz/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-az2k859x/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --pre -- setuptools wheel Cython
  Collecting setuptools
    Using cached setuptools-60.5.0-py3-none-any.whl (958 kB)
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Collecting Cython
    Using cached Cython-3.0.0a10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.2 MB)
  Installing collected packages: wheel, setuptools, Cython
  Successfully installed Cython-3.0.0a10 setuptools-60.5.0 wheel-0.37.1
  Installing build dependencies ... done
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l
  running egg_info
  creating lib/PyYAML.egg-info
  writing manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 288, in <module>
      setup(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 155, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 299, in run
      self.find_sources()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
      mm.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 541, in run
      self.add_defaults()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 312, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "setup.py", line 204, in get_source_files
      self.cython_sources(ext.sources, ext)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  Getting requirements to build wheel ... error
WARNING: Discarding git+https://github.com/yaml/pyyaml.git. Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.

I'm not sure if this is an intended change in Cython 3, or a bug - but given that "pyyaml setup is still relying on a bunch of ancient custom distutils bits" (#584) I thought I'd report it here first. Please let me know if you think I should rather report it there.

[nitzmahone]

It's definitely a regression between Cython 3.0.0a9 and a10; every previous 3.0.0 alpha has been able to build the pyyaml extension fine. This PR (included in a10) swapped out the base class on Cython's custom build_ext distutils command (which pyyaml further subclasses to customize some stuff). I haven't dug down all the way yet to see if it's something we should try to address in pyyaml's customization or if it's just a bug in the new base class.

Thanks for the report- regardless, we'll want to get this sorted one way or another, so I'll figure out where the problem is and work with the Cython/setuptools folks if necessary to get it working.

[tomplex]

I encountered this error when attempting to lock my dependencies with pipenv; I was able to fix it in my case by setting

[pipenv]
allow_prereleases = false

just figured I'd share this in case it helps someone else.

[nitzmahone]

allow_prereleases = false

That'll only work until Cython 3.0 releases, unless we're able to get a change made over there (which it sounds like they're amenable to, if it's not too gnarly)

[RobertoPrevato]

I don't know if this can be of interest for anyone reading here, but I had the same issue in a GitHub Workflow, only happening on Python 3.10. The failed build can be seen here: https://github.com/Neoteroi/BlackSheep/runs/5098546591?check_suite_focus=true

In my case, PyYAML is a dependency and I was testing another wheel with this command:

pip install --pre blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

In my case, removing the --pre option fixed the problem.

pip install blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

[nitzmahone]

Yep, unfortunately --pre seems to affect all deps and build deps as well under pip.

[mgorny]

Ping. Cython 3.0.0 has already reached beta releases, so we need some solution sooner than later.

[nitzmahone]

Cool- we've been poking at doing another release, and since we sat on it long enough, setuptools and pip now fully support PEP517/518 config passthru from the CLI, which should make the replacement of the dynamic bits a whole lot easier.

[jmakov]

Stumbled upon this bug while trying to install Pycaret:

Collecting PyYAML~=5.1                                                                                                                   
#0 30.24   Downloading PyYAML-5.4.1.tar.gz (175 kB)                                                                                               #0 30.24      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 34.3 MB/s eta 0:00:00
#0 30.32   Installing build dependencies: started                                                                                                 
#0 33.08   Installing build dependencies: finished with status 'done'                                                                             
#0 33.08   Getting requirements to build wheel: started                  
#0 33.19   Getting requirements to build wheel: finished with status 'error'                                                                      #0 33.20   error: subprocess-exited-with-error                                                                                                    
#0 33.20                                                                                                                                          
#0 33.20   × Getting requirements to build wheel did not run successfully.
#0 33.20   │ exit code: 1                                                                                                                         
#0 33.20   ╰─> [50 lines of output]                                      
#0 33.20       /tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:516: SetuptoolsDeprecationWarning: T
he license_file parameter is deprecated, use license_files instead.
#0 33.20         warnings.warn(msg, warning_class)                                                                                                
#0 33.20       running egg_info                                          
#0 33.20       writing lib3/PyYAML.egg-info/PKG-INFO                                                                                              
#0 33.20       writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt                                                              
#0 33.20       writing top-level names to lib3/PyYAML.egg-info/top_level.txt                                                                      
#0 33.20       Traceback (most recent call last):
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <
module>
#0 33.20           main()
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in m
ain
#0 33.20           json_out['return_val'] = hook(**hook_input['kwargs'])
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in g
et_requires_for_build_wheel
#0 33.20           return hook(config_settings)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_b
uild_wheel
#0 33.20           return self._get_build_requires(config_settings, requirements=['wheel'])
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_require
s
#0 33.20           self.run_setup()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
#0 33.20           exec(code, locals())
#0 33.20         File "<string>", line 271, in <module>
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
#0 33.20           return distutils.core.setup(**attrs)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
#0 33.20           return run_commands(dist)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#0 33.20           dist.run_commands()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#0 33.20           self.run_command(cmd)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
#0 33.20           super().run_command(command)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#0 33.20           cmd_obj.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run
#0 33.20           self.find_sources()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
#0 33.20           mm.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 570, in run
#0 33.20           self.add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 608, in add_defaults
#0 33.20           sdist.add_defaults(self)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 106, in add_defaults
#0 33.20           super().add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
#0 33.20           self._add_defaults_ext()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
#0 33.20           self.filelist.extend(build_ext.get_source_files())
#0 33.20         File "<string>", line 201, in get_source_files
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
#0 33.20           raise AttributeError(attr)
#0 33.20       AttributeError: cython_sources
#0 33.20       [end of output]

[nitzmahone]

@jmakov the likely short-term fix will be to pin the Cython PEP518 build deps to pre-3.0 Cython in a 6.x release (see #702), but we generally consider the 5.x release branch dead, so hadn't planned on re-publishing the build change back there. You might want to try and get the Pycaret folks (or whoever's ultimately pinning PyYAML to ~=5.1) to update their deps.

[jmakov]

@nitzmahone thank you so much for pointing out the solution! Will link your answer to an issue in their repo.

[benorgera]

Any update on this?

[RubTalha]

https://stackoverflow.com/questions/77490435/attributeerror-cython-sources

[mathomp4]

Query: is this issue now moot with 6.0.2? The 6.0.2 release notes say:

Support for Cython 3.x and Python 3.13.

and in my testing to update the spack package for pyyaml, I did test with cython3 and it seems to work, see spack/spack#45716

[henryiii]

This issue has been moot for a year or two. It's only affecting people stuck on old versions. (<5, IIRC) (Though, I don't remember if it used to be fixed by just limiting Cython - that's gone in version 6 if it used to be true before)