[BUG]

[aragilar]

setuptools version

61.0.0

Python version

3.9

OS

Debian Unstable

Additional environment information

No response

Description

kiwisolver 1.4.0 (I haven't checked earlier versions, but kiwisolver 1.4.0 did modernise their build, so issues with earlier versions will likely be to different code) can no build from source. I filed nucleic/kiwi#130 with the kiwisolver developers, but it appears that setuptools 60.10.0 does work. Based on looking at v60.10.0...v61.0.0 and the error message that appears (see output), I think this is related to the new(?) package discovery mechanism. I'm not a kiwisolver developer, so I can't answer questions about their build system (other than link you to the repository), but I'm filing this as it appears to be a regression (give setuptools 61 came out 10 hours ago when I wrote this).

Expected behavior

I expect pip install kiwisolver would build a wheel and install it.

How to Reproduce

1. Run pip install kiwisolver (given kiwisolver has wheels, you may need to disable using wheels to not download a pre-built wheel).

Output

      /tmp/pip-build-env-nwrt4o9m/overlay/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py:100: _ExperimentalProjectMetadata: Support for project metadata in `pyproject.toml` is still experimental and may be removed (or change) in future releases.
        warnings.warn(msg, _ExperimentalProjectMetadata)
      Install ``trove-classifiers`` to ensure proper validation. Meanwhile a list of classifiers will be downloaded from PyPI.
      error: Multiple top-level packages discovered in a flat-layout: ['py', 'kiwi'].
      
      To avoid accidental inclusion of unwanted files or directories,
      setuptools will not proceed with this build.
      
      If you are trying to create a single distribution with multiple packages
      on purpose, you should not rely on automatic discovery.
      Instead, consider the following options:
      
      1. set up custom discovery (`find` directive with `include` or `exclude`)
      2. use a `src-layout`
      3. explicitly set `py_modules` or `packages` with a list of names
      
      To find more information, look for "package discovery" on setuptools docs.

[MatthieuDartiailh]

I am a maintainer of kiwisolver. I guess that setuptools is confused because kiwisolver is a pure C extensions without any python source. Currently it does not specify a value for py_modules nor for packages. I would appreciate any advice on how to keep things working with the latest setuptools.

[abravalheri]

Thank you very much for reporting this.

My understanding is that unfortunately there is a clash of 2 behaviours:

To make these changes as backwards compatible as possible, I added a condition that only works for packages not using pyproject.toml that will check if ext_modules is given and then skip auto-discovery.

But, going forward my assumption was that it is very common for packages to mix both regular Python code with extensions. So if a project is using pyproject.toml for metadata, auto-discovery is always on.

I did not foresee that kiwisolver was providing metadata in both pyproject.toml and setup.py already before the latest change. Sorry for that.

@MatthieuDartiailh, one way to disable auto discovery is to explicitly set py_modules=[] in setup.py or py-modules = [] in pyproject.toml [tool.setuptools]. Would that work for you?

If that is a viable alternative, it would help a lot! I was hopping to keep the auto-discovery always on for packages using pyproject.toml...

[MatthieuDartiailh]

It would work, but actually my setup is broken since in its current state it does not distribute type annotations which I recently added. So I will need to rework the overall structure.

I had hoped to get rid of everything but the C extension building in setup.py however this _ExperimentalProjectMetadata warning is a bit frightening. Do you really think you may roll back support for PEP 621 ?

[abravalheri]

Do you really think you may roll back support for PEP 621?

I hope not, but it is very early to say. It was just released yesterday, and if it break too many projects I might have to roll back before re-working on it again.

Given that we are not forced to roll back, usage the [project] table should not change, after all it is defined in a PEP, but the shape of [tool.setuptools] might differ a bit in the future...

---

my setup is broken since in its current state it does not distribute type annotations which I recently added.

Is this also caused by the latest release?

[MatthieuDartiailh]

No I added those in the last release and apparently I did not properly test apparently.

[MatthieuDartiailh]

Do you really think you may roll back support for PEP 621?

I hope not, but it is very early to say. It was just released yesterday, and if it break too many projects I might have to roll back before re-working on it again.

Given that we are not forced to roll back, usage the [project] table should change, after all it is defined in a PEP, but the shape of [tool.setuptools] might differ a bit in the future...

Here you mean "[project] table should not change", right ?

my setup is broken since in its current state it does not distribute type annotations which I recently added.

Is this also caused by the latest release?

[tobni]

We observe a similar error (but with different warnings) with scikit-fmm==2022.02.02 on python 3.10.2:

Collecting scikit-fmm==2022.02.02
  Downloading scikit-fmm-2022.2.2.tar.gz (434 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 434.5/434.5 KB 87.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      /tmp/pip-build-env-qhobckeh/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'configuration'
        warnings.warn(msg)
      /tmp/pip-build-env-qhobckeh/overlay/lib/python3.10/site-packages/setuptools/dist.py:511: UserWarning: Normalizing '2022.02.02' to '2022.2.2'
        warnings.warn(tmpl.format(**locals()))
      error: Multiple top-level packages discovered in a flat-layout: ['skfmm', 'profile'].
      
      To avoid accidental inclusion of unwanted files or directories,
      setuptools will not proceed with this build.
      
      If you are trying to create a single distribution with multiple packages
      on purpose, you should not rely on automatic discovery.
      Instead, consider the following options:
      
      1. set up custom discovery (`find` directive with `include` or `exclude`)
      2. use a `src-layout`
      3. explicitly set `py_modules` or `packages` with a list of names
      
      To find more information, look for "package discovery" on setuptools docs.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

[abravalheri]

Hi @tobni, thank you very much for reporting this.
Am I correct in assume that scikit-fmm does not specify any of py_modules, packages or ext_modules keywords?

[abravalheri]

Here you mean "[project] table should not change", right ?

Yes sorry @MatthieuDartiailh, I typed that too fast without checking 😅

[tobni]

@abravalheri Not in the pyproject.toml, no. I don't know exactly where to look in the setup.py file, the scikit-fmm package uses numpy.distutils.misc_util.Configuration which is hard to grok.

[abravalheri]

My question was more in the direction of setup.py. I will have a look on the numpy.distutils.misc_util.Configuration. Thank you very much @tobni.

[MatthieuDartiailh]

Experiment a bit I found another undesirable behavior.

Kiwisolver uses setuptools-scm and after modifying the setup.py to include the following directives (which silences the previously reported error for setuptools 61) the installed package in site-packages includes all the C source files and the tests even though they are not requested. With Setuptools <= 60, the install contains just an __init__.py (added to allow accessing the C extension content), a .pyi file, py.typed and the built extension package.

    package_dir={"kiwisolver": "py"},
    packages=["kiwisolver"],
    install_requires=["typing_extensions;python_version<'3.8'"],
    ext_modules=ext_modules,
    cmdclass={"build_ext": CppyBuildExt},
    package_data={"kiwisolver": ["py.typed", "*.pyi"]},

[abravalheri]

Thanks @MatthieuDartiailh, I will have a look on that too.

This is likely related to the fact that the presence of [project] in pyproject.toml will automatically flip the default value of include_package_data (in setup.cfg/setup.py-only projects it defaults to False, while with pyproject.toml [project] it defaults to True.

This should have been added to the CHANGELOG. Sorry I forgot that.

[MatthieuDartiailh]

I tried setting include_package_data to False but it does not help. The setup I use for testing can be found at nucleic/kiwi#131

[MatthieuDartiailh]

It appears that through some re-organization of kiwi I managed to circumvent the issue (see the PR). But it may not apply to other packages out there.

[TimoRoth]

Not sure if this is the same issue, but building scipy with --no-build-isolation fails as well since the setuptools 61.0 update.
With build isolation their pin to setuptools<60 in pyproject.toml saves the day.

[MatthieuDartiailh]

Note that even though it does not break the build the issue of unwanted files being included can also be seen with https://github.com/nucleic/atom

[abravalheri]

Hi @TimoRoth and @MatthieuDartiailh, thank you for reporting that. I will add those to my list today.

@MatthieuDartiailh, can you give some examples of those unwanted files so I can have a look?

Is this related to the fact that the default value of include_package_data is flipped to True for when pyproject.toml has a [project] table?

[MatthieuDartiailh]

Basically all C source files found under atom/src end being installed. What I find confusing with the include_package_data explanation is that even if I set it to False manually in setup.py I still see the same weird behavior. Do you expect that ?

[abravalheri]

if I set it to False manually in setup.py I still see the same weird behavior. Do you expect that ?

I will work on that. The current code expects include-package-data to be set to false in pyproject.toml [tool.setuptools], but I completely agree it should work also when it is set in setup.py.

[MatthieuDartiailh]

I can confirm that settings include-package-data in pyproject.toml does indeed work.

[sanderr]

I'm encountering this issue on packages that act solely as a meta package for distributing a collection of related packages via dependencies. The package does not have any source packages or modules. I can confirm that explicitly setting packages=[] resolves the issue.

I do still find it strange that what you call an experimental feature in the docs is enabled by default.

[sanderr]

To be fair, I see this behavior is mentioned in your changelog under "breaking changes". That's my bad for not checking there in the first place.

[abravalheri]

Hi @sanderr, I am terribly sorry for the trouble and thank you very much for the understanding.

[bersbersbers]

one way to disable auto discovery is to explicitly set py_modules=[] in setup.py or py-modules = [] in pyproject.toml [tool.setuptools].

I may have misunderstood that workaround - @abravalheri, do you mean to literally write an empty list for py-modules? I tried it out, and while packages using that workaround are successfully installed, I cannot seem to import their modules (unless my working directory is the project directory, in which case installation is not required anyway).

Or was your recommended workaround to populate py-modules with the actual list of module names?

[abravalheri]

@bersbersbers the context for that is a distribution that does not contain any Python packages or modules (so there is nothing really to be imported).

If your project contains those, probably the best advice is to follow one of the suggestions in the error message.