Support setup.py-less setups

[madig]

What's the problem this feature will solve?

Due to PEP 518, projects can now have just a setup.cfg and pyproject.toml file. Pip-compile only supports setup.py when using setuptools, though.

Describe the solution you'd like

Pip-compile can use setup.cfg files to get at dependencies, like pip-compile -U setup.cfg.

Alternative Solutions

I suppose I can copy-paste the dependencies into their own requirements.in? That would be duplication though :(

[atugushev]

Hello @madig,

Thanks for the issue! I think it makes sense to implement pip-compile setup.cfg and pip-compile pyproject.toml, especially after the #908 issue.

[astrojuanlu]

Notice that there are two big families of setup.py-less setups:

• setuptools: build requirements in pyproject.toml, metadata in setup.cfg
• flit: all metadata in pyproject.toml

[merwok]

In theory, there is an unbounded number of possible build backends (the PEP term for tools that take source code and produce wheels and sdists), each of which can find build information (name, source files, dependencies, etc) in a tool-specific manner (setup.py using setuptools, setup.py compat shim from another tool, section in pyproject.toml, shell script, ini file…). Some of the recent tools already have their idea of a lock file, but pip-tools is still very useful for some others (for example, I use flit to build a wheel for an app, but pip-tools to manage its dependencies, which means the wheel is not sufficient for installing).

Tools that only need to install things (e.g. tox) can use a build frontend such as pip or pep517 to abstract away the different build backends; tools that inspect wheel files have a defined structure to work with; dev tools like pip-tools are out of luck for now, they have to special-case every build backend they want to support, so you’d need custom code to handle setuptools and flit.

Now two good things:

• despite using different config files, both setuptools and flit follow the same spec for the requirements itself (PEP 508), so the same code can parse versions, extras, environment markers, URLs
• people are talking about standardizing a pyproject.toml section for dependencies: https://discuss.python.org/t/standardized-way-for-receiving-dependencies/3821

So you could start with custom code to help people now, add support for an official spec later, and eventually remove the custom code. (And some day there may be a standard lock file format too!)

[ajfriend]

Stumbled over this as I was looking into this issue myself for https://github.com/uber/h3-py. Adding my vote for pip-compile pyproject.toml support or something like #908.

[astrojuanlu]

people are talking about standardizing a pyproject.toml section for dependencies: https://discuss.python.org/t/standardized-way-for-receiving-dependencies/3821

Update: PEP 631 has been accepted and is expected to be merged into PEP 621.

[ssbarnea]

The lack of this feature does penalize any projects that already followed existing PEPs recommendations regarding storing project metadata.

Does anyone know an alternative tool that is able to update dependencies inside setup.cfg files?

[astrojuanlu]

Somebody needs to step up and refactor pip-compile so it uses pep517.meta.load from https://pypi.org/project/pep517/ as suggested in #908 (comment) (see example in pypa/setuptools#1951 (comment)). It looks like this already works, regardless of the status of PEP 631 (accepted) and PEP 621 (provisional until March 2021 or until a working implementation appears).

[astrojuanlu]

I tried a very quick proof of concept using pep517.meta.load:

diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
index d21f0de..9dbf621 100755
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -373,7 +373,7 @@ def cli(
     constraints = []
     for src_file in src_files:
         is_setup_file = os.path.basename(src_file) == "setup.py"
-        if is_setup_file or src_file == "-":
+        if src_file == "-":
             # pip requires filenames and not files. Since we want to support
             # piping from stdin, we need to briefly save the input from stdin
             # to a temporary file and have pip read that.  also used for
@@ -400,6 +400,10 @@ def cli(
             for req in reqs:
                 req.comes_from = comes_from
             constraints.extend(reqs)
+        elif is_setup_file:
+            from pep517 import meta
+            from pip._internal.req.constructors import install_req_from_req_string
+            constraints.extend([install_req_from_req_string(req) for req in meta.load(".").requires])
         else:
             constraints.extend(
                 parse_requirements(

However, pip-compile on its own repo fails with this error:

$ pip-compile
ERROR: WARNING: You are using pip version 20.2.3; however, version 21.0 is available.
ERROR: You should consider upgrading via the '/home/juanlu/.pyenv/versions/piptools38/bin/python -m pip install --upgrade pip' command.
ERROR: WARNING: You are using pip version 20.2.3; however, version 21.0 is available.
ERROR: You should consider upgrading via the '/home/juanlu/.pyenv/versions/piptools38/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/home/juanlu/.pyenv/versions/piptools38/bin/pip-compile", line 33, in <module>
    sys.exit(load_entry_point('pip-tools', 'console_scripts', 'pip-compile')())
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/juanlu/Personal/pip-tools/piptools/scripts/compile.py", line 427, in cli
    constraints = [
  File "/home/juanlu/Personal/pip-tools/piptools/scripts/compile.py", line 428, in <listcomp>
    req for req in constraints if req.markers is None or req.markers.evaluate()
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 328, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 244, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/juanlu/.pyenv/versions/piptools38/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 224, in _get_env
    raise UndefinedEnvironmentName(
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

@jaraco perhaps I'm missing something?

[jaraco]

You can see what meta.load().requires returns thus:

pip-tools master $ pip-run -q pep517 -- -c "import pep517.meta as meta; print(meta.load('.').requires)"
['click (>=7)', 'pip (>=20.1)', "pytest-cov ; extra == 'coverage'", "pytest ; extra == 'testing'", "pytest-rerunfailures ; extra == 'testing'"]

Presumably, the 'constraints' variable doesn't recognize extras when indicated as environment markers. You probably need to indicate any 'extras' (or None) when evaluating each requirement (i.e. .evaluate(dict(extra=None))).

[astrojuanlu]

Thanks @jaraco , that kind of worked, in that it produces a proper requirements.txt for pip-tools 🎉 However, I am stuck because test_command_line_setuptools_read and test_command_line_setuptools_output_file fail with status code 1. By inspecting the stderr, I see ERROR: No matching distribution found for setuptools. It looks like the functionality is within reach but I am too obtuse to understand how to debug the tests.

For completeness, the patch is here:

diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
index d21f0de..25cf6cf 100755
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -8,8 +8,12 @@ from typing import Any
 import click
 from click import Command
 from click.utils import safecall
+from pep517 import meta
 from pip._internal.commands import create_command
-from pip._internal.req.constructors import install_req_from_line
+from pip._internal.req.constructors import (
+    install_req_from_line,
+    install_req_from_req_string,
+)
 from pip._internal.utils.misc import redact_auth_from_url
 
 from .._compat import parse_requirements
@@ -373,7 +377,7 @@ def cli(
     constraints = []
     for src_file in src_files:
         is_setup_file = os.path.basename(src_file) == "setup.py"
-        if is_setup_file or src_file == "-":
+        if src_file == "-":
             # pip requires filenames and not files. Since we want to support
             # piping from stdin, we need to briefly save the input from stdin
             # to a temporary file and have pip read that.  also used for
@@ -400,6 +404,13 @@ def cli(
             for req in reqs:
                 req.comes_from = comes_from
             constraints.extend(reqs)
+        elif is_setup_file:
+            constraints.extend(
+                [
+                    install_req_from_req_string(req)
+                    for req in meta.load(".").requires or []
+                ]
+            )
         else:
             constraints.extend(
                 parse_requirements(
@@ -421,7 +432,9 @@ def cli(
 
     # Filter out pip environment markers which do not match (PEP496)
     constraints = [
-        req for req in constraints if req.markers is None or req.markers.evaluate()
+        req
+        for req in constraints
+        if req.markers is None or req.markers.evaluate(dict(extra=None))
     ]
 
     log.debug("Using indexes:")
diff --git a/setup.cfg b/setup.cfg
index 5f30716..b5f2112 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -30,6 +30,7 @@ packages = find:
 zip_safe = false
 install_requires =
     click >= 7
+    pep517
     pip >= 20.1
 
 [options.packages.find]

[atugushev]

@astrojuanlu

By inspecting the stderr, I see ERROR: No matching distribution found for setuptools. It looks like the functionality is within reach but I am too obtuse to understand how to debug the tests.

That's because pip_conf fixture has no-index = true option and pip517 seems tries to install setuptools and wheel packages at the build time, which requires index-url. Try to create different config:

diff --git tests/test_cli_compile.py tests/test_cli_compile.py
index 12db0f4..7bad5f0 100644
--- tests/test_cli_compile.py
+++ tests/test_cli_compile.py
@@ -39,7 +39,16 @@ def test_command_line_overrides_pip_conf(pip_with_index_conf, runner):
     assert "Using indexes:\n  http://override.com" in out.stderr


-def test_command_line_setuptools_read(pip_conf, runner):
+def test_command_line_setuptools_read(runner, make_pip_conf):
+    make_pip_conf(
+        dedent(
+            f"""\
+            [global]
+            find-links = {MINIMAL_WHEELS_PATH}
+            """
+        )
+    )
+
     with open("setup.py", "w") as package:
         package.write(
             dedent(

[LouisStAmour]

So ... what's next for PEP 621 support? I'd really like to eliminate as many ini and cfg files as I can from my repo's root and switching to a single pyproject.toml file that's flexible enough to support multiple sets/groups of dependencies sounds fantastic. PDM is great for what it does, but it's not as compatible or proven yet. Even a simple mode that can translate pyproject.toml as if it were requirements.in would be a nice touch. For more advanced functionality, maybe an argument that can be repeated to specify groups of dependencies to install?

If we really wanted to get fancy, under the assumption that __pypackages__ folders (PEP 582, draft, popularized by PDM) might allow for different versions of dependencies to be resolved, perhaps we could have different modes that could be used for groups of dependencies, such as "peer dependencies" from Node.js: https://classic.yarnpkg.com/en/docs/dependency-types/ But then again, perhaps there will be better tools to do this sort of thing in future using pep517 or something, when it's necessary.

[astrojuanlu]

flit is about to add support for PEP 621, and I suppose Poetry and setuptools will do the same. Not sure pip-tools can do anything related to it, since the front-end (pip) doesn't need to change anything, AFAIU.

[LouisStAmour]

Perhaps

pip-tools/piptools/scripts/compile.py

Line 233 in 77aa19b

if os.path.exists(DEFAULT_REQUIREMENTS_FILE):

could be updated to look for requirements in pyproject.toml, with command line options for compiling from one or more groups?

I’d be happy to work on a proof-of-concept PR, but I haven’t the experience with the piptools project to understand what the most piptools-like way of writing and organizing the code would be…

My use case is that until other tools have better built-in support, I’d like to replace requirements.in with that of pyproject.toml in such a way that I could output a requirements file for different combinations of groups of requirements.

For example: piptools compile --generate-hashes --with-optional-requirements=dev --with-optional-requirements=test --output=requirements-dev.txt with specific groups of dependencies plus required dependencies as if appended to a requirements.in file or a simple piptools compile and it would output a requirements.txt file as if only the required dependencies were specified in a requirements.in file?

My understanding, having played around with piptools just now, is that it won’t compile without either setup.cfg or requirements.in and if I specify piptools compile --generate-hashes pyproject.toml - it doesn’t know how to handle a file of type “toml”, assumes it is a requirements.in file of a weird name, and exits with an error.

My goal is to use piptools to create and update requirements.txt files as “lock files” that contain groups of dependencies and are checked in to version control for reproducibility and updated in future as new dependencies are added or new versions released. I’ve heard good things about using piptools for this use case, and would let me switch faster to using only pyproject.toml even if the rest of the ecosystem (such as pip or Bazel’s rules_python) only understands requirements.txt for now.

The only other option I can think of would be to update a wrapper, such as https://github.com/peterdemin/pip-compile-multi to produce requirements.in files dynamically given a pyproject.toml. Such a tool might actually be useful in the short term because it could convert new toml syntax into older ini file syntax for projects that require older or dedicated config files.

However, it would ultimately be a hack and while it would initially speed up adoption, it would long-term harm community use of pyproject.toml, particularly if third-party tools dislike any of the conventions around the use of toml with their app name. I suppose the alternative would be to prefix all tooling sections in the toml file with the name of the tool that would write or update backwards-compatible generated config files. Even so, my preference would be to use wrappers for each tool, such as FlakeHell for Flake8 to update them to understand pyproject.toml, under the idea that perhaps each tool would eventually receive a PR from the community to match the upstream wrapper’s functionality…

I recognize this is a relatively trivial problem, adopting pyproject.toml, but the more tools that support it natively, the faster it can be adopted to simplify a project’s config files…

[orsinium]

I'm not sure why the last step isn't made yet but after #1311 adding support for PEP-517 is trivial one-line change;

# replace this
is_setup_file = os.path.basename(src_file) == "setup.py"

# by this
is_setup_file = os.path.basename(src_file) in {"setup.py", "pyproject.toml"}

With this change, if you explicitly specify pyproject.toml as the input file, everything works pretty well. Probably, what remains is a few tests and --extras support. It sounds easy, though.

UPD: I'll see what I can do.

[astrojuanlu]

Indeed @orsinium , it's just that. I didn't want to add it in #1311 to go step by step (in fact, at least one minor bug has appeared since we introduced it, #1352). It's probably ready to go, but it would need some tests I think.

[ssbarnea]

Any chance to tag at least a pre-release so we can start consuming the newer feature? Last tag was 6.0.1 and did not include that highly desired feature.

PS. I tested with code from master and looks to work nice but I am going going to install pip-tools from master in my CI pipelines ;)

[orsinium]

You can follow #1362, we want the release to include also #1363