Support extras that reference other extras

[carver]

I want to do something like this (simplified for the example):

setup(
    name='my-project',
    extras_require={
        'tester': [
            "pytest==3.3.2",
        ],
        'dev': [
            "pytest-xdist",
            ".[tester]",
        ],
    },
    ...
)

When I run pip install -e .[dev], I want it to install pytest and xdist. Right now, it fails saying that I supplied an invalid package name.

Also of interest is that the following incantation spits no error, but seems to ignore the tester extra:

        'dev': [
            "pytest-xdist",
            "my-project[tester]",
        ],

I'm open to attempting at a PR, but was curious if someone had a reason that I shouldn't even attempt this.

[benoit-pierre]

You could just do something like that:

extras_require = {
    'tester': [
        "pytest==3.3.2",
    ],
    'dev': [
        "pytest-xdist",
    ],
}
extras_require['dev'] += extras_require['tester']

setup(
    name='my-project',
    extras_require=extras_require,
)

[carver]

Cool, thanks for the suggestion. I will go with that for now.

Do you think there would be opposition to a PR that uses the ".[extra]" or "my-project[extra]" syntax?

[benoit-pierre]

I'm not sure there's a use case for complexifying the syntax supported by setuptools:

• since you can get the behavior you want with the code above
• and the internal format must stay the same (because that's what other tools expect and how it's specified in the official PEPs)

[skirpichev]

@benoit-pierre, your workaround will not work if the project manages package's metadata and options with setup.cfg.

E.g. (for example above):

[options.extras_require]
tester =
    pytest==3.3.2
dev =
    pytest-xdist
    .[tester]

You support similar for install_requires and tests_require, so you should support such things in extras_require as well, just for consistency.

@benoit-pierre, probably this bug should be reopened.

[skirpichev]

@carver, what do you think?

[carver]

@carver, what do you think?

Not up to me. If the official PEP specifically disallows it, then it's highly unlikely the change would be added to setuptools. Your next best options appear to be to either repeat some dependencies, or stop using setup.cfg and use the workaround earlier in the thread.

[benoit-pierre]

@skirpichev: actually, you can rely on the fact that interpolation is enabled:

[options.extras_require]
tester =
    pytest==3.3.2
    pytest-sugar
dev =
    pytest-xdist
    %(tester)s

[skirpichev]

@benoit-pierre, unless I misunderstood, this "feature" currently treated as a bug and eventually disappear?

[benoit-pierre]

It was considered a bug, introduced with Python 3, but it was found that the behaviour has been present all along (with Python 2 too), so disabling interpolation would be a backward incompatible change. Allowing the use case above is a good argument for keeping it, so maybe we should just make it official (mention it in the documentation, and add a test for it). @jaraco, @pganssle: thoughts?

[skirpichev]

backward incompatible change.

IIUC, setuptools allow such changes, just with some period of deprecations, etc.

Allowing the use case above is a good argument for keeping it

IMHO, a more targeted syntax would be preferred.

[pganssle]

I'm finding the INI file syntax for cfg inscrutable, so I can't figure out how to express any dependencies with extras in setup.cfg, but if we pretend that INI is a sane file format and that this would just work, what about something like this:

[metadata]
name=foo
version=1.0.0

[options.extras_require]
tester=
    pytest
    pytest-cov
dev=
    foo[tester]
    pytest-xdist

It seems to work just fine when the dependency is foo and it's mainly a parsing error that foo[tester] can't be parsed. Presumably with a correctly-formatted setup.cfg this would work.

[benoit-pierre]

@pganssle: what's the problem with your example? (Beside the end result not being correct). Parsing works fine.

I don't understand why we would want to use a custom syntax, with the associated need for parsing, validation, handling, etc... when INI interpolation works fine.

[pganssle]

@benoit-pierre I dunno what was wrong, actually. Seems to have started working. Anyway, the end result is correct, I don't think you need INI interpolation or a special syntax. You just need to be explicit about the name of the package rather than having . refer to whatever the current package's name is.

If you do pip install .[dev] with my example setup.cfg it will install all the .[tester] dependencies, plus pytest-xdist.

[benoit-pierre]

That's really fragile, have you tested if an older version of foo happen to be installed (and the dependencies have changed)? Plus pip's dependency resolver leaves a lot to be desired, so there are bound to be other corner cases.

[skirpichev]

You just need to be explicit about the name of the package rather than having "."

I'm not sure if that will work. Tested on: HPAC/matchpy#39. If you add matchpy[tests] to the develop extras and then run

$ pip install .[develop]

from the source tree - the hypothesis package will be missing, for example.

INI Interpolation seems to be a better workaround (so far), then using setup.py, but I'm not certain that this feature will not be removed soon.

[FelixBenning]

Should this be reopened? @benoit-pierre workaround seems to be considered a hack relying on a bug(?) and there is going to be a move to pyproject.toml soonish so the setup.py way won't work forever

[jaraco]

There are a couple of ways to approach this issue, as a Setuptools feature or as a Python Packaging feature.

Setuptools feature

The use-case could be addressed as a Setuptools feature where Setuptools allows self-referential extras and then when building the project expands that definition into the direct dependencies (inlining the self-referential dependencies). That's essentially what ini interpolation does. If users appreciate this feature, I'd recommend to submit a PR with a test capturing the expectation. Such a test will make it more difficult to remove the behavior.

If pyproject.toml doesn't support that form, it will be a feature gap for implementing pyproject.toml support.

Packaging feature

Consider instead a more involved approach where the ecosystem recognizes that extras may reference extras on the same package, either . or with an explicit reference to self's package name. It seems like tools may already have some support for this approach using self[extra].

In this case, there's nothing Setuptools can do without the packaging ecosystem providing support. What's needed is a problem description and a design for how the packaging tools should model the design - details like how the metadata will be modified to support this use-case and address any transitional concerns. Perhaps the solution is as simple as to include .[extra] as an additional extra target. I'd suggest to start the conversation at /pypa/packaging-problems or https://discuss.python.org/c/packaging/14. The effort likely will require a PEP, because it's important to get consensus on how the tools will utilize this new form.

Once there's an established consensus, Setuptools can do its part in implementing the interfaces that will supply the necessary metadata.

[henryiii]

Pip 21.2 allows self dependencies: pypa/pip#10393

The interpolation workaround is appearing in more places - ipython/ipython#13471 and in Twisted. With setup.cfg possibly having numbered days, I'd probably recommend not deprecating the interpolation (#889). But that doesn't solve the issue for PEP 621, such as in #2970. Using a self dependency still causes pip 21.2+ to be required to understand the metadata. I love using setup-cfg-fmt, but it doesn't support interpolation.

Ideally this could be handled at a higher level than pip; if packaging tools recognized .[stuff] (I think ideally) or self-name[stuff] (as pip can handle now) and produce correct metadata, then it could produce wheels that work pre-pip 21.2. Given self-name[stuff] has a meaning in pip, it seems ideal to use .[stuff] for packaging, but that would likely need a PEP update?

PS: #1648 can probably be closed? I'd assume any Python 2.7 issues could be?

[keyvanm]

If I may throw my two cents here, I also think .[extra] syntax would be ideal.

FYI, I am relying on the interpolation feature for this, which works just fine.