Always generated Metadata 2.2

[sbidoul]

This library currently generates Metadata 2.2 only if there is a dynamic field:

https://github.com/FFY00/python-pyproject-metadata/blob/35c93a367bb0cabe38384d9d25ab393e2bc2af1d/pyproject_metadata/__init__.py#L268

Would it make sense to always generate Metadata 2.2, so downstream consumers can rely on the absence of Requires-Dist to mean no dependencies?

[henryiii]

There's also metadata 2.3. Metadata is tricky, as for example you can't get twine to start accepting a newer metadata by updating an existing twine, you have to update its dependencies. Hatching and maturin made a hard switch to 2.3 now that PyPI supports 2.2 and 2.3, and it's caused issues like that to surface.

I think ideally we could allow backends to set it, but provide a default. Then scikit-build-core, meson-python, pdm-backend (and any others) could decide if they needed to, and we could safely bump the default.

I don't think we should bump the default until we drop 3.7, and I don't think we should drop 3.7 until we get a release out with the last 14 months of fixes.

[FFY00]

The motivation behind the current behavior was to keep the smallest metadata version if none of the newer fields were required. Specifically regarding Metadata 2.2, this used to be a big deal since PyPI didn't support it for a while, but as of pypi/warehouse#13606, that is no longer the case, so it alleviates things.

I think it could make sense to let users specify which metadata version to target, but I am not convinced the default behavior should be to use Metadata 2.2. My reasoning for this being that Metadata 2.2 did slightly change the guarantees in the metadata consistency between sdists and wheels — for eg. metadata fields in wheel are now explicitly guaranteed to have the same value as in the sdist, if not marked as dynamic. This changes how the users may use the API a bit, which perhaps we should point out in the docs.

Do you have any example use-cases for your proposal that we could look at?

[henryiii]

In #91, I allow backends to select the metadata version, and leave the default alone. I'll probably move up to 2.2 in scikit-build-core soon, and I'd expect meson-python will want it too, while there might be less pressure on pdm-backend. I probably won't go to 2.3 quite yet - maturin and hatching both did and it's caused some problems (like you can't use twine if you installed it more than two weeks ago, even if you upgrade it). There's a big benefit to using Metadata 2.2, especially if all metadata is static - you can save 700x time or more getting metadata from SDists. I think backends will like having the ability to control this, then the default isn't so important.

Also, setting something in pyproject.toml as dynamic doesn't mean it's going to change when you produce a wheel, so I think we need a new mechanism to specify (truly) dynamic fields. Technically, the current system is somewhat reversed - if there are no dynamic fields, you get the best benefit from metadata 2.2, as you are guaranteed nothing will change SDist -> wheel. If you have some dynamic fields, then it depends how they are filled; much of the time, they are still static from SDist to wheel (like hatch-fancy-pypi-readme). But the current system came from the fact 2.2 wasn't supported by PyPI.

I think #91 is a good start, then we need to work on building a system to mark fields as dynamic/static in the Metadata sense.

[rgommers]

@henryiii's proposal of keeping the default unchanged and allowing backends to upgrade the default at their own pace. For meson-python the upgrade should be straightforward (because very few fields are allowed to be dynamic) and I indeed expect we'd do that by default soon - but not all backends may have it that easy.

PEP 643 encourages producing the latest version: "Source distributions SHOULD use the latest version of the core metadata specification that was available when they were created." However, that is a recommendations for the backends to follow as soon as they are ready, not for pyproject-metadata.

Also, setting something in pyproject.toml as dynamic doesn't mean it's going to change when you produce a wheel,

Good point. I think it will be very likely that there can't be a post-sdist change for most fields that are allowed to be dynamic by backends. This is certainly true for what I think is the most common case: the version field. It's often derived from VCS info, so once we have an sdist and the VCS is out of the picture, it can no longer change.

so I think we need a new mechanism to specify (truly) dynamic fields.

Do you actually have such fields in scikit-build-core? A concrete example would be useful.

[henryiii]

The dynamic-metadata proposal (and now draft library) has a way to specify dynamic metadata. None of the built in plugins are dynamic, though - version can't be dynamic, and the hatch-fancy-pypi-readme wrapper isn't dynamic either. Some examples of dynamic metadata would be adding a runtime-based numpy>=* bound when building or a custom python-requires based on operating system. In general, it's quite rare to have metadata change between SDist and each wheel.