Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 685: address review feedback #2411

Merged
merged 5 commits into from
Mar 10, 2022
Merged

PEP 685: address review feedback #2411

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4f2c217
PEP 685: address review feedback
brettcannon Mar 10, 2022
763b3fa
Fix citation syntax
brettcannon Mar 10, 2022
ab22720
Be stricter about raising errors when writing
brettcannon Mar 10, 2022
510ef04
Smplify what tools are required to do
brettcannon Mar 10, 2022
2e46d89
Delete duplication of text
brettcannon Mar 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 42 additions & 26 deletions pep-0685.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ something tool authors can agree to following.
The issue of there being no standard was brought forward by an
`initial discussion <https://discuss.python.org/t/7614>`__
noting that the extra ``adhoc-ssl`` was not considered equal to the name
``adhoc_ssl`` by pip.
``adhoc_ssl`` by pip 22.


Rationale
Expand All @@ -51,57 +51,70 @@ This collapses any run of the substitution character down to a single
character,
e.g. ``---`` gets collapsed down to ``-``.
This does not produce a valid Python identifier as specified by the
core metadata specification for extra names.
core metadata 2.2 specification for extra names.

`Setuptools does normalization <https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330>`__
`Setuptools 60 does normalization <https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330>`__
via::

re.sub('[^A-Za-z0-9.-]+', '_', name).lower()

The use of an underscore/``_`` differs from PEP 503's use of a
hyphen/``-``.
Runs of characters, unlike PEP 503, do **not** get collapsed,
Runs of ``_``, unlike PEP 503, do **not** get collapsed,
e.g. ``___`` stays the same.

For pip, its
"extra normalisation behaviour is quite convoluted and erratic",
For pip 22, its
"extra normalisation behaviour is quite convoluted and erratic" [pip-erratic]_,
and so its use is not considered.

.. [pip-erratic] https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614/10?


Specification
=============

When comparing extra names, tools MUST normalize the names being compared
using the equivalent semantics of::
using the semantics outlined in `PEP 503 for names <https://peps.python.org/pep-0503/#normalized-names>`__::

re.sub('[^A-Za-z0-9.-]+', '_', name).lower()
re.sub(r"[-_.]+", "-", name).lower()

This normalizes any extra name previously allowed by :pep:`508` in a
fashion consistent with setuptools.
The `core metadata`_ specification will be updated such that the allowed
names for `Provides-Extra`_ matches what :pep:`508` specifies for names.
As this is a superset of what is currently allowed by the core metadata 2.2
specification,
it allows for a *loosening* of the naming requirements.
It will also bring extra naming in line with that of the Name_ field.

For tools writing `core metadata`_,
they MUST write out extra names in their normalized form.
This applies to the ``Provides-Extra`` and ``Provides-Dist`` fields,
both when specifying extras for a distribution as well as the
``extra`` marker.
This will also help enforce the current requirement from the core
metadata specification that extra names be valid Python identifiers.
This applies to the `Provides-Extra`_ field and the `extra marker`_
when used in the `Requires-Dist`_ field.

Tools generating metadata MUST raise an error if a user specified
two or more extra names which would normalize to the same name.
Tools SHOULD warn users when an invalid extra name is read.


Backwards Compatibility
=======================

Older distributions which contain conflicting names when normalized
will no longer have all of their extra names made available to users
as independent options, but instead as a single extra.
It is hoped that relying on setuptools' algorithm for normalization
will minimize the breakage from this.
Moving to :pep:`503` normalization and :pep:`508` name acceptance, it
allows for all preexisting, valid names to continue to be valid.

Based on research looking at a collection of wheels on PyPI [pypi-results]_,
the risk of extra name clashes is limited to 73 clashes when considering
even invalid names,
while *only* looking at valid names leads to only 3 clashes:

As distributions make new releases using tools which implement this PEP,
the backwards-compatibility issues will become less of a concern.
1. dev-test: dev_test, dev-test, dev.test
2. dev-lint: dev-lint, dev.lint, dev_lint
3. apache-beam: apache-beam, apache.beam

By requiring tools writing core metadata to only record the normalized name,
the issue of preexisting, invalid extra names should be diminished over
time.

.. [pypi-results] https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141/17?u=brettcannon


Security Implications
Expand All @@ -110,9 +123,10 @@ Security Implications
It is possible that for a distribution that has conflicting extra names, a
tool ends up installing distributions that somehow weaken the security
of the system.
This is only hypothetical and if it were to occur, it would probably be
more of a security concern for the distributions specifying such extras names
rather than the distribution that pulled them in together.
This is only hypothetical and if it were to occur,
it would probably be more of a security concern for the distributions
specifying such extras names rather than the distribution that pulled
them in together.


How to Teach This
Expand Down Expand Up @@ -160,6 +174,8 @@ CC0-1.0-Universal license, whichever is more permissive.


.. _core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/
.. _extra marker: https://peps.python.org/pep-0508/#extras
.. _Name: https://packaging.python.org/en/latest/specifications/core-metadata/#name
.. _packaging project: https://packaging.pypa.io
.. _Provides-Extra: https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use

.. _Requires-Dist: https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use