Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 674: Leave PyDescr_TYPE() unchanged #2277

Merged
merged 3 commits into from
Jan 26, 2022
Merged

PEP 674: Leave PyDescr_TYPE() unchanged #2277

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a97f340
PEP 674: Leave PyDescr_TYPE() unchanged
vstinner Jan 26, 2022
2fa35b9
add Version History; fix typo; rephrase
vstinner Jan 26, 2022
2574a0e
New batch to typos
vstinner Jan 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 74 additions & 56 deletions pep-0674.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
PEP: 674
Title: Disallow using macros as l-value
Title: Disallow using macros as an l-value
Author: Victor Stinner <[email protected]>
Status: Draft
Type: Standards Track
Expand All @@ -10,15 +10,15 @@ Python-Version: 3.11
Abstract
========

Incompatible C API change disallowing using macros as l-value to:
Incompatible C API change disallowing using macros as an l-value to:

* Allow evolving CPython internals;
* Ease the C API implementation on other Python implementation;
* Help migrating existing C extensions to the HPy API.

On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4
macro changes. Moreover, 24 projects just have to regenerate their
Cython code to use ``Py_SET_TYPE()``.
Only 13 out of the top 5000 PyPI projects (0.3%) are affected by 2 macro
changes. An additional 22 projects just have to regenerate their Cython
code.

In practice, the majority of affected projects only have to make two
changes:
Expand All @@ -32,8 +32,8 @@ changes:
Rationale
=========

Using a macro as a l-value
--------------------------
Using a macro as a an l-value
-----------------------------

In the Python C API, some functions are implemented as macro because
writing a macro is simpler than writing a regular function. If a macro
Expand All @@ -48,7 +48,7 @@ This macro can be used as a **r-value** to **get** an object type::

type = Py_TYPE(object);

It can also be used as **l-value** to **set** an object type::
It can also be used as an **l-value** to **set** an object type::

Py_TYPE(object) = new_type;

Expand All @@ -73,7 +73,7 @@ first to fix this C API compatibility issue. It is a concrete example of
a Python optimization blocked indirectly by the C API.

This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro
has been already modified to disallow using it as a l-value.
has been already modified to disallow using it as an l-value.

These statements are endorsed by Sam Gross (nogil developer).

Expand Down Expand Up @@ -119,8 +119,8 @@ These statements are endorsed by Tim Felgentreff (GraalVM Python developer).
Specification
=============

Disallow using macros as l-value
--------------------------------
Disallow using macros as an l-value
-----------------------------------

PyObject and PyVarObject macros
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -207,12 +207,6 @@ PyDateTime "GET" macros
* ``PyDateTime_TIME_GET_SECOND()``
* ``PyDateTime_TIME_GET_TZINFO()``

PyDescr macros
^^^^^^^^^^^^^^

* ``PyDescr_NAME()``
* ``PyDescr_TYPE()``

Port C extensions to Python 3.11
--------------------------------

Expand All @@ -231,57 +225,71 @@ losing support with older Python versions. The project provides a header
file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and
``Py_SET_SIZE()`` functions to Python 3.8 and older.

PyTuple_GET_ITEM() and PyList_GET_ITEM()
----------------------------------------

The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left
unchanged.

The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and
``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to
the inner ``PyObject**`` array.

Changing these macros is out of the scope of this PEP.


Backwards Compatibility
=======================

The proposed C API changes are backward incompatible on purpose.

At December 1, 2021, a code search on the PyPI top 5000 projects (4760
projects in practice, others don't have a source archive) found that
`only 14 projects are affected
<https://bugs.python.org/issue45476#msg407456>`_ (0.3%):
On January 26, 2022, a code search on the top 5000 PyPI projects (4762
projects in practice; others don't have a source archive) found that
only 13 projects are affected (0.3%):

* Cython (0.29.26)
* PyGObject (3.42.0)
CAM-Gerlach marked this conversation as resolved.
Show resolved Hide resolved
* datatable (1.0.0)
* frozendict (2.1.1)
* guppy3 (3.1.2)
* M2Crypto (0.38.0)
* mecab-python3 (1.0.4)
* mypy (0.910)
* Naked (0.1.31)
* numpy (1.22.1)
* pickle5 (0.0.12)
* psycopg2 (2.9.3)
* pycurl (7.44.1)
CAM-Gerlach marked this conversation as resolved.
Show resolved Hide resolved
* pysha3 (1.0.2)
* python-snappy (0.6.0)
* recordclass (0.16.3)
* recordclass (0.17.1)
* scipy (1.7.3)
* zodbpickle (2.2.0)
* zstd (1.5.0.2)

These 14 projects only use 4 macros as l-value:
Of these 13 projects, only 2 macros are used as an l-value:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Of these 13 projects, only 2 macros are used as an l-value:
In these 13 projects, only 2 macros are used as an l-value:

Grammar

``Py_TYPE()`` and ``Py_SIZE()``.

* ``PyDescr_NAME()`` and ``PyDescr_TYPE()`` (2 projects)
* ``Py_SIZE()`` (8 projects)
* ``Py_TYPE()`` (4 projects)
An additional 22 projects just have to regenerate their Cython code to
use ``Py_SET_TYPE()`` and ``Py_SET_SIZE()``.

Moreover, `24 projects just have to regenerate their Cython code
<https://bugs.python.org/issue45476#msg407416>`_ to use
``Py_SET_TYPE()``.
This change does not follow the :pep:`387` deprecation process. There is
no known way to emit a deprecation warning only when a macro is used as
an l-value, but not when it's used differently (ex: as a r-value).

This change does not follow the :pep:`387` deprecation process. There is no
known way to emit a deprecation warning only when a macro is used as a
l-value, but not when it's used differently (ex: as a r-value).
PyTuple_GET_ITEM() and PyList_GET_ITEM() are left unchanged
-----------------------------------------------------------

The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left
unchanged.

The code patterns ``&PyTuple_GET_ITEM(tuple, 0)`` and
``&PyList_GET_ITEM(list, 0)`` are still commonly used to get access to
the inner ``PyObject**`` array.

Changing these macros is out of the scope of this PEP.

PyDescr_NAME() and PyDescr_TYPE() are left unchanged
----------------------------------------------------

The ``PyDescr_NAME()`` and ``PyDescr_TYPE()`` macros are left unchanged.

These macros give access to ``PyDescrObject.d_name`` and
``PyDescrObject.d_type`` members. They can be used as l-values to set
these members.

The SWIG project uses these macros as l-values to set these members. It
would be possible to modify SWIG to prevent setting ``PyDescrObject``
structure members directly, but it is not really worth it since the
``PyDescrObject`` structure is not performance critical and is unlikely
to change soon.

See the `bpo-46538 <https://bugs.python.org/issue46538>`_ "[C API] Make
the PyDescrObject structure opaque: PyDescr_NAME() and PyDescr_TYPE()"
issue for more details.


Relationship with the HPy project
Expand Down Expand Up @@ -332,8 +340,9 @@ extensions to HPy is expected to take a few years.

This PEP proposes to make the C API "less bad" by fixing one problem
which is clearily identified as causing practical issues: macros used as
l-values. This PEP only requires to update a minority of C extensions,
and usually only a few lines need to be changed in impacted extensions.
l-values. This PEP only requires updating a minority of C
extensions, and usually only a few lines need to be changed in impacted
extensions.

For example, NumPy 1.22 is made of 307,300 lines of C code, and adapting
NumPy to the this PEP only modified 11 lines (use Py_SET_TYPE and
Expand All @@ -358,8 +367,8 @@ example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and
the ``Py_TYPE()`` documentation now requires to use the
``Py_SET_TYPE()`` function to set an object type.

If developers use macros as l-value, it's their responsibility when
their code breaks, not the Python responsibility. We are operating under
If developers use macros as an l-value, it's their responsibility when
their code breaks, not Python's responsibility. We are operating under
the consenting adults principle: we expect users of the Python C API to
use it as documented and expect them to take care of the fallout, if
things break when they don't.
Expand All @@ -370,9 +379,9 @@ API documentation. The majority of developers are only using CPython and
so are not aware of compatibility issues with other Python
implementations.

Moreover, continuing to allow using macros as l-value does not help the
HPy project and leaves the burden of emulating them on GraalVM's Python
implementation.
Moreover, continuing to allow using macros as an l-value does not help
the HPy project, and leaves the burden of emulating them on GraalVM's
Python implementation.


Macros already modified
Expand Down Expand Up @@ -423,6 +432,15 @@ References
<https://bugs.python.org/issue30459>`_ (May 2017)


Version History
===============

* Version 3: No longer change PyDescr_TYPE() and PyDescr_NAME() macros
* Version 2: Add "Relationship with the HPy project" section, remove
the PyPy section
* Version 1: First public version


Copyright
=========

Expand Down