From 9a22e13c07655e435cdf0e7d9e7aeea24f517b2f Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 21 Feb 2022 17:50:21 +0100 Subject: [PATCH 1/6] PEP 670: clarify cast; don't change return type * Clarify how arguments are cast * Limited C API 3.11 no longer cast arguments * No longer remove return value * Require to not change the return type * Don't change macros having multiple return types --- pep-0670.rst | 123 +++++++++++++++++++++++++++------------------------ 1 file changed, 65 insertions(+), 58 deletions(-) diff --git a/pep-0670.rst b/pep-0670.rst index 043cf6c53ab..83ace8e2c10 100644 --- a/pep-0670.rst +++ b/pep-0670.rst @@ -20,12 +20,8 @@ them usable by Python extensions which cannot use macros or static inline functions, like extensions written in a programming languages other than C or C++. -Remove the return value of macros having a return value, whereas they -should not, to aid detecting bugs in C extensions when the C API is -misused. - -Some function arguments are still cast to ``PyObject*`` to prevent -emitting new compiler warnings. +Function arguments of pointer types are still cast and return types are +not changed to prevent emitting new compiler warnings. Macros which can be used as l-value in an assignment are not converted to functions to avoid introducing incompatible changes. @@ -177,6 +173,7 @@ The following macros should not be converted: * Macros which can be used as l-value in an assignment. This change is an incompatible change and is out of the scope of this PEP. Example: ``PyBytes_AS_STRING()``. +* Macros having different return types depending on the code path. Convert static inline functions to regular functions @@ -196,74 +193,80 @@ Using static inline functions in the internal C API is fine: the internal C API exposes implementation details by design and should not be used outside Python. -Cast to PyObject* ------------------ +Cast pointer arguments +---------------------- + +Existing cast +^^^^^^^^^^^^^ + +Currently, most macros accepting pointers cast pointers arguments to the +expected types. For example, in Python 3.6, the ``Py_TYPE()`` macro +casts its argument to ``PyObject*``:: -When a macro is converted to a function and the macro casts its -arguments to ``PyObject*``, the new function comes with a new macro -which cast arguments to ``PyObject*`` to prevent emitting new compiler -warnings. This implies that a converted function will accept pointers to -structures inheriting from ``PyObject`` (ex: ``PyTupleObject``). + #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) -For example, the ``Py_TYPE(obj)`` macro casts its ``obj`` argument to -``PyObject*``:: +The ``Py_TYPE()`` macro accept ``PyObject*`` type, but also any pointer +type like ``PyTupleObject*`` or ``PyDictObject*``. - #define _PyObject_CAST_CONST(op) ((const PyObject*)(op)) +Add a new macro to keep the cast +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - static inline PyTypeObject* _Py_TYPE(const PyObject *ob) { +When a macro is converted to a function and the macro casts at least one +of its argument, a new macro is added to keep the cast. The new macro +and the function have the same name. Example with the ``Py_TYPE()`` +macro converted to a static inline function:: + + static inline PyTypeObject* Py_TYPE(PyObject *ob) { return ob->ob_type; } - #define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob)) - -The undocumented private ``_Py_TYPE()`` function must not be called -directly. Only the documented public ``Py_TYPE()`` macro must be used. + #define Py_TYPE(ob) Py_TYPE((PyObject*)(ob)) -Later, the cast can be removed on a case by case basis, but that is out -of scope for this PEP. +The cast is kept for other pointer types like ``void*``. Removing a cast +to ``void*`` would emit a new warning if the function is called with a +variable of ``const void*`` type. For example, the *data* argument of +the ``PyUnicode_WRITE()`` accepts ``const void*`` type and then casts +*data* to ``void*`` type. -Remove the return value ------------------------ +Avoid the cast in the limited C API 3.11 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -When a macro is implemented as an expression, it has an implicit return -value. This return value can be misused in third party C extensions. -See `bpo-30459 `__ regarding the -misuse of the ``PyList_SET_ITEM()`` and ``PyCell_SET()`` macros. +The cast is removed from the limited C API version 3.11 and newer: the +caller must pass the expected type, or do the cast. Example with the +``Py_TYPE()`` function:: -Such issue is hard to catch while reviewing macro code. Removing the -return value aids detecting bugs in C extensions when the C API is -misused. + static inline PyTypeObject* Py_TYPE(PyObject *ob) { + return ob->ob_type; + } + #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 + # define Py_TYPE(ob) Py_TYPE((PyObject*)(ob)) + #endif -The issue has already been fixed in public C API macros by the -`bpo-30459 `__ in Python 3.10: add a -``(void)`` cast to the affected macros. Example of the -``PyTuple_SET_ITEM()`` macro:: - #define PyTuple_SET_ITEM(op, i, v) ((void)(_PyTuple_CAST(op)->ob_item[i] = v)) +Return type is not changed +-------------------------- -Example of macros currently using a ``(void)`` cast to have no return -value: +When a macro is converted to a function, its return type must not change +to prevent emitting new compiler warnings. -* ``PyCell_SET()`` -* ``PyList_SET_ITEM()`` -* ``PyTuple_SET_ITEM()`` -* ``Py_BUILD_ASSERT()`` -* ``_PyGCHead_SET_FINALIZED()`` -* ``_PyGCHead_SET_NEXT()`` -* ``_PyObject_ASSERT_FROM()`` -* ``_Py_atomic_signal_fence()`` -* ``_Py_atomic_store_64bit()`` -* ``asdl_seq_SET()`` -* ``asdl_seq_SET_UNTYPED()`` +For example, Python 3.7 changed ``PyUnicode_AsUTF8()`` return type from +``char*`` to ``const char*`` (`commit +`__). +The change emitted new compiler warnings when building C extensions +expecting ``char*``. This PEP doesn't change the return type to prevent +this issue. Backwards Compatibility ======================= -Removing the return value of macros is an incompatible API change made -on purpose: see the `Remove the return value`_ section. +The PEP is design to avoid C API incompatible changes. + +Only C extensions targeting explicitly the limited C API version 3.11 +must now pass the expected types to functions: pointer arguments are no +longer cast to the expected types. -Some function arguments are still cast to ``PyObject*`` to prevent -emitting new compiler warnings. +Function arguments of pointer types are still cast and return types are +not changed to prevent emitting new compiler warnings. Macros which can be used as l-value in an assignment are not modified by this PEP to avoid incompatible changes. @@ -275,10 +278,6 @@ Rejected Ideas Keep macros, but fix some macro issues -------------------------------------- -Converting macros to functions is not needed to `remove the return -value`_: adding a ``(void)`` cast is enough. For example, the -``PyList_SET_ITEM()`` macro was already fixed like that. - Macros are always "inlined" with any C compiler. The duplication of side effects can be worked around in the caller of @@ -600,6 +599,14 @@ References (March 2021). +Version History +=============== + +* Version 2: No longer remove return value, remove arguments cast from + the limited C API. +* Version 1: First public version + + Copyright ========= From 977cde60a5f88cb8fc63a3cce895780bd502090c Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 21 Feb 2022 18:19:25 +0100 Subject: [PATCH 2/6] Rephrase --- pep-0670.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/pep-0670.rst b/pep-0670.rst index 83ace8e2c10..9f43586d0d9 100644 --- a/pep-0670.rst +++ b/pep-0670.rst @@ -206,7 +206,7 @@ casts its argument to ``PyObject*``:: #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) The ``Py_TYPE()`` macro accept ``PyObject*`` type, but also any pointer -type like ``PyTupleObject*`` or ``PyDictObject*``. +type like ``PyLongObject*`` or ``PyDictObject*``. Add a new macro to keep the cast ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -221,11 +221,12 @@ macro converted to a static inline function:: } #define Py_TYPE(ob) Py_TYPE((PyObject*)(ob)) -The cast is kept for other pointer types like ``void*``. Removing a cast -to ``void*`` would emit a new warning if the function is called with a -variable of ``const void*`` type. For example, the *data* argument of -the ``PyUnicode_WRITE()`` accepts ``const void*`` type and then casts -*data* to ``void*`` type. +The cast is kept any pointer types, not only ``PyObject*``. + +Removing a cast to ``void*`` would emit a new warning if the function is +called with a variable of ``const void*`` type. For example, the +``PyUnicode_WRITE()`` macro casts its *data* argument to ``void*``, and +so accept ``const void*`` type, even if it writes into *data*. Avoid the cast in the limited C API 3.11 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From a1eae586f0dd7c4d5475c774cf894ebaab3a1a46 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 21 Feb 2022 18:20:28 +0100 Subject: [PATCH 3/6] Typ --- pep-0670.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0670.rst b/pep-0670.rst index 9f43586d0d9..847daa8a9c4 100644 --- a/pep-0670.rst +++ b/pep-0670.rst @@ -221,7 +221,7 @@ macro converted to a static inline function:: } #define Py_TYPE(ob) Py_TYPE((PyObject*)(ob)) -The cast is kept any pointer types, not only ``PyObject*``. +The cast is kept all pointer types, not only ``PyObject*``. Removing a cast to ``void*`` would emit a new warning if the function is called with a variable of ``const void*`` type. For example, the From dde12d3d8609f9469f5c85b5efe291e247cd8591 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 22 Feb 2022 13:08:43 +0100 Subject: [PATCH 4/6] Address CAM's review --- pep-0670.rst | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/pep-0670.rst b/pep-0670.rst index 847daa8a9c4..d9d5178a3b5 100644 --- a/pep-0670.rst +++ b/pep-0670.rst @@ -9,6 +9,9 @@ Created: 19-Oct-2021 Python-Version: 3.11 +.. highlight:: c + + Abstract ======== @@ -197,19 +200,19 @@ Cast pointer arguments ---------------------- Existing cast -^^^^^^^^^^^^^ +''''''''''''' -Currently, most macros accepting pointers cast pointers arguments to the -expected types. For example, in Python 3.6, the ``Py_TYPE()`` macro -casts its argument to ``PyObject*``:: +Currently, most macros accepting pointers cast pointer arguments to +their expected types. For example, in Python 3.6, the ``Py_TYPE()`` +macro casts its argument to ``PyObject*``:: #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) -The ``Py_TYPE()`` macro accept ``PyObject*`` type, but also any pointer -type like ``PyLongObject*`` or ``PyDictObject*``. +The ``Py_TYPE()`` macro accepts the ``PyObject*`` type, but also any +pointer types, such as ``PyLongObject*`` and ``PyDictObject*``. Add a new macro to keep the cast -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +'''''''''''''''''''''''''''''''' When a macro is converted to a function and the macro casts at least one of its argument, a new macro is added to keep the cast. The new macro @@ -229,7 +232,7 @@ called with a variable of ``const void*`` type. For example, the so accept ``const void*`` type, even if it writes into *data*. Avoid the cast in the limited C API 3.11 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +'''''''''''''''''''''''''''''''''''''''' The cast is removed from the limited C API version 3.11 and newer: the caller must pass the expected type, or do the cast. Example with the @@ -260,9 +263,9 @@ this issue. Backwards Compatibility ======================= -The PEP is design to avoid C API incompatible changes. +The PEP is designed to avoid C API incompatible changes. -Only C extensions targeting explicitly the limited C API version 3.11 +Only C extensions explicitly targeting the limited C API version 3.11 must now pass the expected types to functions: pointer arguments are no longer cast to the expected types. @@ -603,8 +606,8 @@ References Version History =============== -* Version 2: No longer remove return value, remove arguments cast from - the limited C API. +* Version 2: No longer remove return values; remove argument casting + from the limited C API. * Version 1: First public version From ba4aadf98e0bb495e63f99f49a3b5f9fec6d19ba Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 22 Feb 2022 13:12:24 +0100 Subject: [PATCH 5/6] Address CAM's review --- pep-0670.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/pep-0670.rst b/pep-0670.rst index d9d5178a3b5..0e993651569 100644 --- a/pep-0670.rst +++ b/pep-0670.rst @@ -215,7 +215,7 @@ Add a new macro to keep the cast '''''''''''''''''''''''''''''''' When a macro is converted to a function and the macro casts at least one -of its argument, a new macro is added to keep the cast. The new macro +of its arguments, a new macro is added to keep the cast. The new macro and the function have the same name. Example with the ``Py_TYPE()`` macro converted to a static inline function:: @@ -224,19 +224,19 @@ macro converted to a static inline function:: } #define Py_TYPE(ob) Py_TYPE((PyObject*)(ob)) -The cast is kept all pointer types, not only ``PyObject*``. +The cast is kept for all pointer types, not only ``PyObject*``. Removing a cast to ``void*`` would emit a new warning if the function is called with a variable of ``const void*`` type. For example, the ``PyUnicode_WRITE()`` macro casts its *data* argument to ``void*``, and -so accept ``const void*`` type, even if it writes into *data*. +so accepts ``const void*`` type, even if it writes into *data*. -Avoid the cast in the limited C API 3.11 -'''''''''''''''''''''''''''''''''''''''' +Avoid the cast in the limited C API version 3.11 +'''''''''''''''''''''''''''''''''''''''''''''''' The cast is removed from the limited C API version 3.11 and newer: the -caller must pass the expected type, or do the cast. Example with the -``Py_TYPE()`` function:: +caller must pass the expected type, or perform the cast. An example with +the ``Py_TYPE()`` function:: static inline PyTypeObject* Py_TYPE(PyObject *ob) { return ob->ob_type; From 08d57fa02cb36e9b66f36ff2698632c62fbc4b04 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 22 Feb 2022 13:14:52 +0100 Subject: [PATCH 6/6] Fix Sphinx --- pep-0670.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/pep-0670.rst b/pep-0670.rst index 0e993651569..bd331f223c6 100644 --- a/pep-0670.rst +++ b/pep-0670.rst @@ -9,9 +9,6 @@ Created: 19-Oct-2021 Python-Version: 3.11 -.. highlight:: c - - Abstract ========