From b06bbdf9c6ace7ac4b2bcd81c5d60637351ce08a Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Thu, 29 Jul 2021 11:06:32 +0100 Subject: [PATCH 01/10] Initial version of PEP for locals() and f_locals --- pep-06xx.rst | 217 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 217 insertions(+) create mode 100644 pep-06xx.rst diff --git a/pep-06xx.rst b/pep-06xx.rst new file mode 100644 index 00000000000..126db6c2466 --- /dev/null +++ b/pep-06xx.rst @@ -0,0 +1,217 @@ +PEP: 6xx +Title: Consistent views of namespaces +Author: Mark Shannon +Status: Draft +Type: Standards +Content-Type: text/x-rst +Created: 30-Jul-2021 +Post-History: XX-Aug-2021 + + +Abstract +======== + +In early versions of Python, back in the 20th century, all namespaces, +whether in functions, classes or modules, were implemented the same way as a dictionary. + +For performance reasons, the implementation of function namespaces was changed. +Unfortunately this meant that the views of the namespaces, ``locals()`` and +``frame.f_locals`` ceased to be consistent and some odd bugs crept in over the yields +as threads, generators and coroutines were added. + +This PEP proposes make the views of namespaces consistent once more. +Modifications to ``locals()`` will show in the underlying variables, +``frame.f_locals`` will be consistent with ``locals()`` and will be +consistent with the underlying variables regardless of threading or coroutines. + +Motivation +========== + +The current implementation of ``locals()`` and ``frame.f_locals`` is slow, +inconsistent and buggy. +We want to make it faster, consistent and most importantly fix the bugs. + +For example:: + + class C: + x = 1 + locals()['x'] = 2 + print(x) + +prints ``2`` + +but:: + + def f(): + x = 1 + locals()['x'] = 2 + print(x) + f() + +prints ``1`` + +This is inconsistent, and confusing. +With this PEP both examples would print ``2``. + +Worse than that, the current behavior can result in strange bugs [1]_ + +There are no compensating advantages for the current behavior; +it is unreliable and slow. + +Rationale +========= + +The current implementation of ``locals()`` and ``frame.f_locals`` +returns a dictionary that it created on the fly from the array of +local variables. This can result in the array getting out of sync, +resulting in writes to the ``locals()`` object not showing as +modifications local variables, or worse writes to local variables +being lost. + +By making ``locals()`` and ``frame.f_locals`` return a view on the +underlying frame, these problems go away. ``locals()`` is always in +sync with the frame, because it is a view of it, not a copy of it. + +Specification +============= + +Python +------ + +``frame.f_locals`` will return a view object on the frame that +implements the ``collections.abc.Mapping`` interface. + +``locals()`` will be defined simply as:: + + def locals(): + return sys._getframe(0).f_locals + + +All writes to the ``f_locals`` mapping will be immediately visible +in the underlying variables. All changes to the underlying variables +will be immediately visible in the mapping. The ``f_locals`` is a full +mapping, and can have arbitrary key-value pairs added to it. + +For example:: + + def test(): + nonlocal var + x = 1 + locals()['x'] = 2 + locals()['y'] = 4 + locals()['z'] = 5 + y + print(dict(locals()), x) + +``test()`` will print ``{'x': 2, 'y': 4, 'z': 5} 2`` + +In 3.10, the above will fail with a ``NameError`` as ``y`` +as the definition ``locals()['y'] = 4`` is lost. + +C-API +----- + +Two new C-API functions will be added:: + + PyObject *PyEval_Locals(int depth) + PyObject *PyFrame_GetLocals(PyFrameObject *f) + +``PyEval_Locals(depth)`` is equivalent to: ``sys._getframe(depth).f_locals`` +``PyFrame_GetLocals(f)`` is equivalent to: ``f.f_locals`` + +The existing C-API function ``PyEval_GetLocals()`` will always raise an +exception with a message like:: + + PyEval_GetLocals() is unsafe. Please use PyEval_Locals() instead. + +This is necessary as ``PyEval_GetLocals()`` +returns a borrowed reference which cannot be made safe. + +Behavior of f_locals for optimized functions +-------------------------------------------- + +Although ``f.f_locals`` behaves as if it were the namespace of the function, +some differences will be observable, +most notably that ``f.f_locals is f.f_locals`` may be ``False``. + +However ``f.f_locals == f.f_locals`` will be ``True`` and +any changes to the underlying variables, by any means, will be +visible to all accesses to those variables. + + + +Backwards Compatibility +======================= + +Python +------ + +The current implementation has many corner cases and oddities. +Code that works around those may need to be changed. +Code that uses ``locals()`` for simple templating, or print debugging, +will continue to work correctly. Debuggers and other tools that use +``f_locals`` to modify local variables, will now work correctly, +even in the presence of threaded code, coroutines and generators. + +C-API +----- + +The change to ``PyEval_GetLocals()`` is a backwards compatibility break. +Code that uses ``PyEval_GetLocals()`` will continue to operate safely, but +will need to be changed to use ``PyEval_Locals()`` to restore functionality. + +This code:: + + locals = PyEval_GetLocals(); + if (locals == NULL) { + goto error_handler; + } + Py_INCREF(locals); + +should be replaced with:: + + locals = PyEval_Locals(0); + if (locals == NULL) { + goto error_handler; + } + + +Reference Implementation +======================== + +TO DO. + + +Rejected Ideas +============== + +[Why certain ideas that were brought while discussing this PEP were not ultimately pursued.] + + +Open Issues +=========== + +[Any points that are still being decided/discussed.] + + +References +========== + +.. [1] https://bugs.python.org/issue30744 + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From ac45cac2bc9e2c0bb8636d5d4f1e4769a4f65189 Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Thu, 29 Jul 2021 11:22:17 +0100 Subject: [PATCH 02/10] Edits --- pep-06xx.rst | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/pep-06xx.rst b/pep-06xx.rst index 126db6c2466..7705acef29f 100644 --- a/pep-06xx.rst +++ b/pep-06xx.rst @@ -12,16 +12,17 @@ Abstract ======== In early versions of Python, back in the 20th century, all namespaces, -whether in functions, classes or modules, were implemented the same way as a dictionary. +whether in functions, classes or modules, were all implemented the same way, +as a dictionary. For performance reasons, the implementation of function namespaces was changed. Unfortunately this meant that the views of the namespaces, ``locals()`` and -``frame.f_locals`` ceased to be consistent and some odd bugs crept in over the yields +``frame.f_locals``, ceased to be consistent and some odd bugs crept in over the years as threads, generators and coroutines were added. This PEP proposes make the views of namespaces consistent once more. -Modifications to ``locals()`` will show in the underlying variables, -``frame.f_locals`` will be consistent with ``locals()`` and will be +Modifications to ``locals()`` will show in the underlying variables. +``frame.f_locals`` will be consistent with ``locals()`` and both will be consistent with the underlying variables regardless of threading or coroutines. Motivation @@ -62,11 +63,11 @@ Rationale ========= The current implementation of ``locals()`` and ``frame.f_locals`` -returns a dictionary that it created on the fly from the array of -local variables. This can result in the array getting out of sync, -resulting in writes to the ``locals()`` object not showing as -modifications local variables, or worse writes to local variables -being lost. +returns a dictionary that is created on the fly from the array of +local variables. This can result in the array and dictionary getting +out of sync with each other. Writes to the ``locals()`` do not show +up as modifications local variables. Writes to local variables can +ven get lost. By making ``locals()`` and ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``locals()`` is always in @@ -89,13 +90,12 @@ implements the ``collections.abc.Mapping`` interface. All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables -will be immediately visible in the mapping. The ``f_locals`` is a full -mapping, and can have arbitrary key-value pairs added to it. +will be immediately visible in the mapping. The ``f_locals`` will be a +full mapping, and can have arbitrary key-value pairs added to it. For example:: def test(): - nonlocal var x = 1 locals()['x'] = 2 locals()['y'] = 4 @@ -105,8 +105,8 @@ For example:: ``test()`` will print ``{'x': 2, 'y': 4, 'z': 5} 2`` -In 3.10, the above will fail with a ``NameError`` as ``y`` -as the definition ``locals()['y'] = 4`` is lost. +In 3.10, the above will fail with a ``NameError``, as the +definition of ``y`` by ``locals()['y'] = 4`` is lost. C-API ----- @@ -134,11 +134,9 @@ Although ``f.f_locals`` behaves as if it were the namespace of the function, some differences will be observable, most notably that ``f.f_locals is f.f_locals`` may be ``False``. -However ``f.f_locals == f.f_locals`` will be ``True`` and -any changes to the underlying variables, by any means, will be -visible to all accesses to those variables. - - +However ``f.f_locals == f.f_locals`` will be ``True``, and +all changes to the underlying variables, by any means, will be +always be visible. Backwards Compatibility ======================= From 3b66fbc89e183b94f52518da80d4c5dc6711807d Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Thu, 29 Jul 2021 20:59:51 +0100 Subject: [PATCH 03/10] More edits. --- pep-06xx.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pep-06xx.rst b/pep-06xx.rst index 7705acef29f..fe7b3259253 100644 --- a/pep-06xx.rst +++ b/pep-06xx.rst @@ -21,9 +21,9 @@ Unfortunately this meant that the views of the namespaces, ``locals()`` and as threads, generators and coroutines were added. This PEP proposes make the views of namespaces consistent once more. -Modifications to ``locals()`` will show in the underlying variables. -``frame.f_locals`` will be consistent with ``locals()`` and both will be -consistent with the underlying variables regardless of threading or coroutines. +Modifications to ``locals()`` will be visible in the underlying variables, +``frame.f_locals`` will be consistent with ``locals()``, and they will all be +consistent regardless of threading or coroutines. Motivation ========== @@ -65,9 +65,9 @@ Rationale The current implementation of ``locals()`` and ``frame.f_locals`` returns a dictionary that is created on the fly from the array of local variables. This can result in the array and dictionary getting -out of sync with each other. Writes to the ``locals()`` do not show -up as modifications local variables. Writes to local variables can -ven get lost. +out of sync with each other. Writes to the ``locals()`` may not show +up as modifications to local variables. Writes to local variables can +get lost. By making ``locals()`` and ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``locals()`` is always in From 36e532e9686f86a70e42935824ca38bece6e3f56 Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 20 Aug 2021 11:54:37 +0100 Subject: [PATCH 04/10] Change behavior of locals() to match PEP 558. Assorted other edits. --- pep-06xx.rst | 324 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 275 insertions(+), 49 deletions(-) diff --git a/pep-06xx.rst b/pep-06xx.rst index fe7b3259253..e687f867499 100644 --- a/pep-06xx.rst +++ b/pep-06xx.rst @@ -11,32 +11,37 @@ Post-History: XX-Aug-2021 Abstract ======== -In early versions of Python, back in the 20th century, all namespaces, -whether in functions, classes or modules, were all implemented the same way, -as a dictionary. - -For performance reasons, the implementation of function namespaces was changed. -Unfortunately this meant that the views of the namespaces, ``locals()`` and -``frame.f_locals``, ceased to be consistent and some odd bugs crept in over the years -as threads, generators and coroutines were added. - -This PEP proposes make the views of namespaces consistent once more. -Modifications to ``locals()`` will be visible in the underlying variables, -``frame.f_locals`` will be consistent with ``locals()``, and they will all be +In early versions of Python all namespaces, whether in functions, +classes or modules, were all implemented the same way; as a dictionary. + +For performance reasons, the implementation of function namespaces was +changed. Unfortunately this meant that accessing these namespaces through +``locals()`` and ``frame.f_locals`` ceased to be consistent and some +odd bugs crept in over the years as threads, generators and coroutines +were added. + +This PEP proposes make these namespaces consistent once more. +Modifications to ``frame.f_locals`` will always be visible in +the underlying variables. Modifications to local variables will +immediately be visible in ``frame.f_locals``, and they will be consistent regardless of threading or coroutines. +The ``locals()`` function will act the same as it does now for class +and modules scopes. For function scopes it will return an instantaneous +snapshot of the underlying ``frame.f_locals``. + Motivation ========== The current implementation of ``locals()`` and ``frame.f_locals`` is slow, inconsistent and buggy. -We want to make it faster, consistent and most importantly fix the bugs. +We want to make it faster, consistent, and most importantly fix the bugs. For example:: class C: x = 1 - locals()['x'] = 2 + sys._getframe().f_locals['x'] = 2 print(x) prints ``2`` @@ -45,7 +50,7 @@ but:: def f(): x = 1 - locals()['x'] = 2 + sys._getframe().f_locals['x'] = 2 print(x) f() @@ -62,15 +67,14 @@ it is unreliable and slow. Rationale ========= -The current implementation of ``locals()`` and ``frame.f_locals`` -returns a dictionary that is created on the fly from the array of -local variables. This can result in the array and dictionary getting -out of sync with each other. Writes to the ``locals()`` may not show -up as modifications to local variables. Writes to local variables can -get lost. +The current implementation of ``frame.f_locals`` returns a dictionary +that is created on the fly from the array of local variables. +This can result in the array and dictionary getting out of sync with +each other. Writes to the ``locals()`` may not show up as +modifications to local variables. Writes to local variables can get lost. -By making ``locals()`` and ``frame.f_locals`` return a view on the -underlying frame, these problems go away. ``locals()`` is always in +By making ``frame.f_locals`` return a view on the +underlying frame, these problems go away. ``frame.f_locals`` is always in sync with the frame, because it is a view of it, not a copy of it. Specification @@ -82,42 +86,57 @@ Python ``frame.f_locals`` will return a view object on the frame that implements the ``collections.abc.Mapping`` interface. -``locals()`` will be defined simply as:: +For module and class scopes ``frame.f_locals`` will be a dictionary, +for function scopes it will be a custom class. - def locals(): - return sys._getframe(0).f_locals +``locals()`` will be defined as:: + def locals(): + f_locals = sys._getframe().f_locals + if not isinstance(dict): + f_locals = dict(f_locals) + return f_locals All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables -will be immediately visible in the mapping. The ``f_locals`` will be a -full mapping, and can have arbitrary key-value pairs added to it. +will be immediately visible in the mapping. The ``f_locals`` object will +be a full mapping, and can have arbitrary key-value pairs added to it. For example:: + def l(): + "Get the locals of caller" + return sys._getframe(1).f_locals + def test(): x = 1 - locals()['x'] = 2 - locals()['y'] = 4 - locals()['z'] = 5 + l()['x'] = 2 + l()['y'] = 4 + l()['z'] = 5 y - print(dict(locals()), x) + print(locals(), x) ``test()`` will print ``{'x': 2, 'y': 4, 'z': 5} 2`` -In 3.10, the above will fail with a ``NameError``, as the -definition of ``y`` by ``locals()['y'] = 4`` is lost. +In Python 3.10, the above will fail with a ``NameError``, +as the definition of ``y`` by ``l()['y'] = 4`` is lost. C-API ----- +Extensions to the API +''''''''''''''''''''' + Two new C-API functions will be added:: - PyObject *PyEval_Locals(int depth) + PyObject *PyEval_Locals(void) PyObject *PyFrame_GetLocals(PyFrameObject *f) -``PyEval_Locals(depth)`` is equivalent to: ``sys._getframe(depth).f_locals`` -``PyFrame_GetLocals(f)`` is equivalent to: ``f.f_locals`` +``PyEval_Locals()`` is equivalent to: ``locals()``. + +``PyFrame_GetLocals(f)`` is equivalent to: ``f.f_locals``. + +Both functions will return a new reference. The existing C-API function ``PyEval_GetLocals()`` will always raise an exception with a message like:: @@ -127,12 +146,29 @@ exception with a message like:: This is necessary as ``PyEval_GetLocals()`` returns a borrowed reference which cannot be made safe. +Changes to existing APIs +'''''''''''''''''''''''' + +The existing C-API function ``PyEval_GetLocals()`` will always raise an +exception with a message like:: + + PyEval_GetLocals() is unsafe. Please use PyEval_Locals() instead. + +This is necessary as ``PyEval_GetLocals()`` +returns a borrowed reference which cannot be made safe. + +The following functions will be retained, but will become no-ops:: + + PyFrame_FastToLocalsWithError() + PyFrame_FastToLocals() + PyFrame_LocalsToFast() + Behavior of f_locals for optimized functions -------------------------------------------- -Although ``f.f_locals`` behaves as if it were the namespace of the function, -some differences will be observable, -most notably that ``f.f_locals is f.f_locals`` may be ``False``. +Although ``f.f_locals`` behaves as if it were the namespace of the function, +some differences will be observable, most notably that +``f.f_locals is f.f_locals`` may be ``False``. However ``f.f_locals == f.f_locals`` will be ``True``, and all changes to the underlying variables, by any means, will be @@ -154,6 +190,9 @@ even in the presence of threaded code, coroutines and generators. C-API ----- +PyEval_GetLocals +'''''''''''''''' + The change to ``PyEval_GetLocals()`` is a backwards compatibility break. Code that uses ``PyEval_GetLocals()`` will continue to operate safely, but will need to be changed to use ``PyEval_Locals()`` to restore functionality. @@ -168,43 +207,230 @@ This code:: should be replaced with:: - locals = PyEval_Locals(0); + locals = PyEval_Locals(); if (locals == NULL) { goto error_handler; } +PyFrame_FastToLocals, etc. +'''''''''''''''''''''''''' + +These functions were designed to convert the internal "fast" representation +of the locals variables of a function to a dictionary, and vice versa. -Reference Implementation -======================== +They are no longer required. C code that directly accesses the `f_locals` +field of a frame should be modified to call ``PyFrame_GetLocals()`` instead:: -TO DO. + PyFrame_FastToLocals(frame); + PyObject *locals = frame.f_locals; + Py_INCREF(locals); +becomes:: + + PyObject *locals = PyFrame_GetLocals(frame); + if (frame == NULL) + goto error_handler; -Rejected Ideas +Implementation ============== -[Why certain ideas that were brought while discussing this PEP were not ultimately pursued.] +Each read of ``frame.f_locals`` will create a new proxy object that gives +the appearance of being the mapping of local (including cell and free) +variable names to the values of those local variables. + +A possible implementation is sketched out below. +All attributes that start with an underscore are invisible and +cannot be accessed directly. +They serve only to illustrate the proposed design. + +:: + + NULL: Object # NULL is a singleton representing the absence of a value. + + class CodeType: + + _name_to_offset_mapping_impl: dict | NULL + _cells: frozenset # Set of indexes of cell and free variables + ... + + def __init__(self, ...): + self._name_to_offset_mapping_impl = NULL + ... + + @property + def _name_to_offset_mapping(self): + "Mapping of names to offsets in local variable array." + if self._name_to_offset_mapping_impl is NULL: + self._name_to_offset_mapping_impl = { + name: index for (index, name) in enumerate(self.co_varnames) + } + return self._name_to_offset_mapping_impl + + class FrameType: + + _locals : array[Object] # The values of the local variables, items may be NULL. + _extra_locals: dict | NULL # Dictionary for storing extra locals not in _locals. + + def __init__(self, ...): + self._extra_locals = NULL + ... + + @property + def f_locals(self): + return FrameLocalsProxy(self) + + class FrameLocalsProxy: + + __slots__ "_frame" + + def __init__(self, frame:FrameType): + self._frame = frame + + def __getitem__(self, name): + f = self._frame + co = f.f_code + if name in co._name_to_offset_mapping: + index = co._name_to_offset_mapping[name] + val = f._locals[index] + if val is NULL: + raise KeyError(name) + if index in co._cells + val = val.cell_contents + if val is NULL: + raise KeyError(name) + return val + else: + if f._extra_locals is NULL: + raise KeyError(name) + return f._extra_locals[name] + + def __setitem__(self, name, value): + f = self._frame + co = f.f_code + if name in co._name_to_offset_mapping: + index = co._name_to_offset_mapping[name] + kind = co._local_kinds[index] + if index in co._cells + cell = f._locals[index] + cell.cell_contents = val + else: + f._locals[index] = val + else: + if f._extra_locals is NULL: + f._extra_locals = {} + f._extra_locals[name] = val + + def __iter__(self): + f = self._frame + co = f.f_code + yield from iter(f._extra_locals) + for index, name in enumerate(co._varnames): + val = f._locals[index] + if val is NULL: + continue + if index in co._cells: + val = val.cell_contents + if val is NULL: + continue + yield name + + def pop(self): + f = self._frame + co = f.f_code + if f._extra_locals: + return f._extra_locals.pop() + for index, _ in enumerate(co._varnames): + val = f._locals[index] + if val is NULL: + continue + if index in co._cells: + cell = val + val = cell.cell_contents + if val is NULL: + continue + cell.cell_contents = NULL + else: + f._locals[index] = NULL + return val + + def __len__(self): + f = self._frame + co = f.f_code + res = 0 + for index, _ in enumerate(co._varnames): + val = f._locals[index] + if val is NULL: + continue + if index in co._cells: + if val.cell_contents is NULL: + continue + res += 1 + return len(self._extra_locals) + res + + +Comparison with PEP 558 +======================= + +This PEP and PEP 558 [2]_ share a common goal: +to make the semantics of ``locals()`` and ``frame.f_locals()`` +intelligible, and their operation reliable. +In the author's opinion, the proposed semantics of PEP 558 are too +complex to be used without constant reference to the documentation. +The proposed operation of PEP 558 has many corner cases, +that will lead to bugs. + +The key difference between this PEP and PEP 558 is that +PEP 558 makes an internal copy of the local variables. +This complicates both the semantics and implementation, +but offers no real advantage, in the authors opinion. + +The semantics of ``frame.f_locals`` +----------------------------------- +In this PEP, ``frame.f_locals`` is simple view onto the underlying frame. +It is always synchronized with the underlying frame. +In PEP 558, there is an additional cache present in the frame which is +updated whenever ``frame.f_locals`` is accessed. +PEP 558 does not make it clear whether calls to ``locals()`` +update ``frame.f_locals`` or not. + +For example consider:: + + def foo(): + x = sys._getframe().f_locals + y = locals() + +Does ``y`` contain ``"x"`` with PEP 558? +Does ``x`` contain ``"x"``? + +With this PEP, ``x`` is always up to date and +reflects the underlying local variables. Open Issues =========== -[Any points that are still being decided/discussed.] +An alternative way to define ``locals()`` would be simply as:: + def locals(): + return sys._getframe(0).f_locals + +This would be simpler and easier to understand. However, +there would be backwards compatibility issues when ``locals`` is assigned +to a local variable or when passed to ``eval``. References ========== .. [1] https://bugs.python.org/issue30744 +.. [2] https://www.python.org/dev/peps/pep-0558/ + Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. - - .. Local Variables: mode: indented-text From ce7955b308e929e3ca40728af7821df843b987bb Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 20 Aug 2021 11:57:14 +0100 Subject: [PATCH 05/10] Give namespace PEP the number 667. --- pep-06xx.rst => pep-0667.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename pep-06xx.rst => pep-0667.rst (99%) diff --git a/pep-06xx.rst b/pep-0667.rst similarity index 99% rename from pep-06xx.rst rename to pep-0667.rst index e687f867499..4e74950e262 100644 --- a/pep-06xx.rst +++ b/pep-0667.rst @@ -1,11 +1,11 @@ -PEP: 6xx +PEP: 677 Title: Consistent views of namespaces Author: Mark Shannon Status: Draft Type: Standards Content-Type: text/x-rst Created: 30-Jul-2021 -Post-History: XX-Aug-2021 +Post-History: 20-Aug-2021 Abstract From e88a199e25c009fd6cc4d9ae1e354889fc152954 Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 20 Aug 2021 14:11:51 +0100 Subject: [PATCH 06/10] More edits --- pep-0667.rst | 64 ++++++++++++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/pep-0667.rst b/pep-0667.rst index 4e74950e262..63b21d5b626 100644 --- a/pep-0667.rst +++ b/pep-0667.rst @@ -11,8 +11,8 @@ Post-History: 20-Aug-2021 Abstract ======== -In early versions of Python all namespaces, whether in functions, -classes or modules, were all implemented the same way; as a dictionary. +In early versions of Python all namespaces, whether in functions, +classes or modules, were all implemented the same way: as a dictionary. For performance reasons, the implementation of function namespaces was changed. Unfortunately this meant that accessing these namespaces through @@ -20,8 +20,8 @@ changed. Unfortunately this meant that accessing these namespaces through odd bugs crept in over the years as threads, generators and coroutines were added. -This PEP proposes make these namespaces consistent once more. -Modifications to ``frame.f_locals`` will always be visible in +This PEP proposes making these namespaces consistent once more. +Modifications to ``frame.f_locals`` will always be visible in the underlying variables. Modifications to local variables will immediately be visible in ``frame.f_locals``, and they will be consistent regardless of threading or coroutines. @@ -68,14 +68,14 @@ Rationale ========= The current implementation of ``frame.f_locals`` returns a dictionary -that is created on the fly from the array of local variables. +that is created on the fly from the array of local variables. This can result in the array and dictionary getting out of sync with -each other. Writes to the ``locals()`` may not show up as +each other. Writes to the ``f_locals`` may not show up as modifications to local variables. Writes to local variables can get lost. By making ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``frame.f_locals`` is always in -sync with the frame, because it is a view of it, not a copy of it. +sync with the frame because it is a view of it, not a copy of it. Specification ============= @@ -93,7 +93,7 @@ for function scopes it will be a custom class. def locals(): f_locals = sys._getframe().f_locals - if not isinstance(dict): + if not isinstance(f_locals, dict): f_locals = dict(f_locals) return f_locals @@ -138,14 +138,6 @@ Two new C-API functions will be added:: Both functions will return a new reference. -The existing C-API function ``PyEval_GetLocals()`` will always raise an -exception with a message like:: - - PyEval_GetLocals() is unsafe. Please use PyEval_Locals() instead. - -This is necessary as ``PyEval_GetLocals()`` -returns a borrowed reference which cannot be made safe. - Changes to existing APIs '''''''''''''''''''''''' @@ -167,8 +159,8 @@ Behavior of f_locals for optimized functions -------------------------------------------- Although ``f.f_locals`` behaves as if it were the namespace of the function, -some differences will be observable, most notably that -``f.f_locals is f.f_locals`` may be ``False``. +there will be some observable differences. +For example, ``f.f_locals is f.f_locals`` may be ``False``. However ``f.f_locals == f.f_locals`` will be ``True``, and all changes to the underlying variables, by any means, will be @@ -193,7 +185,6 @@ C-API PyEval_GetLocals '''''''''''''''' -The change to ``PyEval_GetLocals()`` is a backwards compatibility break. Code that uses ``PyEval_GetLocals()`` will continue to operate safely, but will need to be changed to use ``PyEval_Locals()`` to restore functionality. @@ -218,7 +209,7 @@ PyFrame_FastToLocals, etc. These functions were designed to convert the internal "fast" representation of the locals variables of a function to a dictionary, and vice versa. -They are no longer required. C code that directly accesses the `f_locals` +Calls to them are no longer required. C code that directly accesses the `f_locals` field of a frame should be modified to call ``PyFrame_GetLocals()`` instead:: PyFrame_FastToLocals(frame); @@ -374,23 +365,23 @@ Comparison with PEP 558 This PEP and PEP 558 [2]_ share a common goal: to make the semantics of ``locals()`` and ``frame.f_locals()`` intelligible, and their operation reliable. -In the author's opinion, the proposed semantics of PEP 558 are too -complex to be used without constant reference to the documentation. -The proposed operation of PEP 558 has many corner cases, -that will lead to bugs. + +In the author's opinion, PEP 558 fails to do that as it is too +complex, and has many corner cases which will lead to bugs. The key difference between this PEP and PEP 558 is that -PEP 558 makes an internal copy of the local variables. -This complicates both the semantics and implementation, -but offers no real advantage, in the authors opinion. +PEP 558 requires an internal copy of the local variables, +whereas this PEP does not. +Maintaining a copy would add considerably to the complexity of both +the specification and implementation, and bring no real benefits. The semantics of ``frame.f_locals`` ----------------------------------- -In this PEP, ``frame.f_locals`` is simple view onto the underlying frame. +In this PEP, ``frame.f_locals`` is a view onto the underlying frame. It is always synchronized with the underlying frame. -In PEP 558, there is an additional cache present in the frame which is -updated whenever ``frame.f_locals`` is accessed. +In PEP 558, there is an additional copy of the local variables present +in the frame which is updated whenever ``frame.f_locals`` is accessed. PEP 558 does not make it clear whether calls to ``locals()`` update ``frame.f_locals`` or not. @@ -399,12 +390,17 @@ For example consider:: def foo(): x = sys._getframe().f_locals y = locals() + print(tuple(x)) + print(tuple(y)) + +It is not clear from PEP 558 (at time of writing) what would be printed. +Does the call to ``locals()`` update ``x``? +Would ``"y"`` be present in either ``x`` or ``y``? -Does ``y`` contain ``"x"`` with PEP 558? -Does ``x`` contain ``"x"``? +With this PEP it should be clear that the above would print:: -With this PEP, ``x`` is always up to date and -reflects the underlying local variables. + ('x', 'y') + ('x',) Open Issues =========== From 0d759f1d85b3a4142eea25b496e8d2315d68bb67 Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 20 Aug 2021 14:13:29 +0100 Subject: [PATCH 07/10] Correct number. --- pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0667.rst b/pep-0667.rst index 63b21d5b626..707ed9889ed 100644 --- a/pep-0667.rst +++ b/pep-0667.rst @@ -1,4 +1,4 @@ -PEP: 677 +PEP: 667 Title: Consistent views of namespaces Author: Mark Shannon Status: Draft From e0057b571be8c936d4a71157d48724fd30ff7ced Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 20 Aug 2021 14:16:46 +0100 Subject: [PATCH 08/10] Fix type. --- pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0667.rst b/pep-0667.rst index 707ed9889ed..ae77a4037b3 100644 --- a/pep-0667.rst +++ b/pep-0667.rst @@ -2,7 +2,7 @@ PEP: 667 Title: Consistent views of namespaces Author: Mark Shannon Status: Draft -Type: Standards +Type: Standards Track Content-Type: text/x-rst Created: 30-Jul-2021 Post-History: 20-Aug-2021 From c56ace24df14ce273b6896dcd9dd74d5630437fd Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 20 Aug 2021 14:30:20 +0100 Subject: [PATCH 09/10] Fix definitions of locals() --- pep-0667.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0667.rst b/pep-0667.rst index ae77a4037b3..f4eba660782 100644 --- a/pep-0667.rst +++ b/pep-0667.rst @@ -92,7 +92,7 @@ for function scopes it will be a custom class. ``locals()`` will be defined as:: def locals(): - f_locals = sys._getframe().f_locals + f_locals = sys._getframe(1).f_locals if not isinstance(f_locals, dict): f_locals = dict(f_locals) return f_locals @@ -408,7 +408,7 @@ Open Issues An alternative way to define ``locals()`` would be simply as:: def locals(): - return sys._getframe(0).f_locals + return sys._getframe(1).f_locals This would be simpler and easier to understand. However, there would be backwards compatibility issues when ``locals`` is assigned From 15622193521a1b36b3d09603496e13a72d27a6c0 Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 20 Aug 2021 14:33:11 +0100 Subject: [PATCH 10/10] Fix backticks --- pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0667.rst b/pep-0667.rst index f4eba660782..ec3afc78ff2 100644 --- a/pep-0667.rst +++ b/pep-0667.rst @@ -209,7 +209,7 @@ PyFrame_FastToLocals, etc. These functions were designed to convert the internal "fast" representation of the locals variables of a function to a dictionary, and vice versa. -Calls to them are no longer required. C code that directly accesses the `f_locals` +Calls to them are no longer required. C code that directly accesses the ``f_locals`` field of a frame should be modified to call ``PyFrame_GetLocals()`` instead:: PyFrame_FastToLocals(frame);