From a6c7693c1293ae52bae6e7165aa58dd2dca9cc17 Mon Sep 17 00:00:00 2001 From: David Foster Date: Sat, 26 Feb 2022 11:48:45 -0800 Subject: [PATCH 1/4] PEP 638: Typographic fixes. Leave many comments. --- pep-0638.rst | 116 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 104 insertions(+), 12 deletions(-) diff --git a/pep-0638.rst b/pep-0638.rst index 710303462cc..e6a398799ac 100644 --- a/pep-0638.rst +++ b/pep-0638.rst @@ -6,6 +6,9 @@ Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2020 +.. + What is the Discussions-To for this PEP? python-dev? + Abstract ======== @@ -147,6 +150,28 @@ Statement form macro_stmt = MACRO_NAME testlist [ "import" NAME ] [ "as" NAME ] [ ":" NEWLINE suite ] +.. + It is difficult for a newcomer to understand the permutations that + this grammar rule makes possible. + + Perhaps it would be useful to provide illustrative examples of + the syntax here, or link to examples later in the PEP (such as the + §"Examples" section)? + +.. + What is `testlist`? It is not defined in this PEP, nor is it defined + in the [Python grammar specification](https://docs.python.org/3/reference/grammar.html). + +.. + - The `[ "import" NAME ]` part appears to only be useful for supporting + the `from!` macro. + - The `[ "as" NAME ]` part appears to only be useful for supporting + the example `with! open(filename) as fd:` macro. + + It feels odd specifying such special-purpose syntax directly in the grammar... + Consider making the grammar more general with something like: + MACRO_NAME ( expression | )* ( NEWLINE | ":" NEWLINE suite ) + Expression form ~~~~~~~~~~~~~~~ @@ -154,6 +179,13 @@ Expression form macro_expr = MACRO_NAME "(" testlist ")" +.. + Again, what is `testlist`? And is it really the same as the `testlist` + used in §"Statement form"? + + Perhaps: + ( expression ( "," expression )* ","? )? + Resolving ambiguity ~~~~~~~~~~~~~~~~~~~ @@ -161,6 +193,12 @@ The statement form of a macro takes precedence, so that the code ``macro_name!(x)`` will be parsed as a macro statement, not as an expression statement containing a macro expression. +.. + Wouldn't it be desirable to be able to define an expression macro processor + that could be invoked properly even if the invoking expression did occur + on a line by itself (and thus would be interpreted as a macro statement + by this rule)? + Semantics ''''''''' @@ -169,7 +207,7 @@ Compilation Upon encountering a ``macro`` during translation to bytecode, the code generator will look up the macro processor registered for the macro, -and pass the AST, rooted at the macro to the processor function. +and pass the AST rooted at the macro to the processor function. The returned AST will then be substituted for the original tree. For macros with multiple names, @@ -177,6 +215,11 @@ several trees will be passed to the macro processor, but only one will be returned and substituted, shorting the enclosing block of statements. +.. + "Multiple names"? Oh. You mean macros with "additional names", + as described later in the PEP. As I comment later in this PEP, I think + it would clearer to use a term like "additional clause names". + This process can be repeated, to enable macros to return AST nodes including other macros. @@ -208,6 +251,13 @@ Note that, since ``import!`` and ``from!`` only define the macro for the scope in which the import is present, all uses of a macro must be preceded by an explicit ``import!`` or ``from!`` to improve clarity. +.. + That is, all macros (other than `import!` and `from!`) must be imported + (using `import!` or `from!`) before they are used. + + The original wording - expecially the "to improve clarity" part - isn't + clear to me exactly what it is trying to stress. + For example, to import the macro "compile" from "my.compiler": :: @@ -219,18 +269,30 @@ Defining macro processors ~~~~~~~~~~~~~~~~~~~~~~~~~ A macro processor is defined by a four-tuple, consisting of -``(func, kind, version, additional_names)`` +``(func, kind, version, additional_names)``: * ``func`` must be a callable that takes ``len(additional_names)+1`` arguments, all of which are abstract syntax trees, and returns a single abstract syntax tree. * ``kind`` must be one of the following: - * ``macros.STMT_MACRO`` A statement macro where the body of the macro is indented. This is the only form allowed to have additional names. - * ``macros.SIBLING_MACRO`` A statement macro where the body of the macro is the next statement is the same block. The following statement is moved into the macro as its body. - * ``macros.EXPR_MACRO`` An expression macro. + * ``macros.STMT_MACRO``: A statement macro where the body of the macro is indented. This is the only form allowed to have additional names. + * ``macros.SIBLING_MACRO``: A statement macro where the body of the macro is the next statement in the same block. The following statement is moved into the macro as its body. + * ``macros.EXPR_MACRO``: An expression macro. * ``version`` is used to track versions of macros, so that generated bytecodes can be correctly cached. It must be an integer. * ``additional_names`` are the names of the additional parts of the macro, and must be a tuple of strings. +.. + A statement macro that has a `testlist`, an `"import" NAME` part, + an `"as" NAME` part, or a (proposed) `( expression | )*` part + will need to take those as arguments as well. + + Taking only ``len(additional_names)+1`` arguments, which cover clauses only, + will not be enough. + +.. + Seems to me that "additional_names" might make more sense to call + "additional clauses" or "additional clause names". + :: # (func, _ast.STMT_MACRO, VERSION, ()) @@ -278,22 +340,19 @@ Two new AST nodes will be needed to express macros, ``macro_stmt`` and ``macro_e :: class macro_stmt(_ast.stmt): - _fields = "name", "args", "importname", "asname", "body" class macro_expr(_ast.expr): - _fields = "name", "args" -In addition, macro processors will needs a means to express control flow or side-effecting code, that produces a value. -To support this, a new ast node, called ``stmt_expr``, that combines a statement and expression will be added. +In addition, macro processors will need a means to express control flow or side-effecting code, that produces a value. +To support this, a new ast node will be added called ``stmt_expr`` that combines a statement and an expression. This new ast node will be a subtype of ``expr``, but include a statement to allow side effects. It will be compiled to bytecode by compiling the statement, then compiling the value. :: class stmt_expr(_ast.expr): - _fields = "stmt", "value" Hygiene and debugging @@ -303,11 +362,20 @@ Macro processors will often need to create new variables. Those variables need to named in such as way as to avoid contaminating the original code and other macros. No rules for naming will be enforced, but to ensure hygiene and help debugging, the following naming scheme is recommended: +.. + This appears to imply that it is up to the macro processor author to + take special care that their macro is hygenic. That is, *unhygenic* macros + are the default. Wouldn't it be safer to define a system where *hygenic* + macros would be the default instead? + * All generated variable names should start with a ``$`` * Purely artificial variable names should start ``$$mname`` where ``mname`` is the name of the macro. * Variables derived from real variables should start ``$vname`` where ``vname`` is the name of the variable. * All variable names should include the line number and the column offset, separated by an underscore. +.. + How will the line number and column offset be obtained by the macro processor? + Examples: * Purely generated name: ``$$macro_17_0`` @@ -317,6 +385,12 @@ Examples: Examples '''''''' +.. + Not a single example in this section defines an actual macro processor + function. Recommend implementing at least a toy macro end-to-end + (including the macro processor function) for full clarity, + for each type of macro (i.e. statements, sibling, and expression). + Compile-time-checked data structures ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -450,8 +524,8 @@ can be replaced with the zero-cost macro: def foo(...): ... -Protyping language extensions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Prototyping language extensions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Although macros would be most valuable for domain-specific extensions, it is possible to demonstrate possible language extensions using macros. @@ -526,6 +600,12 @@ Backwards Compatibility This PEP is fully backwards compatible. +.. + In §"Implementation" below, it is mentioned that nodes in the + ``_ast`` module would be made immutable. + + That sounds like a backward-incompatible change to me. + Performance Implications ======================== @@ -561,11 +641,23 @@ Currently, all AST nodes are allocated using an arena allocator. Changing to use the standard allocator might slow compilation down a little, but has advantages in terms of maintenance, as much code can be deleted. +.. + I presume the arena allocator was introduced in the first place for a reason. + Perhaps to improve performance? By removing the arena allocator are there + potential downsides other than a performance regression? + Reference Implementation '''''''''''''''''''''''' None as yet. +.. + Seems like you could get a prototype off the ground by implementing an + initial version as a fake Python source file text encoding. + + Then you could put something like `# coding=macros` at the top of + a source file to have it preprocessed by a prototype macro system. + Copyright ========= From 48abe0872eb6747f871fedae79cfe14668dfa93d Mon Sep 17 00:00:00 2001 From: David Foster Date: Sun, 27 Feb 2022 06:42:30 -0800 Subject: [PATCH 2/4] Remove inline comments. --- pep-0638.rst | 95 ---------------------------------------------------- 1 file changed, 95 deletions(-) diff --git a/pep-0638.rst b/pep-0638.rst index e6a398799ac..52952e0a077 100644 --- a/pep-0638.rst +++ b/pep-0638.rst @@ -6,9 +6,6 @@ Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2020 -.. - What is the Discussions-To for this PEP? python-dev? - Abstract ======== @@ -150,28 +147,6 @@ Statement form macro_stmt = MACRO_NAME testlist [ "import" NAME ] [ "as" NAME ] [ ":" NEWLINE suite ] -.. - It is difficult for a newcomer to understand the permutations that - this grammar rule makes possible. - - Perhaps it would be useful to provide illustrative examples of - the syntax here, or link to examples later in the PEP (such as the - §"Examples" section)? - -.. - What is `testlist`? It is not defined in this PEP, nor is it defined - in the [Python grammar specification](https://docs.python.org/3/reference/grammar.html). - -.. - - The `[ "import" NAME ]` part appears to only be useful for supporting - the `from!` macro. - - The `[ "as" NAME ]` part appears to only be useful for supporting - the example `with! open(filename) as fd:` macro. - - It feels odd specifying such special-purpose syntax directly in the grammar... - Consider making the grammar more general with something like: - MACRO_NAME ( expression | )* ( NEWLINE | ":" NEWLINE suite ) - Expression form ~~~~~~~~~~~~~~~ @@ -179,13 +154,6 @@ Expression form macro_expr = MACRO_NAME "(" testlist ")" -.. - Again, what is `testlist`? And is it really the same as the `testlist` - used in §"Statement form"? - - Perhaps: - ( expression ( "," expression )* ","? )? - Resolving ambiguity ~~~~~~~~~~~~~~~~~~~ @@ -193,12 +161,6 @@ The statement form of a macro takes precedence, so that the code ``macro_name!(x)`` will be parsed as a macro statement, not as an expression statement containing a macro expression. -.. - Wouldn't it be desirable to be able to define an expression macro processor - that could be invoked properly even if the invoking expression did occur - on a line by itself (and thus would be interpreted as a macro statement - by this rule)? - Semantics ''''''''' @@ -215,11 +177,6 @@ several trees will be passed to the macro processor, but only one will be returned and substituted, shorting the enclosing block of statements. -.. - "Multiple names"? Oh. You mean macros with "additional names", - as described later in the PEP. As I comment later in this PEP, I think - it would clearer to use a term like "additional clause names". - This process can be repeated, to enable macros to return AST nodes including other macros. @@ -251,13 +208,6 @@ Note that, since ``import!`` and ``from!`` only define the macro for the scope in which the import is present, all uses of a macro must be preceded by an explicit ``import!`` or ``from!`` to improve clarity. -.. - That is, all macros (other than `import!` and `from!`) must be imported - (using `import!` or `from!`) before they are used. - - The original wording - expecially the "to improve clarity" part - isn't - clear to me exactly what it is trying to stress. - For example, to import the macro "compile" from "my.compiler": :: @@ -281,18 +231,6 @@ A macro processor is defined by a four-tuple, consisting of * ``version`` is used to track versions of macros, so that generated bytecodes can be correctly cached. It must be an integer. * ``additional_names`` are the names of the additional parts of the macro, and must be a tuple of strings. -.. - A statement macro that has a `testlist`, an `"import" NAME` part, - an `"as" NAME` part, or a (proposed) `( expression | )*` part - will need to take those as arguments as well. - - Taking only ``len(additional_names)+1`` arguments, which cover clauses only, - will not be enough. - -.. - Seems to me that "additional_names" might make more sense to call - "additional clauses" or "additional clause names". - :: # (func, _ast.STMT_MACRO, VERSION, ()) @@ -362,20 +300,11 @@ Macro processors will often need to create new variables. Those variables need to named in such as way as to avoid contaminating the original code and other macros. No rules for naming will be enforced, but to ensure hygiene and help debugging, the following naming scheme is recommended: -.. - This appears to imply that it is up to the macro processor author to - take special care that their macro is hygenic. That is, *unhygenic* macros - are the default. Wouldn't it be safer to define a system where *hygenic* - macros would be the default instead? - * All generated variable names should start with a ``$`` * Purely artificial variable names should start ``$$mname`` where ``mname`` is the name of the macro. * Variables derived from real variables should start ``$vname`` where ``vname`` is the name of the variable. * All variable names should include the line number and the column offset, separated by an underscore. -.. - How will the line number and column offset be obtained by the macro processor? - Examples: * Purely generated name: ``$$macro_17_0`` @@ -385,12 +314,6 @@ Examples: Examples '''''''' -.. - Not a single example in this section defines an actual macro processor - function. Recommend implementing at least a toy macro end-to-end - (including the macro processor function) for full clarity, - for each type of macro (i.e. statements, sibling, and expression). - Compile-time-checked data structures ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -600,12 +523,6 @@ Backwards Compatibility This PEP is fully backwards compatible. -.. - In §"Implementation" below, it is mentioned that nodes in the - ``_ast`` module would be made immutable. - - That sounds like a backward-incompatible change to me. - Performance Implications ======================== @@ -641,23 +558,11 @@ Currently, all AST nodes are allocated using an arena allocator. Changing to use the standard allocator might slow compilation down a little, but has advantages in terms of maintenance, as much code can be deleted. -.. - I presume the arena allocator was introduced in the first place for a reason. - Perhaps to improve performance? By removing the arena allocator are there - potential downsides other than a performance regression? - Reference Implementation '''''''''''''''''''''''' None as yet. -.. - Seems like you could get a prototype off the ground by implementing an - initial version as a fake Python source file text encoding. - - Then you could put something like `# coding=macros` at the top of - a source file to have it preprocessed by a prototype macro system. - Copyright ========= From b472b06f2905fd5800c864c73d721559a1b16bdd Mon Sep 17 00:00:00 2001 From: David Foster Date: Sun, 27 Feb 2022 06:47:25 -0800 Subject: [PATCH 3/4] Add initial Discussions-To and Post-History. --- pep-0638.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pep-0638.rst b/pep-0638.rst index 52952e0a077..c9613809ac1 100644 --- a/pep-0638.rst +++ b/pep-0638.rst @@ -1,10 +1,12 @@ PEP: 638 Title: Syntactic Macros Author: Mark Shannon +Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/U4C4XHNRC4SHS3TPZWCTY4SN4QU3TT6V/ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2020 +Post-History: 26-Sep-2020 Abstract ======== From 0cb64d27a5a401142c4557409f2e6b68e993ee48 Mon Sep 17 00:00:00 2001 From: David Foster Date: Mon, 7 Mar 2022 21:22:13 -0800 Subject: [PATCH 4/4] Adjust bulleted list item Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- pep-0638.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0638.rst b/pep-0638.rst index c9613809ac1..7498d3200e5 100644 --- a/pep-0638.rst +++ b/pep-0638.rst @@ -286,7 +286,7 @@ Two new AST nodes will be needed to express macros, ``macro_stmt`` and ``macro_e _fields = "name", "args" In addition, macro processors will need a means to express control flow or side-effecting code, that produces a value. -To support this, a new ast node will be added called ``stmt_expr`` that combines a statement and an expression. +A new AST node called ``stmt_expr`` will be added, combining a statement and an expression. This new ast node will be a subtype of ``expr``, but include a statement to allow side effects. It will be compiled to bytecode by compiling the statement, then compiling the value.