From 8244e980d3f3714eae93dd406f3002d007c33f6e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?B=C3=A9n=C3=A9dikt=20Tran?= <10796600+picnixz@users.noreply.github.com> Date: Sun, 1 Dec 2024 17:05:02 +0100 Subject: [PATCH 1/2] Format internal docs and fix internal links. --- InternalDocs/README.md | 1 - InternalDocs/adaptive.md | 8 +- InternalDocs/changing_grammar.md | 6 +- InternalDocs/compiler.md | 221 ++++++++++++++--------------- InternalDocs/exception_handling.md | 67 ++++----- InternalDocs/frames.md | 5 +- InternalDocs/garbage_collector.md | 59 ++++---- InternalDocs/generators.md | 1 - InternalDocs/interpreter.md | 33 +++-- InternalDocs/parser.md | 107 +++++++------- InternalDocs/string_interning.md | 3 + 11 files changed, 258 insertions(+), 253 deletions(-) diff --git a/InternalDocs/README.md b/InternalDocs/README.md index f6aa3db3b384af..8cdd06d189f362 100644 --- a/InternalDocs/README.md +++ b/InternalDocs/README.md @@ -1,4 +1,3 @@ - # CPython Internals Documentation The documentation in this folder is intended for CPython maintainers. diff --git a/InternalDocs/adaptive.md b/InternalDocs/adaptive.md index 4ae9e85b387f39..7cfa8e52310460 100644 --- a/InternalDocs/adaptive.md +++ b/InternalDocs/adaptive.md @@ -96,6 +96,7 @@ quality of specialization and keeping the overhead of specialization low. Specialized instructions must be fast. In order to be fast, specialized instructions should be tailored for a particular set of values that allows them to: + 1. Verify that incoming value is part of that set with low overhead. 2. Perform the operation quickly. @@ -107,9 +108,11 @@ For example, `LOAD_GLOBAL_MODULE` is specialized for `globals()` dictionaries that have a keys with the expected version. This can be tested quickly: + * `globals->keys->dk_version == expected_version` and the operation can be performed quickly: + * `value = entries[cache->index].me_value;`. Because it is impossible to measure the performance of an instruction without @@ -122,10 +125,11 @@ base instruction. ### Implementation of specialized instructions In general, specialized instructions should be implemented in two parts: + 1. A sequence of guards, each of the form - `DEOPT_IF(guard-condition-is-false, BASE_NAME)`. + `DEOPT_IF(guard-condition-is-false, BASE_NAME)`. 2. The operation, which should ideally have no branches and - a minimum number of dependent memory accesses. + a minimum number of dependent memory accesses. In practice, the parts may overlap, as data required for guards can be re-used in the operation. diff --git a/InternalDocs/changing_grammar.md b/InternalDocs/changing_grammar.md index 1a5eebdc1418dc..3851766e314ced 100644 --- a/InternalDocs/changing_grammar.md +++ b/InternalDocs/changing_grammar.md @@ -18,7 +18,7 @@ Below is a checklist of things that may need to change. to regenerate [`Parser/parser.c`](../Parser/parser.c). (This runs Python's parser generator, [`Tools/peg_generator`](../Tools/peg_generator)). -* [`Grammar/Tokens`](../Grammar/Tokens) is a place for adding new token types. After +* [`Grammar/Tokens`](../Grammar/Tokens) is a place for adding new token types. After changing it, run ``make regen-token`` to regenerate [`Include/internal/pycore_token.h`](../Include/internal/pycore_token.h), [`Parser/token.c`](../Parser/token.c), [`Lib/token.py`](../Lib/token.py) @@ -32,7 +32,7 @@ Below is a checklist of things that may need to change. [`Include/internal/pycore_ast.h`](../Include/internal/pycore_ast.h) and [`Python/Python-ast.c`](../Python/Python-ast.c). -* [`Parser/lexer/`](../Parser/lexer/) contains the tokenization code. +* [`Parser/lexer/`](../Parser/lexer) contains the tokenization code. This is where you would add a new type of comment or string literal, for example. * [`Python/ast.c`](../Python/ast.c) will need changes to validate AST objects @@ -60,4 +60,4 @@ Below is a checklist of things that may need to change. to the tokenizer. * Documentation must be written! Specifically, one or more of the pages in - [`Doc/reference/`](../Doc/reference/) will need to be updated. + [`Doc/reference/`](../Doc/reference) will need to be updated. diff --git a/InternalDocs/compiler.md b/InternalDocs/compiler.md index ed4cfb23ca51f7..1a355bd96d228b 100644 --- a/InternalDocs/compiler.md +++ b/InternalDocs/compiler.md @@ -1,4 +1,3 @@ - Compiler design =============== @@ -7,8 +6,8 @@ Abstract In CPython, the compilation from source code to bytecode involves several steps: -1. Tokenize the source code [Parser/lexer/](../Parser/lexer/) - and [Parser/tokenizer/](../Parser/tokenizer/). +1. Tokenize the source code [Parser/lexer/](../Parser/lexer) + and [Parser/tokenizer/](../Parser/tokenizer). 2. Parse the stream of tokens into an Abstract Syntax Tree [Parser/parser.c](../Parser/parser.c). 3. Transform AST into an instruction sequence @@ -21,8 +20,8 @@ In CPython, the compilation from source code to bytecode involves several steps: This document outlines how these steps of the process work. This document only describes parsing in enough depth to explain what is needed -for understanding compilation. This document provides a detailed, though not -exhaustive, view of the how the entire system works. You will most likely need +for understanding compilation. This document provides a detailed, though not +exhaustive, view of the how the entire system works. You will most likely need to read some source code to have an exact understanding of all details. @@ -37,7 +36,7 @@ parsers. The grammar file for Python can be found in [Grammar/python.gram](../Grammar/python.gram). The definitions for literal tokens (such as `:`, numbers, etc.) can be found in -[Grammar/Tokens](../Grammar/Tokens). Various C files, including +[Grammar/Tokens](../Grammar/Tokens). Various C files, including [Parser/parser.c](../Parser/parser.c) are generated from these. See Also: @@ -55,7 +54,7 @@ Abstract syntax trees (AST) The abstract syntax tree (AST) is a high-level representation of the program structure without the necessity of containing the source code; -it can be thought of as an abstract representation of the source code. The +it can be thought of as an abstract representation of the source code. The specification of the AST nodes is specified using the Zephyr Abstract Syntax Definition Language (ASDL) [^1], [^2]. @@ -64,9 +63,9 @@ The definition of the AST nodes for Python is found in the file Each AST node (representing statements, expressions, and several specialized types, like list comprehensions and exception handlers) is -defined by the ASDL. Most definitions in the AST correspond to a +defined by the ASDL. Most definitions in the AST correspond to a particular source construct, such as an 'if' statement or an attribute -lookup. The definition is independent of its realization in any +lookup. The definition is independent of its realization in any particular programming language. The following fragment of the Python ASDL construct demonstrates the @@ -85,7 +84,7 @@ approach and syntax: The preceding example describes two different kinds of statements and an expression: function definitions, return statements, and yield expressions. All three kinds are considered of type `stmt` as shown by `|` separating -the various kinds. They all take arguments of various kinds and amounts. +the various kinds. They all take arguments of various kinds and amounts. Modifiers on the argument type specify the number of values needed; `?` means it is optional, `*` means 0 or more, while no modifier means only one @@ -129,47 +128,47 @@ The statement definitions above generate the following C structure type: ``` Also generated are a series of constructor functions that allocate (in -this case) a `stmt_ty` struct with the appropriate initialization. The -`kind` field specifies which component of the union is initialized. The +this case) a `stmt_ty` struct with the appropriate initialization. The +`kind` field specifies which component of the union is initialized. The `FunctionDef()` constructor function sets 'kind' to `FunctionDef_kind` and initializes the *name*, *args*, *body*, and *attributes* fields. See also [Green Tree Snakes - The missing Python AST docs](https://greentreesnakes.readthedocs.io/en/latest) - by Thomas Kluyver. +by Thomas Kluyver. Memory management ================= Before discussing the actual implementation of the compiler, a discussion of -how memory is handled is in order. To make memory management simple, an **arena** +how memory is handled is in order. To make memory management simple, an **arena** is used that pools memory in a single location for easy -allocation and removal. This enables the removal of explicit memory -deallocation. Because memory allocation for all needed memory in the compiler +allocation and removal. This enables the removal of explicit memory +deallocation. Because memory allocation for all needed memory in the compiler registers that memory with the arena, a single call to free the arena is all that is needed to completely free all memory used by the compiler. In general, unless you are working on the critical core of the compiler, memory -management can be completely ignored. But if you are working at either the +management can be completely ignored. But if you are working at either the very beginning of the compiler or the end, you need to care about how the arena -works. All code relating to the arena is in either +works. All code relating to the arena is in either [Include/internal/pycore_pyarena.h](../Include/internal/pycore_pyarena.h) or [Python/pyarena.c](../Python/pyarena.c). -`PyArena_New()` will create a new arena. The returned `PyArena` structure -will store pointers to all memory given to it. This does the bookkeeping of +`PyArena_New()` will create a new arena. The returned `PyArena` structure +will store pointers to all memory given to it. This does the bookkeeping of what memory needs to be freed when the compiler is finished with the memory it -used. That freeing is done with `PyArena_Free()`. This only needs to be +used. That freeing is done with `PyArena_Free()`. This only needs to be called in strategic areas where the compiler exits. As stated above, in general you should not have to worry about memory -management when working on the compiler. The technical details of memory +management when working on the compiler. The technical details of memory management have been designed to be hidden from you for most cases. -The only exception comes about when managing a PyObject. Since the rest +The only exception comes about when managing a PyObject. Since the rest of Python uses reference counting, there is extra support added -to the arena to cleanup each PyObject that was allocated. These cases -are very rare. However, if you've allocated a PyObject, you must tell +to the arena to cleanup each PyObject that was allocated. These cases +are very rare. However, if you've allocated a PyObject, you must tell the arena about it by calling `PyArena_AddPyObject()`. @@ -183,22 +182,22 @@ The AST is generated from source code using the function After some checks, a helper function in [Parser/parser.c](../Parser/parser.c) begins applying production rules on the source code it receives; converting source -code to tokens and matching these tokens recursively to their corresponding rule. The -production rule's corresponding rule function is called on every match. These rule -functions follow the format `xx_rule`. Where *xx* is the grammar rule +code to tokens and matching these tokens recursively to their corresponding rule. The +production rule's corresponding rule function is called on every match. These rule +functions follow the format `xx_rule`. Where *xx* is the grammar rule that the function handles and is automatically derived from [Grammar/python.gram](../Grammar/python.gram) by [Tools/peg_generator/pegen/c_generator.py](../Tools/peg_generator/pegen/c_generator.py). -Each rule function in turn creates an AST node as it goes along. It does this +Each rule function in turn creates an AST node as it goes along. It does this by allocating all the new nodes it needs, calling the proper AST node creation functions for any required supporting functions and connecting them as needed. -This continues until all nonterminal symbols are replaced with terminals. If an -error occurs, the rule functions backtrack and try another rule function. If +This continues until all nonterminal symbols are replaced with terminals. If an +error occurs, the rule functions backtrack and try another rule function. If there are no more rules, an error is set and the parsing ends. The AST node creation helper functions have the name `_PyAST_{xx}` -where *xx* is the AST node that the function creates. These are defined by the +where *xx* is the AST node that the function creates. These are defined by the ASDL grammar and contained in [Python/Python-ast.c](../Python/Python-ast.c) (which is generated by [Parser/asdl_c.py](../Parser/asdl_c.py) from [Parser/Python.asdl](../Parser/Python.asdl)). @@ -206,9 +205,9 @@ This all leads to a sequence of AST nodes stored in `asdl_seq` structs. To demonstrate everything explained so far, here's the rule function responsible for a simple named import statement such as -`import sys`. Note that error-checking and debugging code has been -omitted. Removed parts are represented by `...`. -Furthermore, some comments have been added for explanation. These comments +`import sys`. Note that error-checking and debugging code has been +omitted. Removed parts are represented by `...`. +Furthermore, some comments have been added for explanation. These comments may not be present in the actual code. @@ -249,23 +248,23 @@ may not be present in the actual code. To improve backtracking performance, some rules (chosen by applying a -`(memo)` flag in the grammar file) are memoized. Each rule function checks if +`(memo)` flag in the grammar file) are memoized. Each rule function checks if a memoized version exists and returns that if so, else it continues in the manner stated in the previous paragraphs. There are macros for creating and using `asdl_xx_seq *` types, where *xx* is -a type of the ASDL sequence. Three main types are defined -manually -- `generic`, `identifier` and `int`. These types are found in +a type of the ASDL sequence. Three main types are defined +manually -- `generic`, `identifier` and `int`. These types are found in [Python/asdl.c](../Python/asdl.c) and its corresponding header file [Include/internal/pycore_asdl.h](../Include/internal/pycore_asdl.h). Functions and macros for creating `asdl_xx_seq *` types are as follows: -`_Py_asdl_generic_seq_new(Py_ssize_t, PyArena *)` - Allocate memory for an `asdl_generic_seq` of the specified length -`_Py_asdl_identifier_seq_new(Py_ssize_t, PyArena *)` - Allocate memory for an `asdl_identifier_seq` of the specified length -`_Py_asdl_int_seq_new(Py_ssize_t, PyArena *)` - Allocate memory for an `asdl_int_seq` of the specified length +* `_Py_asdl_generic_seq_new(Py_ssize_t, PyArena *)`: + Allocate memory for an `asdl_generic_seq` of the specified length +* `_Py_asdl_identifier_seq_new(Py_ssize_t, PyArena *)`: + Allocate memory for an `asdl_identifier_seq` of the specified length +* `_Py_asdl_int_seq_new(Py_ssize_t, PyArena *)`: + Allocate memory for an `asdl_int_seq` of the specified length In addition to the three types mentioned above, some ASDL sequence types are automatically generated by [Parser/asdl_c.py](../Parser/asdl_c.py) and found in @@ -273,27 +272,27 @@ automatically generated by [Parser/asdl_c.py](../Parser/asdl_c.py) and found in Macros for using both manually defined and automatically generated ASDL sequence types are as follows: -`asdl_seq_GET(asdl_xx_seq *, int)` - Get item held at a specific position in an `asdl_xx_seq` -`asdl_seq_SET(asdl_xx_seq *, int, stmt_ty)` - Set a specific index in an `asdl_xx_seq` to the specified value +* `asdl_seq_GET(asdl_xx_seq *, int)`: + Get item held at a specific position in an `asdl_xx_seq` +* `asdl_seq_SET(asdl_xx_seq *, int, stmt_ty)`: + Set a specific index in an `asdl_xx_seq` to the specified value -Untyped counterparts exist for some of the typed macros. These are useful +Untyped counterparts exist for some of the typed macros. These are useful when a function needs to manipulate a generic ASDL sequence: -`asdl_seq_GET_UNTYPED(asdl_seq *, int)` - Get item held at a specific position in an `asdl_seq` -`asdl_seq_SET_UNTYPED(asdl_seq *, int, stmt_ty)` - Set a specific index in an `asdl_seq` to the specified value -`asdl_seq_LEN(asdl_seq *)` - Return the length of an `asdl_seq` or `asdl_xx_seq` +* `asdl_seq_GET_UNTYPED(asdl_seq *, int)`: + Get item held at a specific position in an `asdl_seq` +* `asdl_seq_SET_UNTYPED(asdl_seq *, int, stmt_ty)`: + Set a specific index in an `asdl_seq` to the specified value +* `asdl_seq_LEN(asdl_seq *)`: + Return the length of an `asdl_seq` or `asdl_xx_seq` Note that typed macros and functions are recommended over their untyped -counterparts. Typed macros carry out checks in debug mode and aid +counterparts. Typed macros carry out checks in debug mode and aid debugging errors caused by incorrectly casting from `void *`. If you are working with statements, you must also worry about keeping -track of what line number generated the statement. Currently the line +track of what line number generated the statement. Currently the line number is passed as the last parameter to each `stmt_ty` function. See also [PEP 617: New PEG parser for CPython](https://peps.python.org/pep-0617/). @@ -303,12 +302,12 @@ Control flow graphs =================== A **control flow graph** (often referenced by its acronym, **CFG**) is a -directed graph that models the flow of a program. A node of a CFG is +directed graph that models the flow of a program. A node of a CFG is not an individual bytecode instruction, but instead represents a sequence of bytecode instructions that always execute sequentially. Each node is called a *basic block* and must always execute from start to finish, with a single entry point at the beginning and a -single exit point at the end. If some bytecode instruction *a* needs +single exit point at the end. If some bytecode instruction *a* needs to jump to some other bytecode instruction *b*, then *a* must occur at the end of its basic block, and *b* must occur at the start of its basic block. @@ -326,8 +325,8 @@ end() The `x < 10` guard is represented by its own basic block that compares `x` with `10` and then ends in a conditional jump based on -the result of the comparison. This conditional jump allows the block -to point to both the body of the `if` and the body of the `else`. The +the result of the comparison. This conditional jump allows the block +to point to both the body of the `if` and the body of the `else`. The `if` basic block contains the `f1()` and `f2()` calls and points to the `end()` basic block. The `else` basic block contains the `g()` call and similarly points to the `end()` block. @@ -352,7 +351,7 @@ The first step is to construct the symbol table. This is implemented by `_PySymtable_Build()` in [Python/symtable.c](../Python/symtable.c). This function begins by entering the starting code block for the AST (passed-in) and then calling the proper `symtable_visit_{xx}` function (with *xx* being the -AST node type). Next, the AST tree is walked with the various code blocks that +AST node type). Next, the AST tree is walked with the various code blocks that delineate the reach of a local variable as blocks are entered and exited using `symtable_enter_block()` and `symtable_exit_block()`, respectively. @@ -362,10 +361,10 @@ These are similar to bytecode, but in some cases they are more abstract, and are resolved later into actual bytecode. The construction of this instruction sequence is handled by several functions that break the task down by various AST node types. The functions are all named `compiler_visit_{xx}` where *xx* is the name of the node -type (such as `stmt`, `expr`, etc.). Each function receives a `struct compiler *` -and `{xx}_ty` where *xx* is the AST node type. Typically these functions +type (such as `stmt`, `expr`, etc.). Each function receives a `struct compiler *` +and `{xx}_ty` where *xx* is the AST node type. Typically these functions consist of a large 'switch' statement, branching based on the kind of -node type passed to it. Simple things are handled inline in the +node type passed to it. Simple things are handled inline in the 'switch' statement with more complex transformations farmed out to other functions named `compiler_{xx}` with *xx* being a descriptive name of what is being handled. @@ -373,39 +372,39 @@ being handled. When transforming an arbitrary AST node, use the `VISIT()` macro. The appropriate `compiler_visit_{xx}` function is called, based on the value passed in for (so `VISIT({c}, expr, {node})` calls -`compiler_visit_expr({c}, {node})`). The `VISIT_SEQ()` macro is very similar, +`compiler_visit_expr({c}, {node})`). The `VISIT_SEQ()` macro is very similar, but is called on AST node sequences (those values that were created as arguments to a node that used the '*' modifier). Emission of bytecode is handled by the following macros: -* `ADDOP(struct compiler *, location, int)` - add a specified opcode -* `ADDOP_IN_SCOPE(struct compiler *, location, int)` - like `ADDOP`, but also exits current scope; used for adding return value - opcodes in lambdas and closures -* `ADDOP_I(struct compiler *, location, int, Py_ssize_t)` - add an opcode that takes an integer argument -* `ADDOP_O(struct compiler *, location, int, PyObject *, TYPE)` - add an opcode with the proper argument based on the position of the - specified PyObject in PyObject sequence object, but with no handling of - mangled names; used for when you - need to do named lookups of objects such as globals, consts, or - parameters where name mangling is not possible and the scope of the - name is known; *TYPE* is the name of PyObject sequence - (`names` or `varnames`) -* `ADDOP_N(struct compiler *, location, int, PyObject *, TYPE)` - just like `ADDOP_O`, but steals a reference to PyObject -* `ADDOP_NAME(struct compiler *, location, int, PyObject *, TYPE)` - just like `ADDOP_O`, but name mangling is also handled; used for - attribute loading or importing based on name -* `ADDOP_LOAD_CONST(struct compiler *, location, PyObject *)` - add the `LOAD_CONST` opcode with the proper argument based on the - position of the specified PyObject in the consts table. -* `ADDOP_LOAD_CONST_NEW(struct compiler *, location, PyObject *)` - just like `ADDOP_LOAD_CONST_NEW`, but steals a reference to PyObject -* `ADDOP_JUMP(struct compiler *, location, int, basicblock *)` - create a jump to a basic block +* `ADDOP(struct compiler *, location, int)`: + add a specified opcode +* `ADDOP_IN_SCOPE(struct compiler *, location, int)`: + like `ADDOP`, but also exits current scope; used for adding return value + opcodes in lambdas and closures +* `ADDOP_I(struct compiler *, location, int, Py_ssize_t)`: + add an opcode that takes an integer argument +* `ADDOP_O(struct compiler *, location, int, PyObject *, TYPE)`: + add an opcode with the proper argument based on the position of the + specified PyObject in PyObject sequence object, but with no handling of + mangled names; used for when you + need to do named lookups of objects such as globals, consts, or + parameters where name mangling is not possible and the scope of the + name is known; *TYPE* is the name of PyObject sequence + (`names` or `varnames`) +* `ADDOP_N(struct compiler *, location, int, PyObject *, TYPE)`: + just like `ADDOP_O`, but steals a reference to PyObject +* `ADDOP_NAME(struct compiler *, location, int, PyObject *, TYPE)`: + just like `ADDOP_O`, but name mangling is also handled; used for + attribute loading or importing based on name +* `ADDOP_LOAD_CONST(struct compiler *, location, PyObject *)`: + add the `LOAD_CONST` opcode with the proper argument based on the + position of the specified PyObject in the consts table. +* `ADDOP_LOAD_CONST_NEW(struct compiler *, location, PyObject *)`: + just like `ADDOP_LOAD_CONST_NEW`, but steals a reference to PyObject +* `ADDOP_JUMP(struct compiler *, location, int, basicblock *)`: + create a jump to a basic block The `location` argument is a struct with the source location to be associated with this instruction. It is typically extracted from an @@ -417,7 +416,7 @@ line in the source code. There are several helper functions that will emit pseudo-instructions and are named `compiler_{xx}()` where *xx* is what the function helps -with (`list`, `boolop`, etc.). A rather useful one is `compiler_nameop()`. +with (`list`, `boolop`, etc.). A rather useful one is `compiler_nameop()`. This function looks up the scope of a variable and, based on the expression context, emits the proper opcode to load, store, or delete the variable. @@ -433,7 +432,7 @@ Finally, the sequence of pseudo-instructions is converted into actual bytecode. This includes transforming pseudo instructions into actual instructions, converting jump targets from logical labels to relative offsets, and construction of the [exception table](exception_handling.md) and -[locations table](locations.md). +[locations table](code_objects.md#source-code-locations). The bytecode and tables are then wrapped into a `PyCodeObject` along with additional metadata, including the `consts` and `names` arrays, information about function reference to the source code (filename, etc). All of this is implemented by @@ -453,7 +452,7 @@ in [Python/ceval.c](../Python/ceval.c). Important files =============== -* [Parser/](../Parser/) +* [Parser/](../Parser) * [Parser/Python.asdl](../Parser/Python.asdl): ASDL syntax file. @@ -463,15 +462,15 @@ Important files Reads in an ASDL description and parses it into an AST that describes it. * [Parser/asdl_c.py](../Parser/asdl_c.py): - Generate C code from an ASDL description. Generates + Generate C code from an ASDL description. Generates [Python/Python-ast.c](../Python/Python-ast.c) and [Include/internal/pycore_ast.h](../Include/internal/pycore_ast.h). * [Parser/parser.c](../Parser/parser.c): - The new PEG parser introduced in Python 3.9. Generated by + The new PEG parser introduced in Python 3.9. Generated by [Tools/peg_generator/pegen/c_generator.py](../Tools/peg_generator/pegen/c_generator.py) from the grammar [Grammar/python.gram](../Grammar/python.gram). - Creates the AST from source code. Rule functions for their corresponding production + Creates the AST from source code. Rule functions for their corresponding production rules are found here. * [Parser/peg_api.c](../Parser/peg_api.c): @@ -480,7 +479,7 @@ Important files * [Parser/pegen.c](../Parser/pegen.c): Contains helper functions which are used by functions in - [Parser/parser.c](../Parser/parser.c) to construct the AST. Also contains + [Parser/parser.c](../Parser/parser.c) to construct the AST. Also contains helper functions which help raise better error messages when parsing source code. * [Parser/pegen.h](../Parser/pegen.h): @@ -490,7 +489,7 @@ Important files * [Python/](../Python) * [Python/Python-ast.c](../Python/Python-ast.c): - Creates C structs corresponding to the ASDL types. Also contains code for + Creates C structs corresponding to the ASDL types. Also contains code for marshalling AST nodes (core ASDL types have marshalling code in [Python/asdl.c](../Python/asdl.c)). File automatically generated by [Parser/asdl_c.py](../Parser/asdl_c.py). @@ -501,7 +500,7 @@ Important files * [Python/asdl.c](../Python/asdl.c): Contains code to handle the ASDL sequence type. Also has code to handle marshalling the core ASDL types, such as number - and identifier. Used by [Python/Python-ast.c](../Python/Python-ast.c) + and identifier. Used by [Python/Python-ast.c](../Python/Python-ast.c) for marshalling AST nodes. * [Python/ast.c](../Python/ast.c): @@ -534,7 +533,7 @@ Important files * [Python/instruction_sequence.c](../Python/instruction_sequence.c): A data structure representing a sequence of bytecode-like pseudo-instructions. -* [Include/](../Include/) +* [Include/](../Include) * [Include/cpython/code.h](../Include/cpython/code.h) : Header file for [Objects/codeobject.c](../Objects/codeobject.c); @@ -556,7 +555,7 @@ Important files : Declares `_PyAST_Validate()` external (from [Python/ast.c](../Python/ast.c)). * [Include/internal/pycore_symtable.h](../Include/internal/pycore_symtable.h) - : Header for [Python/symtable.c](../Python/symtable.c). + : Header for [Python/symtable.c](../Python/symtable.c). `struct symtable` and `PySTEntryObject` are defined here. * [Include/internal/pycore_parser.h](../Include/internal/pycore_parser.h) @@ -570,7 +569,7 @@ Important files by [Tools/cases_generator/opcode_id_generator.py](../Tools/cases_generator/opcode_id_generator.py). -* [Objects/](../Objects/) +* [Objects/](../Objects) * [Objects/codeobject.c](../Objects/codeobject.c) : Contains PyCodeObject-related code. @@ -579,7 +578,7 @@ Important files : Contains the `frame_setlineno()` function which should determine whether it is allowed to make a jump between two points in a bytecode. -* [Lib/](../Lib/) +* [Lib/](../Lib) * [Lib/opcode.py](../Lib/opcode.py) : opcode utilities exposed to Python. @@ -591,7 +590,7 @@ Important files Objects ======= -* [Locations](locations.md): Describes the location table +* [Locations](code_objects.md#source-code-locations): Describes the location table * [Frames](frames.md): Describes frames and the frame stack * [Objects/object_layout.md](../Objects/object_layout.md): Describes object layout for 3.11 and later * [Exception Handling](exception_handling.md): Describes the exception table @@ -611,9 +610,9 @@ References ========== [^1]: Daniel C. Wang, Andrew W. Appel, Jeff L. Korn, and Chris - S. Serra. `The Zephyr Abstract Syntax Description Language.`_ - In Proceedings of the Conference on Domain-Specific Languages, - pp. 213--227, 1997. +S. Serra. `The Zephyr Abstract Syntax Description Language.`_ +In Proceedings of the Conference on Domain-Specific Languages, +pp. 213--227, 1997. [^2]: The Zephyr Abstract Syntax Description Language.: - https://www.cs.princeton.edu/research/techreps/TR-554-97 +https://www.cs.princeton.edu/research/techreps/TR-554-97 diff --git a/InternalDocs/exception_handling.md b/InternalDocs/exception_handling.md index 14066a5864b4da..322f83a4ef59d0 100644 --- a/InternalDocs/exception_handling.md +++ b/InternalDocs/exception_handling.md @@ -87,10 +87,10 @@ offset of the raising instruction should be pushed to the stack. Handling an exception, once an exception table entry is found, consists of the following steps: - 1. pop values from the stack until it matches the stack depth for the handler. - 2. if `lasti` is true, then push the offset that the exception was raised at. - 3. push the exception to the stack. - 4. jump to the target offset and resume execution. +1. pop values from the stack until it matches the stack depth for the handler. +2. if `lasti` is true, then push the offset that the exception was raised at. +3. push the exception to the stack. +4. jump to the target offset and resume execution. Reraising Exceptions and `lasti` @@ -107,13 +107,12 @@ Format of the exception table ----------------------------- Conceptually, the exception table consists of a sequence of 5-tuples: -``` - 1. `start-offset` (inclusive) - 2. `end-offset` (exclusive) - 3. `target` - 4. `stack-depth` - 5. `push-lasti` (boolean) -``` + +1. `start-offset` (inclusive) +2. `end-offset` (exclusive) +3. `target` +4. `stack-depth` +5. `push-lasti` (boolean) All offsets and lengths are in code units, not bytes. @@ -123,18 +122,19 @@ For it to be searchable quickly, we need to support binary search giving us log( Binary search typically assumes fixed size entries, but that is not necessary, as long as we can identify the start of an entry. It is worth noting that the size (end-start) is always smaller than the end, so we encode the entries as: - `start, size, target, depth, push-lasti`. +`start, size, target, depth, push-lasti`. Also, sizes are limited to 2**30 as the code length cannot exceed 2**31 and each code unit takes 2 bytes. It also happens that depth is generally quite small. So, we need to encode: + ``` - `start` (up to 30 bits) - `size` (up to 30 bits) - `target` (up to 30 bits) - `depth` (up to ~8 bits) - `lasti` (1 bit) +start (up to 30 bits) +size (up to 30 bits) +target (up to 30 bits) +depth (up to ~8 bits) +lasti (1 bit) ``` We need a marker for the start of the entry, so the first byte of entry will have the most significant bit set. @@ -145,29 +145,32 @@ The 8 bits of a byte are (msb left) SXdddddd where S is the start bit. X is the In addition, we combine `depth` and `lasti` into a single value, `((depth<<1)+lasti)`, before encoding. For example, the exception entry: + ``` - `start`: 20 - `end`: 28 - `target`: 100 - `depth`: 3 - `lasti`: False +start: 20 +end: 28 +target: 100 +depth: 3 +lasti: False ``` is encoded by first converting to the more compact four value form: + ``` - `start`: 20 - `size`: 8 - `target`: 100 - `depth<<1+lasti`: 6 +start: 20 +size: 8 +target: 100 +depth<<1+lasti: 6 ``` which is then encoded as: + ``` - 148 (MSB + 20 for start) - 8 (size) - 65 (Extend bit + 1) - 36 (Remainder of target, 100 == (1<<6)+36) - 6 +148 (MSB + 20 for start) +8 (size) +65 (Extend bit + 1) +36 (Remainder of target, 100 == (1<<6)+36) +6 ``` for a total of five bytes. @@ -187,5 +190,5 @@ Exception Chaining Implementation refers to setting the `__context__` and `__cause__` fields of an exception as it is being raised. The `__context__` field is set by `_PyErr_SetObject()` in [Python/errors.c](../Python/errors.c) (which is ultimately called by all -`PyErr_Set*()` functions). The `__cause__` field (explicit chaining) is set by +`PyErr_Set*()` functions). The `__cause__` field (explicit chaining) is set by the `RAISE_VARARGS` bytecode. diff --git a/InternalDocs/frames.md b/InternalDocs/frames.md index 06dc8f0702c3d9..efdf9927e7a25e 100644 --- a/InternalDocs/frames.md +++ b/InternalDocs/frames.md @@ -27,6 +27,7 @@ objects, so are not allocated in the per-thread stack. See `PyGenObject` in ## Layout Each activation record is laid out as: + * Specials * Locals * Stack @@ -124,10 +125,10 @@ the next instruction to be executed. During a call to a python function, to see in an exception traceback. The `return_offset` field determines where a `RETURN` should go in the caller, -relative to `instr_ptr`. It is only meaningful to the callee, so it needs to +relative to `instr_ptr`. It is only meaningful to the callee, so it needs to be set in any instruction that implements a call (to a Python function), including CALL, SEND and BINARY_SUBSCR_GETITEM, among others. If there is no -callee, then return_offset is meaningless. It is necessary to have a separate +callee, then return_offset is meaningless. It is necessary to have a separate field for the return offset because (1) if we apply this offset to `instr_ptr` while executing the `RETURN`, this is too early and would lose us information about the previous instruction which we could need for introspecting and diff --git a/InternalDocs/garbage_collector.md b/InternalDocs/garbage_collector.md index 272a0834cbfe24..8f4b67f8320d19 100644 --- a/InternalDocs/garbage_collector.md +++ b/InternalDocs/garbage_collector.md @@ -1,4 +1,3 @@ - Garbage collector design ======================== @@ -56,7 +55,7 @@ Starting in version 3.13, CPython contains two GC implementations: performing a collection for thread safety. Both implementations use the same basic algorithms, but operate on different -data structures. See the section on +data structures. See the section on [Differences between GC implementations](#Differences-between-GC-implementations) for the details. @@ -65,7 +64,7 @@ Memory layout and object structure ================================== The garbage collector requires additional fields in Python objects to support -garbage collection. These extra fields are different in the default and the +garbage collection. These extra fields are different in the default and the free-threaded builds. @@ -112,12 +111,12 @@ that in the [Optimization: incremental collection](#Optimization-incremental-col they are also reused to fulfill other purposes when the full doubly linked list structure is not needed as a memory optimization. -Doubly linked lists are used because they efficiently support the most frequently required operations. In +Doubly linked lists are used because they efficiently support the most frequently required operations. In general, the collection of all objects tracked by GC is partitioned into disjoint sets, each in its own -doubly linked list. Between collections, objects are partitioned into "generations", reflecting how -often they've survived collection attempts. During collections, the generation(s) being collected -are further partitioned into, for example, sets of reachable and unreachable objects. Doubly linked lists -support moving an object from one partition to another, adding a new object, removing an object +doubly linked list. Between collections, objects are partitioned into "generations", reflecting how +often they've survived collection attempts. During collections, the generation(s) being collected +are further partitioned into, for example, sets of reachable and unreachable objects. Doubly linked lists +support moving an object from one partition to another, adding a new object, removing an object entirely (objects tracked by GC are most often reclaimed by the refcounting system when GC isn't running at all!), and merging partitions, all with a small constant number of pointer updates. With care, they also support iterating over a partition while objects are being added to - and @@ -129,7 +128,7 @@ GC for the free-threaded build In the free-threaded build, Python objects contain a 1-byte field `ob_gc_bits` that is used to track garbage collection related state. The field exists in all objects, including ones that do not support cyclic -garbage collection. The field is used to identify objects that are tracked +garbage collection. The field is used to identify objects that are tracked by the collector, ensure that finalizers are called only once per object, and, during garbage collection, differentiate reachable vs. unreachable objects. @@ -193,7 +192,7 @@ the interpreter create cycles everywhere. Some notable examples: have internal links to themselves. To correctly dispose of these objects once they become unreachable, they need -to be identified first. To understand how the algorithm works, let’s take +to be identified first. To understand how the algorithm works, let’s take the case of a circular linked list which has one link referenced by a variable `A`, and one self-referencing object which is completely unreachable: @@ -221,15 +220,15 @@ unreachable: 2 ``` -The GC starts with a set of candidate objects it wants to scan. In the +The GC starts with a set of candidate objects it wants to scan. In the default build, these "objects to scan" might be all container objects or a -smaller subset (or "generation"). In the free-threaded build, the collector +smaller subset (or "generation"). In the free-threaded build, the collector always scans all container objects. -The objective is to identify all the unreachable objects. The collector does +The objective is to identify all the unreachable objects. The collector does this by identifying reachable objects; the remaining objects must be -unreachable. The first step is to identify all of the "to scan" objects that -are **directly** reachable from outside the set of candidate objects. These +unreachable. The first step is to identify all of the "to scan" objects that +are **directly** reachable from outside the set of candidate objects. These objects have a refcount larger than the number of incoming references from within the candidate set. @@ -242,7 +241,7 @@ interpreter will not modify the real reference count field. ![gc-image1](images/python-cyclic-gc-1-new-page.png) The GC then iterates over all containers in the first list and decrements by one the -`gc_ref` field of any other object that container is referencing. Doing +`gc_ref` field of any other object that container is referencing. Doing this makes use of the `tp_traverse` slot in the container class (implemented using the C API or inherited by a superclass) to know what objects are referenced by each container. After all the objects have been scanned, only the objects that have @@ -274,7 +273,7 @@ When the GC encounters an object which is reachable (`gc_ref > 0`), it traverses its references using the `tp_traverse` slot to find all the objects that are reachable from it, moving them to the end of the list of reachable objects (where they started originally) and setting its `gc_ref` field to 1. This is what happens -to `link_2` and `link_3` below as they are reachable from `link_1`. From the +to `link_2` and `link_3` below as they are reachable from `link_1`. From the state in the previous image and after examining the objects referred to by `link_1` the GC knows that `link_3` is reachable after all, so it is moved back to the original list and its `gc_ref` field is set to 1 so that if the GC visits it again, @@ -294,7 +293,7 @@ list are really unreachable and can thus be garbage collected. Pragmatically, it's important to note that no recursion is required by any of this, and neither does it in any other way require additional memory proportional to the -number of objects, number of pointers, or the lengths of pointer chains. Apart from +number of objects, number of pointers, or the lengths of pointer chains. Apart from `O(1)` storage for internal C needs, the objects themselves contain all the storage the GC algorithms require. @@ -318,8 +317,8 @@ list. So instead of not moving at all, the reachable objects B and A are each moved twice. Why is this a win? A straightforward algorithm to move the reachable objects instead would move A, B, and C once each. The key is that this dance leaves the objects in -order C, B, A - it's reversed from the original order. On all *subsequent* scans, -none of them will move. Since most objects aren't in cycles, this can save an +order C, B, A - it's reversed from the original order. On all *subsequent* scans, +none of them will move. Since most objects aren't in cycles, this can save an unbounded number of moves across an unbounded number of later collections. The only time the cost can be higher is the first time the chain is scanned. @@ -332,7 +331,7 @@ follows these steps in order: 1. Handle and clear weak references (if any). Weak references to unreachable objects are set to `None`. If the weak reference has an associated callback, the callback - is enqueued to be called once the clearing of weak references is finished. We only + is enqueued to be called once the clearing of weak references is finished. We only invoke callbacks for weak references that are themselves reachable. If both the weak reference and the pointed-to object are unreachable we do not execute the callback. This is partly for historical reasons: the callback could resurrect an unreachable @@ -491,7 +490,7 @@ to the size of the data, often a word or multiple thereof. This discrepancy leaves a few of the least significant bits of the pointer unused, which can be used for tags or to keep other information – most often as a bit field (each bit a separate tag) – as long as code that uses the pointer masks out these -bits before accessing memory. For example, on a 32-bit architecture (for both +bits before accessing memory. For example, on a 32-bit architecture (for both addresses and word size), a word is 32 bits = 4 bytes, so word-aligned addresses are always a multiple of 4, hence end in `00`, leaving the last 2 bits available; while on a 64-bit architecture, a word is 64 bits = 8 bytes, so @@ -520,10 +519,10 @@ of `PyGC_Head` discussed in the `Memory layout and object structure`_ section: - The `_gc_next` field is used as the "next" pointer to maintain the doubly linked list but during collection its lowest bit is used to keep the `NEXT_MASK_UNREACHABLE` flag that indicates if an object is tentatively - unreachable during the cycle detection algorithm. This is a drawback to using only + unreachable during the cycle detection algorithm. This is a drawback to using only doubly linked lists to implement partitions: while most needed operations are constant-time, there is no efficient way to determine which partition an object is - currently in. Instead, when that's needed, ad hoc tricks (like the + currently in. Instead, when that's needed, ad hoc tricks (like the `NEXT_MASK_UNREACHABLE` flag) are employed. Optimization: delayed untracking containers @@ -582,29 +581,29 @@ structure, while the free-threaded build implementation does not use that data structure. - The default build implementation stores all tracked objects in a doubly - linked list using `PyGC_Head`. The free-threaded build implementation + linked list using `PyGC_Head`. The free-threaded build implementation instead relies on the embedded mimalloc memory allocator to scan the heap for tracked objects. - The default build implementation uses `PyGC_Head` for the unreachable - object list. The free-threaded build implementation repurposes the + object list. The free-threaded build implementation repurposes the `ob_tid` field to store a unreachable objects linked list. - The default build implementation stores flags in the `_gc_prev` field of - `PyGC_Head`. The free-threaded build implementation stores these flags + `PyGC_Head`. The free-threaded build implementation stores these flags in `ob_gc_bits`. The default build implementation relies on the [global interpreter lock](https://docs.python.org/3/glossary.html#term-global-interpreter-lock) -for thread safety. The free-threaded build implementation has two "stop the +for thread safety. The free-threaded build implementation has two "stop the world" pauses, in which all other executing threads are temporarily paused so that the GC can safely access reference counts and object attributes. -The default build implementation is a generational collector. The +The default build implementation is a generational collector. The free-threaded build is non-generational; each collection scans the entire heap. - Keeping track of object generations is simple and inexpensive in the default - build. The free-threaded build relies on mimalloc for finding tracked + build. The free-threaded build relies on mimalloc for finding tracked objects; identifying "young" objects without scanning the entire heap would be more difficult. diff --git a/InternalDocs/generators.md b/InternalDocs/generators.md index d53f0f9bdff4e4..afa8b8f4bb8040 100644 --- a/InternalDocs/generators.md +++ b/InternalDocs/generators.md @@ -1,4 +1,3 @@ - Generators ========== diff --git a/InternalDocs/interpreter.md b/InternalDocs/interpreter.md index 4c10cbbed37735..63ae51dcd3eb7a 100644 --- a/InternalDocs/interpreter.md +++ b/InternalDocs/interpreter.md @@ -1,4 +1,3 @@ - The bytecode interpreter ======================== @@ -32,7 +31,7 @@ It also has a reference to the `CodeObject` itself. In addition to the frame, `_PyEval_EvalFrame()` also receives a [`Thread State`](https://docs.python.org/3/c-api/init.html#c.PyThreadState) object, `tstate`, which includes things like the exception state and the -recursion depth. The thread state also provides access to the per-interpreter +recursion depth. The thread state also provides access to the per-interpreter state (`tstate->interp`), which has a pointer to the per-runtime (that is, truly global) state (`tstate->interp->runtime`). @@ -131,7 +130,7 @@ The size of the inline cache for a particular instruction is fixed by its `opcod Moreover, the inline cache size for all instructions in a [family of specialized/specializable instructions](adaptive.md) (for example, `LOAD_ATTR`, `LOAD_ATTR_SLOT`, `LOAD_ATTR_MODULE`) must all be -the same. Cache entries are reserved by the compiler and initialized with zeros. +the same. Cache entries are reserved by the compiler and initialized with zeros. Although they are represented by code units, cache entries do not conform to the `opcode` / `oparg` format. @@ -139,7 +138,7 @@ If an instruction has an inline cache, the layout of its cache is described by a `struct` definition in (`pycore_code.h`)[../Include/internal/pycore_code.h]. This allows us to access the cache by casting `next_instr` to a pointer to this `struct`. The size of such a `struct` must be independent of the machine architecture, word size -and alignment requirements. For a 32-bit field, the `struct` should use `_Py_CODEUNIT field[2]`. +and alignment requirements. For a 32-bit field, the `struct` should use `_Py_CODEUNIT field[2]`. The instruction implementation is responsible for advancing `next_instr` past the inline cache. For example, if an instruction's inline cache is four bytes (that is, two code units) in size, @@ -211,12 +210,12 @@ In 3.10 and before, this was the case even when a Python function called another Python function: The `CALL` opcode would call the `tp_call` dispatch function of the callee, which would extract the code object, create a new frame for the call -stack, and then call back into the interpreter. This approach is very general +stack, and then call back into the interpreter. This approach is very general but consumes several C stack frames for each nested Python call, thereby increasing the risk of an (unrecoverable) C stack overflow. Since 3.11, the `CALL` instruction special-cases function objects to "inline" -the call. When a call gets inlined, a new frame gets pushed onto the call +the call. When a call gets inlined, a new frame gets pushed onto the call stack and the interpreter "jumps" to the start of the callee's bytecode. When an inlined callee executes a `RETURN_VALUE` instruction, the frame is popped off the call stack and the interpreter returns to its caller, @@ -249,12 +248,12 @@ In this case we allocate a proper `PyFrameObject` and initialize it from the Things get more complicated when generators are involved, since those do not follow the push/pop model. This includes async functions, which are based on -the same mechanism. A generator object has space for a `_PyInterpreterFrame` +the same mechanism. A generator object has space for a `_PyInterpreterFrame` structure, including the variable-size part (used for locals and the eval stack). When a generator (or async) function is first called, a special opcode `RETURN_GENERATOR` is executed, which is responsible for creating the -generator object. The generator object's `_PyInterpreterFrame` is initialized -with a copy of the current stack frame. The current stack frame is then popped +generator object. The generator object's `_PyInterpreterFrame` is initialized +with a copy of the current stack frame. The current stack frame is then popped off the frame stack and the generator object is returned. (Details differ depending on the `is_entry` flag.) When the generator is resumed, the interpreter pushes its `_PyInterpreterFrame` @@ -318,9 +317,9 @@ With a new bytecode you must also change what is called the "magic number" for .pyc files: bump the value of the variable `MAGIC_NUMBER` in [`Lib/importlib/_bootstrap_external.py`](../Lib/importlib/_bootstrap_external.py). Changing this number will lead to all .pyc files with the old `MAGIC_NUMBER` -to be recompiled by the interpreter on import. Whenever `MAGIC_NUMBER` is +to be recompiled by the interpreter on import. Whenever `MAGIC_NUMBER` is changed, the ranges in the `magic_values` array in -[`PC/launcher.c`](../PC/launcher.c) may also need to be updated. Changes to +[`PC/launcher.c`](../PC/launcher.c) may also need to be updated. Changes to [`Lib/importlib/_bootstrap_external.py`](../Lib/importlib/_bootstrap_external.py) will take effect only after running `make regen-importlib`. @@ -334,12 +333,12 @@ will take effect only after running `make regen-importlib`. > On Windows, running the `./build.bat` script will automatically > regenerate the required files without requiring additional arguments. -Finally, you need to introduce the use of the new bytecode. Update +Finally, you need to introduce the use of the new bytecode. Update [`Python/codegen.c`](../Python/codegen.c) to emit code with this bytecode. Optimizations in [`Python/flowgraph.c`](../Python/flowgraph.c) may also -need to be updated. If the new opcode affects a control flow or the block +need to be updated. If the new opcode affects a control flow or the block stack, you may have to update the `frame_setlineno()` function in -[`Objects/frameobject.c`](../Objects/frameobject.c). It may also be necessary +[`Objects/frameobject.c`](../Objects/frameobject.c). It may also be necessary to update [`Lib/dis.py`](../Lib/dis.py) if the new opcode interprets its argument in a special way (like `FORMAT_VALUE` or `MAKE_FUNCTION`). @@ -348,12 +347,12 @@ is already in existence and you do not change the magic number, make sure to delete your old .py(c|o) files! Even though you will end up changing the magic number if you change the bytecode, while you are debugging your work you may be changing the bytecode output without constantly bumping up the -magic number. This can leave you with stale .pyc files that will not be +magic number. This can leave you with stale .pyc files that will not be recreated. Running `find . -name '*.py[co]' -exec rm -f '{}' +` should delete all .pyc files you have, forcing new ones to be created and thus allow you test out your -new bytecode properly. Run `make regen-importlib` for updating the -bytecode of frozen importlib files. You have to run `make` again after this +new bytecode properly. Run `make regen-importlib` for updating the +bytecode of frozen importlib files. You have to run `make` again after this to recompile the generated C files. Additional resources diff --git a/InternalDocs/parser.md b/InternalDocs/parser.md index 348988b7c2f003..942c940236c3fc 100644 --- a/InternalDocs/parser.md +++ b/InternalDocs/parser.md @@ -1,4 +1,3 @@ - Guide to the parser =================== @@ -57,11 +56,11 @@ an input string as its argument, and yields one of the following results: Note that "failure" results do not imply that the program is incorrect, nor do they necessarily mean that the parsing has failed. Since the choice operator is -ordered, a failure very often merely indicates "try the following option". A +ordered, a failure very often merely indicates "try the following option". A direct implementation of a PEG parser as a recursive descent parser will present exponential time performance in the worst case, because PEG parsers have infinite lookahead (this means that they can consider an arbitrary number of -tokens before deciding for a rule). Usually, PEG parsers avoid this exponential +tokens before deciding for a rule). Usually, PEG parsers avoid this exponential time complexity with a technique called ["packrat parsing"](https://pdos.csail.mit.edu/~baford/packrat/thesis/) which not only loads the entire program in memory before parsing it but also @@ -444,15 +443,15 @@ How to regenerate the parser Once you have made the changes to the grammar files, to regenerate the `C` parser (the one used by the interpreter) just execute: -``` - make regen-pegen +```shell +$ make regen-pegen ``` -using the `Makefile` in the main directory. If you are on Windows you can +using the `Makefile` in the main directory. If you are on Windows you can use the Visual Studio project files to regenerate the parser or to execute: -``` - ./PCbuild/build.bat --regen +```dos +PCbuild/build.bat --regen ``` The generated parser file is located at [`Parser/parser.c`](../Parser/parser.c). @@ -468,15 +467,15 @@ any modifications to this file (in order to implement new Pegen features) you wi need to regenerate the meta-parser (the parser that parses the grammar files). To do so just execute: -``` - make regen-pegen-metaparser +```shell +$ make regen-pegen-metaparser ``` If you are on Windows you can use the Visual Studio project files to regenerate the parser or to execute: -``` - ./PCbuild/build.bat --regen +```dos +PCbuild/build.bat --regen ``` @@ -516,15 +515,15 @@ be found in the [`Grammar/Tokens`](../Grammar/Tokens) file. If you change this file to add new tokens, make sure to regenerate the files by executing: -``` - make regen-token +```shell +$ make regen-token ``` If you are on Windows you can use the Visual Studio project files to regenerate the tokens or to execute: -``` - ./PCbuild/build.bat --regen +```dos +PCbuild/build.bat --regen ``` How tokens are generated and the rules governing this are completely up to the tokenizer @@ -546,8 +545,8 @@ by default** except for rules with the special marker `memo` after the rule name (and type, if present): ``` - rule_name[typr] (memo): - ... +rule_name[typr] (memo): + ... ``` By selectively turning on memoization for a handful of rules, the parser becomes @@ -593,25 +592,25 @@ are always reserved words, even in positions where they make no sense meaning in context. Trying to use a hard keyword as a variable will always fail: -``` - >>> class = 3 - File "", line 1 - class = 3 - ^ - SyntaxError: invalid syntax - >>> foo(class=3) - File "", line 1 - foo(class=3) - ^^^^^ - SyntaxError: invalid syntax +```pycon +>>> class = 3 +File "", line 1 + class = 3 + ^ +SyntaxError: invalid syntax +>>> foo(class=3) +File "", line 1 + foo(class=3) + ^^^^^ +SyntaxError: invalid syntax ``` While soft keywords don't have this limitation if used in a context other the one where they are defined as keywords: -``` - >>> match = 45 - >>> foo(match="Yeah!") +```pycon +>>> match = 45 +>>> foo(match="Yeah!") ``` The `match` and `case` keywords are soft keywords, so that they are @@ -621,21 +620,21 @@ argument names. You can get a list of all keywords defined in the grammar from Python: -``` - >>> import keyword - >>> keyword.kwlist - ['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break', - 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', - 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', - 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield'] +```pycon +>>> import keyword +>>> keyword.kwlist +['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break', +'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', +'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', +'pass', 'raise', 'return', 'try', 'while', 'with', 'yield'] ``` as well as soft keywords: -``` - >>> import keyword - >>> keyword.softkwlist - ['_', 'case', 'match'] +```pycon +>>> import keyword +>>> keyword.softkwlist +['_', 'case', 'match'] ``` > [!CAUTION] @@ -736,7 +735,7 @@ displayed when the error is reported. > rule or not. For example: ``` - $ 42 + $ 42 ``` should trigger the syntax error in the `$` character. If your rule is not correctly defined this @@ -744,7 +743,7 @@ won't happen. As another example, suppose that you try to define a rule to match `print` statements in order to create a better error message and you define it as: ``` - invalid_print: "print" expression +invalid_print: "print" expression ``` This will **seem** to work because the parser will correctly parse `print(something)` because it is valid @@ -756,7 +755,7 @@ will be reported there instead of the `$` character. Generating AST objects ---------------------- -The output of the C parser used by CPython, which is generated from the +The output of the C parser used by CPython, which is generated from the [grammar file](../Grammar/python.gram), is a Python AST object (using C structures). This means that the actions in the grammar file generate AST objects when they succeed. Constructing these objects can be quite cumbersome @@ -798,7 +797,7 @@ Check the contents of these files to know which is the best place for new tests, depending on the nature of the new feature you are adding. Tests for the parser generator itself can be found in the -[test_peg_generator](../Lib/test_peg_generator) directory. +[test_peg_generator](../Lib/test/test_peg_generator) directory. Debugging generated parsers @@ -816,15 +815,15 @@ For this reason it is a good idea to experiment first by generating a Python parser. To do this, you can go to the [Tools/peg_generator](../Tools/peg_generator) directory on the CPython repository and manually call the parser generator by executing: -``` - $ python -m pegen python +```shell +$ python -m pegen python ``` This will generate a file called `parse.py` in the same directory that you can use to parse some input: -``` - $ python parse.py file_with_source_code_to_test.py +```shell +$ python parse.py file_with_source_code_to_test.py ``` As the generated `parse.py` file is just Python code, you can modify it @@ -848,8 +847,8 @@ can be a bit hard to understand at first. To activate verbose mode you can add the `-d` flag when executing Python: -``` - $ python -d file_to_test.py +```shell +$ python -d file_to_test.py ``` This will print **a lot** of output to `stderr` so it is probably better to dump @@ -857,7 +856,7 @@ it to a file for further analysis. The output consists of trace lines with the following structure:: ``` - ('>'|'-'|'+'|'!') []: ... + ('>'|'-'|'+'|'!') []: ... ``` Every line is indented by a different amount (``) depending on how diff --git a/InternalDocs/string_interning.md b/InternalDocs/string_interning.md index e0d20632516142..26a5197c6e70f3 100644 --- a/InternalDocs/string_interning.md +++ b/InternalDocs/string_interning.md @@ -2,6 +2,7 @@ *Interned* strings are conceptually part of an interpreter-global *set* of interned strings, meaning that: + - no two interned strings have the same content (across an interpreter); - two interned strings can be safely compared using pointer equality (Python `is`). @@ -61,6 +62,7 @@ if it's interned and mortal it needs extra processing in The converse is not true: interned strings can be mortal. For mortal interned strings: + - the 2 references from the interned dict (key & value) are excluded from their refcount - the deallocator (`unicode_dealloc`) removes the string from the interned dict @@ -90,6 +92,7 @@ modify in place. The functions take ownership of (“steal”) the reference to their argument, and update the argument with a *new* reference. This means: + - They're “reference neutral”. - They must not be called with a borrowed reference. From f6476e86390cdbd643b62c80fc95baa10ace0897 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?B=C3=A9n=C3=A9dikt=20Tran?= <10796600+picnixz@users.noreply.github.com> Date: Sun, 1 Dec 2024 17:29:52 +0100 Subject: [PATCH 2/2] cancel some edits --- InternalDocs/changing_grammar.md | 2 +- InternalDocs/compiler.md | 117 ++++++++++++++--------------- InternalDocs/exception_handling.md | 2 +- InternalDocs/frames.md | 4 +- InternalDocs/garbage_collector.md | 56 +++++++------- InternalDocs/interpreter.md | 32 ++++---- InternalDocs/parser.md | 6 +- 7 files changed, 109 insertions(+), 110 deletions(-) diff --git a/InternalDocs/changing_grammar.md b/InternalDocs/changing_grammar.md index 3851766e314ced..c6b895135a360d 100644 --- a/InternalDocs/changing_grammar.md +++ b/InternalDocs/changing_grammar.md @@ -18,7 +18,7 @@ Below is a checklist of things that may need to change. to regenerate [`Parser/parser.c`](../Parser/parser.c). (This runs Python's parser generator, [`Tools/peg_generator`](../Tools/peg_generator)). -* [`Grammar/Tokens`](../Grammar/Tokens) is a place for adding new token types. After +* [`Grammar/Tokens`](../Grammar/Tokens) is a place for adding new token types. After changing it, run ``make regen-token`` to regenerate [`Include/internal/pycore_token.h`](../Include/internal/pycore_token.h), [`Parser/token.c`](../Parser/token.c), [`Lib/token.py`](../Lib/token.py) diff --git a/InternalDocs/compiler.md b/InternalDocs/compiler.md index 1a355bd96d228b..9e99f348acbd8f 100644 --- a/InternalDocs/compiler.md +++ b/InternalDocs/compiler.md @@ -20,8 +20,8 @@ In CPython, the compilation from source code to bytecode involves several steps: This document outlines how these steps of the process work. This document only describes parsing in enough depth to explain what is needed -for understanding compilation. This document provides a detailed, though not -exhaustive, view of the how the entire system works. You will most likely need +for understanding compilation. This document provides a detailed, though not +exhaustive, view of the how the entire system works. You will most likely need to read some source code to have an exact understanding of all details. @@ -36,7 +36,7 @@ parsers. The grammar file for Python can be found in [Grammar/python.gram](../Grammar/python.gram). The definitions for literal tokens (such as `:`, numbers, etc.) can be found in -[Grammar/Tokens](../Grammar/Tokens). Various C files, including +[Grammar/Tokens](../Grammar/Tokens). Various C files, including [Parser/parser.c](../Parser/parser.c) are generated from these. See Also: @@ -54,7 +54,7 @@ Abstract syntax trees (AST) The abstract syntax tree (AST) is a high-level representation of the program structure without the necessity of containing the source code; -it can be thought of as an abstract representation of the source code. The +it can be thought of as an abstract representation of the source code. The specification of the AST nodes is specified using the Zephyr Abstract Syntax Definition Language (ASDL) [^1], [^2]. @@ -63,9 +63,9 @@ The definition of the AST nodes for Python is found in the file Each AST node (representing statements, expressions, and several specialized types, like list comprehensions and exception handlers) is -defined by the ASDL. Most definitions in the AST correspond to a +defined by the ASDL. Most definitions in the AST correspond to a particular source construct, such as an 'if' statement or an attribute -lookup. The definition is independent of its realization in any +lookup. The definition is independent of its realization in any particular programming language. The following fragment of the Python ASDL construct demonstrates the @@ -84,7 +84,7 @@ approach and syntax: The preceding example describes two different kinds of statements and an expression: function definitions, return statements, and yield expressions. All three kinds are considered of type `stmt` as shown by `|` separating -the various kinds. They all take arguments of various kinds and amounts. +the various kinds. They all take arguments of various kinds and amounts. Modifiers on the argument type specify the number of values needed; `?` means it is optional, `*` means 0 or more, while no modifier means only one @@ -128,47 +128,46 @@ The statement definitions above generate the following C structure type: ``` Also generated are a series of constructor functions that allocate (in -this case) a `stmt_ty` struct with the appropriate initialization. The -`kind` field specifies which component of the union is initialized. The +this case) a `stmt_ty` struct with the appropriate initialization. The +`kind` field specifies which component of the union is initialized. The `FunctionDef()` constructor function sets 'kind' to `FunctionDef_kind` and initializes the *name*, *args*, *body*, and *attributes* fields. -See also -[Green Tree Snakes - The missing Python AST docs](https://greentreesnakes.readthedocs.io/en/latest) -by Thomas Kluyver. +See also [Green Tree Snakes - The missing Python AST docs]( +https://greentreesnakes.readthedocs.io/en/latest) by Thomas Kluyver. Memory management ================= Before discussing the actual implementation of the compiler, a discussion of -how memory is handled is in order. To make memory management simple, an **arena** +how memory is handled is in order. To make memory management simple, an **arena** is used that pools memory in a single location for easy -allocation and removal. This enables the removal of explicit memory -deallocation. Because memory allocation for all needed memory in the compiler +allocation and removal. This enables the removal of explicit memory +deallocation. Because memory allocation for all needed memory in the compiler registers that memory with the arena, a single call to free the arena is all that is needed to completely free all memory used by the compiler. In general, unless you are working on the critical core of the compiler, memory -management can be completely ignored. But if you are working at either the +management can be completely ignored. But if you are working at either the very beginning of the compiler or the end, you need to care about how the arena -works. All code relating to the arena is in either +works. All code relating to the arena is in either [Include/internal/pycore_pyarena.h](../Include/internal/pycore_pyarena.h) or [Python/pyarena.c](../Python/pyarena.c). -`PyArena_New()` will create a new arena. The returned `PyArena` structure -will store pointers to all memory given to it. This does the bookkeeping of +`PyArena_New()` will create a new arena. The returned `PyArena` structure +will store pointers to all memory given to it. This does the bookkeeping of what memory needs to be freed when the compiler is finished with the memory it -used. That freeing is done with `PyArena_Free()`. This only needs to be +used. That freeing is done with `PyArena_Free()`. This only needs to be called in strategic areas where the compiler exits. As stated above, in general you should not have to worry about memory -management when working on the compiler. The technical details of memory +management when working on the compiler. The technical details of memory management have been designed to be hidden from you for most cases. -The only exception comes about when managing a PyObject. Since the rest +The only exception comes about when managing a PyObject. Since the rest of Python uses reference counting, there is extra support added -to the arena to cleanup each PyObject that was allocated. These cases -are very rare. However, if you've allocated a PyObject, you must tell +to the arena to cleanup each PyObject that was allocated. These cases +are very rare. However, if you've allocated a PyObject, you must tell the arena about it by calling `PyArena_AddPyObject()`. @@ -182,22 +181,22 @@ The AST is generated from source code using the function After some checks, a helper function in [Parser/parser.c](../Parser/parser.c) begins applying production rules on the source code it receives; converting source -code to tokens and matching these tokens recursively to their corresponding rule. The -production rule's corresponding rule function is called on every match. These rule -functions follow the format `xx_rule`. Where *xx* is the grammar rule +code to tokens and matching these tokens recursively to their corresponding rule. The +production rule's corresponding rule function is called on every match. These rule +functions follow the format `xx_rule`. Where *xx* is the grammar rule that the function handles and is automatically derived from [Grammar/python.gram](../Grammar/python.gram) by [Tools/peg_generator/pegen/c_generator.py](../Tools/peg_generator/pegen/c_generator.py). -Each rule function in turn creates an AST node as it goes along. It does this +Each rule function in turn creates an AST node as it goes along. It does this by allocating all the new nodes it needs, calling the proper AST node creation functions for any required supporting functions and connecting them as needed. -This continues until all nonterminal symbols are replaced with terminals. If an -error occurs, the rule functions backtrack and try another rule function. If +This continues until all nonterminal symbols are replaced with terminals. If an +error occurs, the rule functions backtrack and try another rule function. If there are no more rules, an error is set and the parsing ends. The AST node creation helper functions have the name `_PyAST_{xx}` -where *xx* is the AST node that the function creates. These are defined by the +where *xx* is the AST node that the function creates. These are defined by the ASDL grammar and contained in [Python/Python-ast.c](../Python/Python-ast.c) (which is generated by [Parser/asdl_c.py](../Parser/asdl_c.py) from [Parser/Python.asdl](../Parser/Python.asdl)). @@ -205,9 +204,9 @@ This all leads to a sequence of AST nodes stored in `asdl_seq` structs. To demonstrate everything explained so far, here's the rule function responsible for a simple named import statement such as -`import sys`. Note that error-checking and debugging code has been -omitted. Removed parts are represented by `...`. -Furthermore, some comments have been added for explanation. These comments +`import sys`. Note that error-checking and debugging code has been +omitted. Removed parts are represented by `...`. +Furthermore, some comments have been added for explanation. These comments may not be present in the actual code. @@ -248,13 +247,13 @@ may not be present in the actual code. To improve backtracking performance, some rules (chosen by applying a -`(memo)` flag in the grammar file) are memoized. Each rule function checks if +`(memo)` flag in the grammar file) are memoized. Each rule function checks if a memoized version exists and returns that if so, else it continues in the manner stated in the previous paragraphs. There are macros for creating and using `asdl_xx_seq *` types, where *xx* is -a type of the ASDL sequence. Three main types are defined -manually -- `generic`, `identifier` and `int`. These types are found in +a type of the ASDL sequence. Three main types are defined +manually -- `generic`, `identifier` and `int`. These types are found in [Python/asdl.c](../Python/asdl.c) and its corresponding header file [Include/internal/pycore_asdl.h](../Include/internal/pycore_asdl.h). Functions and macros for creating `asdl_xx_seq *` types are as follows: @@ -288,11 +287,11 @@ when a function needs to manipulate a generic ASDL sequence: Return the length of an `asdl_seq` or `asdl_xx_seq` Note that typed macros and functions are recommended over their untyped -counterparts. Typed macros carry out checks in debug mode and aid +counterparts. Typed macros carry out checks in debug mode and aid debugging errors caused by incorrectly casting from `void *`. If you are working with statements, you must also worry about keeping -track of what line number generated the statement. Currently the line +track of what line number generated the statement. Currently the line number is passed as the last parameter to each `stmt_ty` function. See also [PEP 617: New PEG parser for CPython](https://peps.python.org/pep-0617/). @@ -302,12 +301,12 @@ Control flow graphs =================== A **control flow graph** (often referenced by its acronym, **CFG**) is a -directed graph that models the flow of a program. A node of a CFG is +directed graph that models the flow of a program. A node of a CFG is not an individual bytecode instruction, but instead represents a sequence of bytecode instructions that always execute sequentially. Each node is called a *basic block* and must always execute from start to finish, with a single entry point at the beginning and a -single exit point at the end. If some bytecode instruction *a* needs +single exit point at the end. If some bytecode instruction *a* needs to jump to some other bytecode instruction *b*, then *a* must occur at the end of its basic block, and *b* must occur at the start of its basic block. @@ -325,8 +324,8 @@ end() The `x < 10` guard is represented by its own basic block that compares `x` with `10` and then ends in a conditional jump based on -the result of the comparison. This conditional jump allows the block -to point to both the body of the `if` and the body of the `else`. The +the result of the comparison. This conditional jump allows the block +to point to both the body of the `if` and the body of the `else`. The `if` basic block contains the `f1()` and `f2()` calls and points to the `end()` basic block. The `else` basic block contains the `g()` call and similarly points to the `end()` block. @@ -351,7 +350,7 @@ The first step is to construct the symbol table. This is implemented by `_PySymtable_Build()` in [Python/symtable.c](../Python/symtable.c). This function begins by entering the starting code block for the AST (passed-in) and then calling the proper `symtable_visit_{xx}` function (with *xx* being the -AST node type). Next, the AST tree is walked with the various code blocks that +AST node type). Next, the AST tree is walked with the various code blocks that delineate the reach of a local variable as blocks are entered and exited using `symtable_enter_block()` and `symtable_exit_block()`, respectively. @@ -361,10 +360,10 @@ These are similar to bytecode, but in some cases they are more abstract, and are resolved later into actual bytecode. The construction of this instruction sequence is handled by several functions that break the task down by various AST node types. The functions are all named `compiler_visit_{xx}` where *xx* is the name of the node -type (such as `stmt`, `expr`, etc.). Each function receives a `struct compiler *` -and `{xx}_ty` where *xx* is the AST node type. Typically these functions +type (such as `stmt`, `expr`, etc.). Each function receives a `struct compiler *` +and `{xx}_ty` where *xx* is the AST node type. Typically these functions consist of a large 'switch' statement, branching based on the kind of -node type passed to it. Simple things are handled inline in the +node type passed to it. Simple things are handled inline in the 'switch' statement with more complex transformations farmed out to other functions named `compiler_{xx}` with *xx* being a descriptive name of what is being handled. @@ -372,7 +371,7 @@ being handled. When transforming an arbitrary AST node, use the `VISIT()` macro. The appropriate `compiler_visit_{xx}` function is called, based on the value passed in for (so `VISIT({c}, expr, {node})` calls -`compiler_visit_expr({c}, {node})`). The `VISIT_SEQ()` macro is very similar, +`compiler_visit_expr({c}, {node})`). The `VISIT_SEQ()` macro is very similar, but is called on AST node sequences (those values that were created as arguments to a node that used the '*' modifier). @@ -416,7 +415,7 @@ line in the source code. There are several helper functions that will emit pseudo-instructions and are named `compiler_{xx}()` where *xx* is what the function helps -with (`list`, `boolop`, etc.). A rather useful one is `compiler_nameop()`. +with (`list`, `boolop`, etc.). A rather useful one is `compiler_nameop()`. This function looks up the scope of a variable and, based on the expression context, emits the proper opcode to load, store, or delete the variable. @@ -462,15 +461,15 @@ Important files Reads in an ASDL description and parses it into an AST that describes it. * [Parser/asdl_c.py](../Parser/asdl_c.py): - Generate C code from an ASDL description. Generates + Generate C code from an ASDL description. Generates [Python/Python-ast.c](../Python/Python-ast.c) and [Include/internal/pycore_ast.h](../Include/internal/pycore_ast.h). * [Parser/parser.c](../Parser/parser.c): - The new PEG parser introduced in Python 3.9. Generated by + The new PEG parser introduced in Python 3.9. Generated by [Tools/peg_generator/pegen/c_generator.py](../Tools/peg_generator/pegen/c_generator.py) from the grammar [Grammar/python.gram](../Grammar/python.gram). - Creates the AST from source code. Rule functions for their corresponding production + Creates the AST from source code. Rule functions for their corresponding production rules are found here. * [Parser/peg_api.c](../Parser/peg_api.c): @@ -479,7 +478,7 @@ Important files * [Parser/pegen.c](../Parser/pegen.c): Contains helper functions which are used by functions in - [Parser/parser.c](../Parser/parser.c) to construct the AST. Also contains + [Parser/parser.c](../Parser/parser.c) to construct the AST. Also contains helper functions which help raise better error messages when parsing source code. * [Parser/pegen.h](../Parser/pegen.h): @@ -489,7 +488,7 @@ Important files * [Python/](../Python) * [Python/Python-ast.c](../Python/Python-ast.c): - Creates C structs corresponding to the ASDL types. Also contains code for + Creates C structs corresponding to the ASDL types. Also contains code for marshalling AST nodes (core ASDL types have marshalling code in [Python/asdl.c](../Python/asdl.c)). File automatically generated by [Parser/asdl_c.py](../Parser/asdl_c.py). @@ -500,7 +499,7 @@ Important files * [Python/asdl.c](../Python/asdl.c): Contains code to handle the ASDL sequence type. Also has code to handle marshalling the core ASDL types, such as number - and identifier. Used by [Python/Python-ast.c](../Python/Python-ast.c) + and identifier. Used by [Python/Python-ast.c](../Python/Python-ast.c) for marshalling AST nodes. * [Python/ast.c](../Python/ast.c): @@ -610,9 +609,9 @@ References ========== [^1]: Daniel C. Wang, Andrew W. Appel, Jeff L. Korn, and Chris -S. Serra. `The Zephyr Abstract Syntax Description Language.`_ -In Proceedings of the Conference on Domain-Specific Languages, -pp. 213--227, 1997. + S. Serra. `The Zephyr Abstract Syntax Description Language.`_ + In Proceedings of the Conference on Domain-Specific Languages, + pp. 213--227, 1997. [^2]: The Zephyr Abstract Syntax Description Language.: -https://www.cs.princeton.edu/research/techreps/TR-554-97 + https://www.cs.princeton.edu/research/techreps/TR-554-97 diff --git a/InternalDocs/exception_handling.md b/InternalDocs/exception_handling.md index 322f83a4ef59d0..28589787e1fad7 100644 --- a/InternalDocs/exception_handling.md +++ b/InternalDocs/exception_handling.md @@ -190,5 +190,5 @@ Exception Chaining Implementation refers to setting the `__context__` and `__cause__` fields of an exception as it is being raised. The `__context__` field is set by `_PyErr_SetObject()` in [Python/errors.c](../Python/errors.c) (which is ultimately called by all -`PyErr_Set*()` functions). The `__cause__` field (explicit chaining) is set by +`PyErr_Set*()` functions). The `__cause__` field (explicit chaining) is set by the `RAISE_VARARGS` bytecode. diff --git a/InternalDocs/frames.md b/InternalDocs/frames.md index efdf9927e7a25e..2598873ca98479 100644 --- a/InternalDocs/frames.md +++ b/InternalDocs/frames.md @@ -125,10 +125,10 @@ the next instruction to be executed. During a call to a python function, to see in an exception traceback. The `return_offset` field determines where a `RETURN` should go in the caller, -relative to `instr_ptr`. It is only meaningful to the callee, so it needs to +relative to `instr_ptr`. It is only meaningful to the callee, so it needs to be set in any instruction that implements a call (to a Python function), including CALL, SEND and BINARY_SUBSCR_GETITEM, among others. If there is no -callee, then return_offset is meaningless. It is necessary to have a separate +callee, then return_offset is meaningless. It is necessary to have a separate field for the return offset because (1) if we apply this offset to `instr_ptr` while executing the `RETURN`, this is too early and would lose us information about the previous instruction which we could need for introspecting and diff --git a/InternalDocs/garbage_collector.md b/InternalDocs/garbage_collector.md index 8f4b67f8320d19..9e01a5864e33f8 100644 --- a/InternalDocs/garbage_collector.md +++ b/InternalDocs/garbage_collector.md @@ -55,7 +55,7 @@ Starting in version 3.13, CPython contains two GC implementations: performing a collection for thread safety. Both implementations use the same basic algorithms, but operate on different -data structures. See the section on +data structures. See the section on [Differences between GC implementations](#Differences-between-GC-implementations) for the details. @@ -64,7 +64,7 @@ Memory layout and object structure ================================== The garbage collector requires additional fields in Python objects to support -garbage collection. These extra fields are different in the default and the +garbage collection. These extra fields are different in the default and the free-threaded builds. @@ -111,11 +111,11 @@ that in the [Optimization: incremental collection](#Optimization-incremental-col they are also reused to fulfill other purposes when the full doubly linked list structure is not needed as a memory optimization. -Doubly linked lists are used because they efficiently support the most frequently required operations. In +Doubly linked lists are used because they efficiently support the most frequently required operations. In general, the collection of all objects tracked by GC is partitioned into disjoint sets, each in its own -doubly linked list. Between collections, objects are partitioned into "generations", reflecting how -often they've survived collection attempts. During collections, the generation(s) being collected -are further partitioned into, for example, sets of reachable and unreachable objects. Doubly linked lists +doubly linked list. Between collections, objects are partitioned into "generations", reflecting how +often they've survived collection attempts. During collections, the generation(s) being collected +are further partitioned into, for example, sets of reachable and unreachable objects. Doubly linked lists support moving an object from one partition to another, adding a new object, removing an object entirely (objects tracked by GC are most often reclaimed by the refcounting system when GC isn't running at all!), and merging partitions, all with a small constant number of pointer updates. @@ -128,7 +128,7 @@ GC for the free-threaded build In the free-threaded build, Python objects contain a 1-byte field `ob_gc_bits` that is used to track garbage collection related state. The field exists in all objects, including ones that do not support cyclic -garbage collection. The field is used to identify objects that are tracked +garbage collection. The field is used to identify objects that are tracked by the collector, ensure that finalizers are called only once per object, and, during garbage collection, differentiate reachable vs. unreachable objects. @@ -192,7 +192,7 @@ the interpreter create cycles everywhere. Some notable examples: have internal links to themselves. To correctly dispose of these objects once they become unreachable, they need -to be identified first. To understand how the algorithm works, let’s take +to be identified first. To understand how the algorithm works, let’s take the case of a circular linked list which has one link referenced by a variable `A`, and one self-referencing object which is completely unreachable: @@ -220,15 +220,15 @@ unreachable: 2 ``` -The GC starts with a set of candidate objects it wants to scan. In the +The GC starts with a set of candidate objects it wants to scan. In the default build, these "objects to scan" might be all container objects or a -smaller subset (or "generation"). In the free-threaded build, the collector +smaller subset (or "generation"). In the free-threaded build, the collector always scans all container objects. -The objective is to identify all the unreachable objects. The collector does +The objective is to identify all the unreachable objects. The collector does this by identifying reachable objects; the remaining objects must be -unreachable. The first step is to identify all of the "to scan" objects that -are **directly** reachable from outside the set of candidate objects. These +unreachable. The first step is to identify all of the "to scan" objects that +are **directly** reachable from outside the set of candidate objects. These objects have a refcount larger than the number of incoming references from within the candidate set. @@ -241,7 +241,7 @@ interpreter will not modify the real reference count field. ![gc-image1](images/python-cyclic-gc-1-new-page.png) The GC then iterates over all containers in the first list and decrements by one the -`gc_ref` field of any other object that container is referencing. Doing +`gc_ref` field of any other object that container is referencing. Doing this makes use of the `tp_traverse` slot in the container class (implemented using the C API or inherited by a superclass) to know what objects are referenced by each container. After all the objects have been scanned, only the objects that have @@ -273,7 +273,7 @@ When the GC encounters an object which is reachable (`gc_ref > 0`), it traverses its references using the `tp_traverse` slot to find all the objects that are reachable from it, moving them to the end of the list of reachable objects (where they started originally) and setting its `gc_ref` field to 1. This is what happens -to `link_2` and `link_3` below as they are reachable from `link_1`. From the +to `link_2` and `link_3` below as they are reachable from `link_1`. From the state in the previous image and after examining the objects referred to by `link_1` the GC knows that `link_3` is reachable after all, so it is moved back to the original list and its `gc_ref` field is set to 1 so that if the GC visits it again, @@ -293,7 +293,7 @@ list are really unreachable and can thus be garbage collected. Pragmatically, it's important to note that no recursion is required by any of this, and neither does it in any other way require additional memory proportional to the -number of objects, number of pointers, or the lengths of pointer chains. Apart from +number of objects, number of pointers, or the lengths of pointer chains. Apart from `O(1)` storage for internal C needs, the objects themselves contain all the storage the GC algorithms require. @@ -317,8 +317,8 @@ list. So instead of not moving at all, the reachable objects B and A are each moved twice. Why is this a win? A straightforward algorithm to move the reachable objects instead would move A, B, and C once each. The key is that this dance leaves the objects in -order C, B, A - it's reversed from the original order. On all *subsequent* scans, -none of them will move. Since most objects aren't in cycles, this can save an +order C, B, A - it's reversed from the original order. On all *subsequent* scans, +none of them will move. Since most objects aren't in cycles, this can save an unbounded number of moves across an unbounded number of later collections. The only time the cost can be higher is the first time the chain is scanned. @@ -331,7 +331,7 @@ follows these steps in order: 1. Handle and clear weak references (if any). Weak references to unreachable objects are set to `None`. If the weak reference has an associated callback, the callback - is enqueued to be called once the clearing of weak references is finished. We only + is enqueued to be called once the clearing of weak references is finished. We only invoke callbacks for weak references that are themselves reachable. If both the weak reference and the pointed-to object are unreachable we do not execute the callback. This is partly for historical reasons: the callback could resurrect an unreachable @@ -490,7 +490,7 @@ to the size of the data, often a word or multiple thereof. This discrepancy leaves a few of the least significant bits of the pointer unused, which can be used for tags or to keep other information – most often as a bit field (each bit a separate tag) – as long as code that uses the pointer masks out these -bits before accessing memory. For example, on a 32-bit architecture (for both +bits before accessing memory. For example, on a 32-bit architecture (for both addresses and word size), a word is 32 bits = 4 bytes, so word-aligned addresses are always a multiple of 4, hence end in `00`, leaving the last 2 bits available; while on a 64-bit architecture, a word is 64 bits = 8 bytes, so @@ -519,10 +519,10 @@ of `PyGC_Head` discussed in the `Memory layout and object structure`_ section: - The `_gc_next` field is used as the "next" pointer to maintain the doubly linked list but during collection its lowest bit is used to keep the `NEXT_MASK_UNREACHABLE` flag that indicates if an object is tentatively - unreachable during the cycle detection algorithm. This is a drawback to using only + unreachable during the cycle detection algorithm. This is a drawback to using only doubly linked lists to implement partitions: while most needed operations are constant-time, there is no efficient way to determine which partition an object is - currently in. Instead, when that's needed, ad hoc tricks (like the + currently in. Instead, when that's needed, ad hoc tricks (like the `NEXT_MASK_UNREACHABLE` flag) are employed. Optimization: delayed untracking containers @@ -581,29 +581,29 @@ structure, while the free-threaded build implementation does not use that data structure. - The default build implementation stores all tracked objects in a doubly - linked list using `PyGC_Head`. The free-threaded build implementation + linked list using `PyGC_Head`. The free-threaded build implementation instead relies on the embedded mimalloc memory allocator to scan the heap for tracked objects. - The default build implementation uses `PyGC_Head` for the unreachable - object list. The free-threaded build implementation repurposes the + object list. The free-threaded build implementation repurposes the `ob_tid` field to store a unreachable objects linked list. - The default build implementation stores flags in the `_gc_prev` field of - `PyGC_Head`. The free-threaded build implementation stores these flags + `PyGC_Head`. The free-threaded build implementation stores these flags in `ob_gc_bits`. The default build implementation relies on the [global interpreter lock](https://docs.python.org/3/glossary.html#term-global-interpreter-lock) -for thread safety. The free-threaded build implementation has two "stop the +for thread safety. The free-threaded build implementation has two "stop the world" pauses, in which all other executing threads are temporarily paused so that the GC can safely access reference counts and object attributes. -The default build implementation is a generational collector. The +The default build implementation is a generational collector. The free-threaded build is non-generational; each collection scans the entire heap. - Keeping track of object generations is simple and inexpensive in the default - build. The free-threaded build relies on mimalloc for finding tracked + build. The free-threaded build relies on mimalloc for finding tracked objects; identifying "young" objects without scanning the entire heap would be more difficult. diff --git a/InternalDocs/interpreter.md b/InternalDocs/interpreter.md index 63ae51dcd3eb7a..ab149e43471072 100644 --- a/InternalDocs/interpreter.md +++ b/InternalDocs/interpreter.md @@ -31,7 +31,7 @@ It also has a reference to the `CodeObject` itself. In addition to the frame, `_PyEval_EvalFrame()` also receives a [`Thread State`](https://docs.python.org/3/c-api/init.html#c.PyThreadState) object, `tstate`, which includes things like the exception state and the -recursion depth. The thread state also provides access to the per-interpreter +recursion depth. The thread state also provides access to the per-interpreter state (`tstate->interp`), which has a pointer to the per-runtime (that is, truly global) state (`tstate->interp->runtime`). @@ -130,7 +130,7 @@ The size of the inline cache for a particular instruction is fixed by its `opcod Moreover, the inline cache size for all instructions in a [family of specialized/specializable instructions](adaptive.md) (for example, `LOAD_ATTR`, `LOAD_ATTR_SLOT`, `LOAD_ATTR_MODULE`) must all be -the same. Cache entries are reserved by the compiler and initialized with zeros. +the same. Cache entries are reserved by the compiler and initialized with zeros. Although they are represented by code units, cache entries do not conform to the `opcode` / `oparg` format. @@ -138,7 +138,7 @@ If an instruction has an inline cache, the layout of its cache is described by a `struct` definition in (`pycore_code.h`)[../Include/internal/pycore_code.h]. This allows us to access the cache by casting `next_instr` to a pointer to this `struct`. The size of such a `struct` must be independent of the machine architecture, word size -and alignment requirements. For a 32-bit field, the `struct` should use `_Py_CODEUNIT field[2]`. +and alignment requirements. For a 32-bit field, the `struct` should use `_Py_CODEUNIT field[2]`. The instruction implementation is responsible for advancing `next_instr` past the inline cache. For example, if an instruction's inline cache is four bytes (that is, two code units) in size, @@ -210,12 +210,12 @@ In 3.10 and before, this was the case even when a Python function called another Python function: The `CALL` opcode would call the `tp_call` dispatch function of the callee, which would extract the code object, create a new frame for the call -stack, and then call back into the interpreter. This approach is very general +stack, and then call back into the interpreter. This approach is very general but consumes several C stack frames for each nested Python call, thereby increasing the risk of an (unrecoverable) C stack overflow. Since 3.11, the `CALL` instruction special-cases function objects to "inline" -the call. When a call gets inlined, a new frame gets pushed onto the call +the call. When a call gets inlined, a new frame gets pushed onto the call stack and the interpreter "jumps" to the start of the callee's bytecode. When an inlined callee executes a `RETURN_VALUE` instruction, the frame is popped off the call stack and the interpreter returns to its caller, @@ -248,12 +248,12 @@ In this case we allocate a proper `PyFrameObject` and initialize it from the Things get more complicated when generators are involved, since those do not follow the push/pop model. This includes async functions, which are based on -the same mechanism. A generator object has space for a `_PyInterpreterFrame` +the same mechanism. A generator object has space for a `_PyInterpreterFrame` structure, including the variable-size part (used for locals and the eval stack). When a generator (or async) function is first called, a special opcode `RETURN_GENERATOR` is executed, which is responsible for creating the -generator object. The generator object's `_PyInterpreterFrame` is initialized -with a copy of the current stack frame. The current stack frame is then popped +generator object. The generator object's `_PyInterpreterFrame` is initialized +with a copy of the current stack frame. The current stack frame is then popped off the frame stack and the generator object is returned. (Details differ depending on the `is_entry` flag.) When the generator is resumed, the interpreter pushes its `_PyInterpreterFrame` @@ -317,9 +317,9 @@ With a new bytecode you must also change what is called the "magic number" for .pyc files: bump the value of the variable `MAGIC_NUMBER` in [`Lib/importlib/_bootstrap_external.py`](../Lib/importlib/_bootstrap_external.py). Changing this number will lead to all .pyc files with the old `MAGIC_NUMBER` -to be recompiled by the interpreter on import. Whenever `MAGIC_NUMBER` is +to be recompiled by the interpreter on import. Whenever `MAGIC_NUMBER` is changed, the ranges in the `magic_values` array in -[`PC/launcher.c`](../PC/launcher.c) may also need to be updated. Changes to +[`PC/launcher.c`](../PC/launcher.c) may also need to be updated. Changes to [`Lib/importlib/_bootstrap_external.py`](../Lib/importlib/_bootstrap_external.py) will take effect only after running `make regen-importlib`. @@ -333,12 +333,12 @@ will take effect only after running `make regen-importlib`. > On Windows, running the `./build.bat` script will automatically > regenerate the required files without requiring additional arguments. -Finally, you need to introduce the use of the new bytecode. Update +Finally, you need to introduce the use of the new bytecode. Update [`Python/codegen.c`](../Python/codegen.c) to emit code with this bytecode. Optimizations in [`Python/flowgraph.c`](../Python/flowgraph.c) may also -need to be updated. If the new opcode affects a control flow or the block +need to be updated. If the new opcode affects a control flow or the block stack, you may have to update the `frame_setlineno()` function in -[`Objects/frameobject.c`](../Objects/frameobject.c). It may also be necessary +[`Objects/frameobject.c`](../Objects/frameobject.c). It may also be necessary to update [`Lib/dis.py`](../Lib/dis.py) if the new opcode interprets its argument in a special way (like `FORMAT_VALUE` or `MAKE_FUNCTION`). @@ -347,12 +347,12 @@ is already in existence and you do not change the magic number, make sure to delete your old .py(c|o) files! Even though you will end up changing the magic number if you change the bytecode, while you are debugging your work you may be changing the bytecode output without constantly bumping up the -magic number. This can leave you with stale .pyc files that will not be +magic number. This can leave you with stale .pyc files that will not be recreated. Running `find . -name '*.py[co]' -exec rm -f '{}' +` should delete all .pyc files you have, forcing new ones to be created and thus allow you test out your -new bytecode properly. Run `make regen-importlib` for updating the -bytecode of frozen importlib files. You have to run `make` again after this +new bytecode properly. Run `make regen-importlib` for updating the +bytecode of frozen importlib files. You have to run `make` again after this to recompile the generated C files. Additional resources diff --git a/InternalDocs/parser.md b/InternalDocs/parser.md index 942c940236c3fc..445b866fc0cb96 100644 --- a/InternalDocs/parser.md +++ b/InternalDocs/parser.md @@ -56,11 +56,11 @@ an input string as its argument, and yields one of the following results: Note that "failure" results do not imply that the program is incorrect, nor do they necessarily mean that the parsing has failed. Since the choice operator is -ordered, a failure very often merely indicates "try the following option". A +ordered, a failure very often merely indicates "try the following option". A direct implementation of a PEG parser as a recursive descent parser will present exponential time performance in the worst case, because PEG parsers have infinite lookahead (this means that they can consider an arbitrary number of -tokens before deciding for a rule). Usually, PEG parsers avoid this exponential +tokens before deciding for a rule). Usually, PEG parsers avoid this exponential time complexity with a technique called ["packrat parsing"](https://pdos.csail.mit.edu/~baford/packrat/thesis/) which not only loads the entire program in memory before parsing it but also @@ -447,7 +447,7 @@ parser (the one used by the interpreter) just execute: $ make regen-pegen ``` -using the `Makefile` in the main directory. If you are on Windows you can +using the `Makefile` in the main directory. If you are on Windows you can use the Visual Studio project files to regenerate the parser or to execute: ```dos