Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conflict between pragmatic locations and pure spans #94758

Open
markshannon opened this issue Jul 11, 2022 · 1 comment
Open

Conflict between pragmatic locations and pure spans #94758

markshannon opened this issue Jul 11, 2022 · 1 comment
Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@markshannon
Copy link
Member

markshannon commented Jul 11, 2022

The PEG parser produces locations for an AST that span the entire AST node. This is well defined, and has a number of desirable properties such as the span of a node including all its children's spans.

Unfortunately, this does not produce good locations for tracing and debugging, so we transform some locations in the compiler.
This is problematic as it means that tools producing as AST (e.g. pytest) need to second guess what transformations that the compiler will make.

I proposed moving these transformations in the parser in #94694, but as @pablogsal points out, this means that tools which consume the AST (e.g. type checkers) need to undo the transformations to get the original spans back.

The core problem is that there are tools that consume the AST produced by the parser, and tools that produce ASTs for consumption by the compiler, and they want subtly different ASTs.

One possible fix for this is to add a keyword argument to compile() to specify whether it should modify the locations. This would allow pytest, etc. to pass location information through compile() unmodified when needed.

@markshannon markshannon added type-bug An unexpected behavior, bug, or error 3.12 bugs and security fixes labels Jul 11, 2022
@brandtbucher
Copy link
Member

Related: #93691

@iritkatriel iritkatriel added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants