Fix #139: newlines in async for comprehension #140
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I decided to take a stab at this. I don't know if this is the right fix, and if the test is the right place (should it be in
test_tokenize.py
?).I traced back the difference between the parsing of a for comprehension and an async one to the tokenization. In a normal for-comprehension, the
]
token has\n
as a prefix, while in the case ofasync for
it hasNEWLINE
token which isn't part of the DFA transitions and is thus aSyntaxError
.Removing
async
from theALWAYS_BREAK_TOKENS
fixes this, all the tests are passing. I don't know if this can have consequences in other uses ofasync
. Codes like this are still rejected:Now it also correctly parses all possible insertion of newlines in the async for-comprehension, just like a sync one.