Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #139: newlines in async for comprehension #140

Merged
merged 1 commit into from
Jun 29, 2020
Merged

Fix #139: newlines in async for comprehension #140

merged 1 commit into from
Jun 29, 2020

Conversation

Kazy
Copy link
Contributor

@Kazy Kazy commented Jun 28, 2020

I decided to take a stab at this. I don't know if this is the right fix, and if the test is the right place (should it be in test_tokenize.py ?).

I traced back the difference between the parsing of a for comprehension and an async one to the tokenization. In a normal for-comprehension, the ] token has \n as a prefix, while in the case of async for it has NEWLINE token which isn't part of the DFA transitions and is thus a SyntaxError.

Removing async from the ALWAYS_BREAK_TOKENS fixes this, all the tests are passing. I don't know if this can have consequences in other uses of async. Codes like this are still rejected:

async
def foo():
   ....

# or
async
    def foo():
        ...

Now it also correctly parses all possible insertion of newlines in the async for-comprehension, just like a sync one.

async def foo():
    [1 
    async
    for
    _
    in
    range(5)
    ]

@isidentical isidentical linked an issue Jun 28, 2020 that may be closed by this pull request
test/test_pgen2.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@isidentical isidentical left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and Thank you for your contribution. (for a better look on the test, if you want you can move the last triple quotes to the upper line (so it wont break the indentation)).

@Kazy
Copy link
Contributor Author

Kazy commented Jun 29, 2020

(for a better look on the test, if you want you can move the last triple quotes to the upper line (so it wont break the indentation))

Done !

@isidentical isidentical merged commit c88a267 into davidhalter:master Jun 29, 2020
@Kazy Kazy deleted the fix-139-async-for-newline branch June 29, 2020 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Syntax error in async for comprehension depending on newline
3 participants