Fix escape sequences in Python's raw strings #1508
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@tuxu Sorry again about the regression. How does this look? |
pyrmont
commented
Apr 24, 2020
tuxu
reviewed
Apr 25, 2020
There was a problem hiding this comment.
Thanks a lot for the fast fix, @pyrmont!
I stumbled upon a section in the Python manual about string literals and ran your changes over a couple more raw strings. Looks good to me 👍
|
Looks great, let's get this merged :) |
tueksta
approved these changes
May 12, 2020
pyrmont
added a commit
that referenced
this pull request
May 12, 2020
The use of a ternary in code added to the Python lexer as part of the #1508 pull request results in a RuboCop warning concerning parentheses and grouped expressions. This commit fixes that warning.
mattt
pushed a commit
to NSHipster/rouge
that referenced
this pull request
May 21, 2020
Version 3.18.0 of Rouge included a new mechanism for handling strings in the Python lexer. One of the consequences of that change was that raw strings would break if they included 'invalid' escape sequences. This is a mistake as raw strings do not have 'invalid' escape sequences. This commit fixes this error by changing the approach that the Python lexer takes to escape sequences more generally. Rather than dividing recognised and unrecognised escape sequences into 'valid' and 'invalid', it simply treats unrecognised escape sequences as ordinary strings. The result of this is that all escape sequences in raw strings produce `Str` tokens. In other types of strings, recognised escape sequences produce `Str::Escape` tokens and unrecognised escape sequences produce `Str` tokens.
mattt
pushed a commit
to NSHipster/rouge
that referenced
this pull request
May 21, 2020
The use of a ternary in code added to the Python lexer as part of the rouge-ruby#1508 pull request results in a RuboCop warning concerning parentheses and grouped expressions. This commit fixes that warning.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Version 3.18.0 of Rouge included a new mechanism for handling strings in the Python lexer. One of the consequences of that change was that raw strings would break if they included 'invalid' escape sequences. This is a mistake as raw strings do not have 'invalid' escape sequences.
This PR fixes that error by adding a special state for escape sequences with raw strings. A separate state is still necessary because
\can be used in raw strings to prevent the particular string's delimiter from matching.This fixes #1507.