-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Consider 2-character EOL before line continuation (#12035)
## Summary This PR fixes a bug introduced in #12008 which didn't consider the two character newline after the line continuation character. For example, consider the following code highlighted with whitespaces: ```py call(foo # comment \\r\n \r\n def bar():\r\n ....pass\r\n ``` The lexer is at `def` when it's running the re-lexing logic and trying to move back to a newline character. It encounters `\n` and it's being escaped (incorrect) but `\r` is being escaped, so it moves the lexer to `\n` character. This creates an overlap in token ranges which causes the panic. ``` Name 0..4 Lpar 4..5 Name 5..8 Comment 9..20 NonLogicalNewline 20..22 <-- overlap between Newline 21..22 <-- these two tokens NonLogicalNewline 22..23 Def 23..26 ... ``` fixes: #12028 ## Test Plan Add a test case with line continuation and windows style newline character.
- Loading branch information
1 parent
7cb2619
commit 47c9ed0
Showing
4 changed files
with
125 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 4 additions & 0 deletions
4
crates/ruff_python_parser/resources/invalid/re_lexing/line_continuation_windows_eol.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
call(a, b, # comment \ | ||
|
||
def bar(): | ||
pass |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
89 changes: 89 additions & 0 deletions
89
...on_parser/tests/snapshots/invalid_syntax@re_lexing__line_continuation_windows_eol.py.snap
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,89 @@ | ||
--- | ||
source: crates/ruff_python_parser/tests/fixtures.rs | ||
input_file: crates/ruff_python_parser/resources/invalid/re_lexing/line_continuation_windows_eol.py | ||
--- | ||
## AST | ||
|
||
``` | ||
Module( | ||
ModModule { | ||
range: 0..46, | ||
body: [ | ||
Expr( | ||
StmtExpr { | ||
range: 0..10, | ||
value: Call( | ||
ExprCall { | ||
range: 0..10, | ||
func: Name( | ||
ExprName { | ||
range: 0..4, | ||
id: "call", | ||
ctx: Load, | ||
}, | ||
), | ||
arguments: Arguments { | ||
range: 4..10, | ||
args: [ | ||
Name( | ||
ExprName { | ||
range: 5..6, | ||
id: "a", | ||
ctx: Load, | ||
}, | ||
), | ||
Name( | ||
ExprName { | ||
range: 8..9, | ||
id: "b", | ||
ctx: Load, | ||
}, | ||
), | ||
], | ||
keywords: [], | ||
}, | ||
}, | ||
), | ||
}, | ||
), | ||
FunctionDef( | ||
StmtFunctionDef { | ||
range: 26..46, | ||
is_async: false, | ||
decorator_list: [], | ||
name: Identifier { | ||
id: "bar", | ||
range: 30..33, | ||
}, | ||
type_params: None, | ||
parameters: Parameters { | ||
range: 33..35, | ||
posonlyargs: [], | ||
args: [], | ||
vararg: None, | ||
kwonlyargs: [], | ||
kwarg: None, | ||
}, | ||
returns: None, | ||
body: [ | ||
Pass( | ||
StmtPass { | ||
range: 42..46, | ||
}, | ||
), | ||
], | ||
}, | ||
), | ||
], | ||
}, | ||
) | ||
``` | ||
## Errors | ||
|
||
| | ||
1 | call(a, b, # comment \ | ||
2 | / | ||
3 | | def bar(): | ||
| |_^ Syntax Error: Expected ')', found newline | ||
4 | pass | ||
| |