Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flake8-type-checking] Support auto-quoting when annotations contain quotes #11811

Merged
merged 23 commits into from
Oct 23, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
65a38e9
test: add test case for auto quoting
Glyphack Jun 9, 2024
c8df851
feat: quote annotations that contain quotes
Glyphack Jun 9, 2024
d97e2bd
test: update test for auto quoting
Glyphack Jun 10, 2024
8ca1f5c
refactor: cargo fmt
Glyphack Jun 10, 2024
29e0d4a
refactor: remove unused parameter
Glyphack Jun 10, 2024
1b58828
Update crates/ruff_linter/src/rules/flake8_type_checking/helpers.rs
Glyphack Jun 11, 2024
14f51e3
test: test cases with same and opposite quotes
Glyphack Jun 11, 2024
e8c0f90
Only flip quotes if there is no Literal or Annotation
Glyphack Aug 23, 2024
c6ef3db
Try transformer
Glyphack Aug 24, 2024
75f4407
Remove unnecessary quotes in annotation expressions
Glyphack Aug 25, 2024
3a253f6
Update crates/ruff_linter/src/rules/flake8_type_checking/helpers.rs
Glyphack Sep 11, 2024
bf363e2
Update crates/ruff_linter/src/rules/flake8_type_checking/helpers.rs
Glyphack Sep 11, 2024
f27ae89
Apply comments
Glyphack Sep 11, 2024
2b58cbb
Fix clippy
Glyphack Sep 11, 2024
fbf0bb1
Add test cases for quoting annotations in Literal and Annotated
Glyphack Sep 12, 2024
5079a8c
Revert max iterations count
Glyphack Sep 16, 2024
f7e19b4
TEMP Remove problematic case from tests
Glyphack Oct 14, 2024
26081a6
Remove quote style
Glyphack Oct 14, 2024
5c4b4df
Remove import
Glyphack Oct 14, 2024
d4602a2
Add case for attributes in subscript
Glyphack Oct 14, 2024
114edd0
Merge branch 'main' into auto-quote-annotation-quotes
dhruvmanila Oct 23, 2024
ff973d4
Use semantic model to resolve imports; update tests
dhruvmanila Oct 23, 2024
2c10896
Rename `QuoteAnnotation` -> `QuoteAnnotator`
dhruvmanila Oct 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
from typing import Literal


def f():
from pandas import DataFrame

Expand Down Expand Up @@ -97,3 +100,83 @@ def f():

def func(self) -> DataFrame | list[Series]:
pass

def f():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I know why the test is failing within the current MAX_ITERATIONS value. All of the different test cases are within the same function here so when some of them tries to add the if TYPE_CHECKING block along with the TYPE_CHECKING import, the edits collide and thus it won't be applied which will then require another iteration. We should split the test cases, grouping them based on similarity, in similar way as done in the top half of the file.

Copy link
Contributor Author

@Glyphack Glyphack Oct 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. I think I can fix that easily. There's just another problem I encountered. Aside from the new tests I've written the old ones also have this problem that each time the type annotation is changed a new import is added and the duplicate imports I think are removed in the end. So now the file fails again.

---- rules::flake8_type_checking::tests::quote::rule_typingonlythirdpartyimport_path_new_quote_py_expects stdout ----
thread 'rules::flake8_type_checking::tests::quote::rule_typingonlythirdpartyimport_path_new_quote_py_expects' panicked at crates/ruff_linter/src/test.rs:167:17:
Failed to converge after 10 iterations. This likely indicates a bug in the implementation of the fix. Last diagnostics:
quote.py:17:24: TCH002 [*] Move third-party import `pandas.DataFrame` into a type-checking block
   |
16 | def f():
17 |     from pandas import DataFrame
   |                        ^^^^^^^^^ TCH002
18 |
19 |     def baz() -> DataFrame:
   |
   = help: Move into type-checking block

ℹ Unsafe fix
3  3  | if TYPE_CHECKING:
4  4  |     from pandas import DataFrame
5  5  |     from pandas import DataFrame
   6  |+    from pandas import DataFrame
6  7  |     import pandas as pd
7  8  |     import pandas as pd
8  9  |     import pandas as pd
--------------------------------------------------------------------------------
14 15 |
15 16 |
16 17 | def f():
17    |-    from pandas import DataFrame
18 18 |
19    |-    def baz() -> DataFrame:
   19 |+    def baz() -> "DataFrame":
20 20 |         ...
21 21 |
22 22 |

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It can be reproduced by just duplicating one of the functions multiple times in master:

def f():
    from pandas import DataFrame

    def baz() -> DataFrame[int]:
        ...

def f():
    from pandas import DataFrame

    def baz() -> DataFrame[int]:
        ...

def f():
    from pandas import DataFrame

    def baz() -> DataFrame[int]:
        ...

My guess is that two rules are conflicting here. One is moving the import to type checking block but since now that import is in type checking block another function later in the quote.py is using that import and it is resolved when the symbol is looked up here

I'm not sure if I'm right because the import should be resolved to the import inside the function itself. It's my guess. I will investigate this later after resolving other issues.
For now I'm going to create a new file that will avoid this issue. Moving my new test cases to a file works fine.

from django.contrib.auth.models import AbstractBaseUser
from typing import Annotated

def test_remove_inner_quotes_double(self, user: AbstractBaseUser["int"], view: "type[CondorBaseViewSet]"):
pass

def test_remove_inner_quotes_single(self, user: AbstractBaseUser['int']):
pass

def test_remove_inner_quotes_mixed(self, user: AbstractBaseUser['int', "str"]):
pass

def test_literal_annotation_args_contain_quotes(self, type1: AbstractBaseUser[Literal["user", "admin"], Annotated["int", "1", 2]]):
pass

def test_union_contain_inner_quotes(self, type1: AbstractBaseUser["int" | Literal["int"]]):
pass

def test_literal_cannot_remove(user: AbstractBaseUser[Literal['user', "admin"]]):
pass

def test_literal_cannot_remove2(user: AbstractBaseUser[Literal['int'], str]):
pass

def test_type_literal_mixed_quotes(view: type["CondorBaseViewSet"], role: Literal["admin", 'user']):
pass

def test_mixed_quotes_literal(user: AbstractBaseUser[Literal['user'], "int"]):
pass

def test_annotated_literal_mixed_quotes(user: Annotated[str, "max_length=20", Literal['user', "admin"]]):
pass


def test_type_annotated_mixed(view: type["CondorBaseViewSet"], data: Annotated[int, "min_value=1", Literal["admin", 'superuser']]):
pass

def test_union_literal_mixed_quotes(user: Union[Literal['active', "inactive"], str]):
pass

def test_callable_literal_mixed_quotes(callable_fn: Callable[["int", Literal['admin', "user"]], 'bool']):
pass

def test_callable_annotated_literal(callable_fn: Callable[[int, Annotated[str, Literal['active', "inactive"]]], bool]):
pass

def test_generic_alias_literal_mixed_quotes(user: Union["List[int]", Literal['admin', "user"], 'Dict[str, Any]']):
pass

def test_literal_no_inner_removal(arg: Literal["user", "admin"]):
pass

def test_quoted_annotation_literal(arg: "Literal['user', 'admin']"):
pass

def test_annotated_with_expression(arg: Annotated[str, "min_length=3"]):
pass

def test_union_literal_no_removal(arg: Union[Literal["true", "false"], None]):
pass

def test_optional_literal_no_removal(arg: Optional[Literal["red", "blue"]]):
pass

def test_type_alias_with_removal(arg: "List[Dict[str, int]]"):
pass

def test_final_literal_no_removal(arg: Final[Literal["open", "close"]]):
pass

def test_callable_literal_no_removal(arg: Callable[[], Literal["success", "failure"]]):
pass

def test_classvar_union_literal_no_removal(arg: ClassVar[Union[Literal["yes", "no"], None]]):
pass

def test_tuple_literal_no_removal(arg: tuple[Literal["apple", "orange"], int]):
pass
133 changes: 113 additions & 20 deletions crates/ruff_linter/src/rules/flake8_type_checking/helpers.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
use anyhow::Result;
use ast::str::Quote;
use ast::visitor::source_order;
use ruff_python_ast::visitor::source_order::SourceOrderVisitor;
use std::cmp::Reverse;

use ruff_diagnostics::Edit;
Expand All @@ -9,7 +12,6 @@ use ruff_python_codegen::{Generator, Stylist};
use ruff_python_semantic::{
analyze, Binding, BindingKind, Modules, NodeId, ResolvedReference, ScopeKind, SemanticModel,
};
use ruff_source_file::Locator;
use ruff_text_size::Ranged;

use crate::rules::flake8_type_checking::settings::Settings;
Expand Down Expand Up @@ -221,9 +223,7 @@ pub(crate) fn is_singledispatch_implementation(
pub(crate) fn quote_annotation(
Glyphack marked this conversation as resolved.
Show resolved Hide resolved
node_id: NodeId,
semantic: &SemanticModel,
locator: &Locator,
stylist: &Stylist,
generator: Generator,
) -> Result<Edit> {
let expr = semantic.expression(node_id).expect("Expression not found");
if let Some(parent_id) = semantic.parent_expression_id(node_id) {
Expand All @@ -233,51 +233,42 @@ pub(crate) fn quote_annotation(
// If we're quoting the value of a subscript, we need to quote the entire
// expression. For example, when quoting `DataFrame` in `DataFrame[int]`, we
// should generate `"DataFrame[int]"`.
return quote_annotation(parent_id, semantic, locator, stylist, generator);
return quote_annotation(parent_id, semantic, stylist);
}
}
Some(Expr::Attribute(parent)) => {
if expr == parent.value.as_ref() {
// If we're quoting the value of an attribute, we need to quote the entire
// expression. For example, when quoting `DataFrame` in `pd.DataFrame`, we
// should generate `"pd.DataFrame"`.
return quote_annotation(parent_id, semantic, locator, stylist, generator);
return quote_annotation(parent_id, semantic, stylist);
}
}
Some(Expr::Call(parent)) => {
if expr == parent.func.as_ref() {
// If we're quoting the function of a call, we need to quote the entire
// expression. For example, when quoting `DataFrame` in `DataFrame()`, we
// should generate `"DataFrame()"`.
return quote_annotation(parent_id, semantic, locator, stylist, generator);
return quote_annotation(parent_id, semantic, stylist);
}
}
Some(Expr::BinOp(parent)) => {
if parent.op.is_bit_or() {
// If we're quoting the left or right side of a binary operation, we need to
// quote the entire expression. For example, when quoting `DataFrame` in
// `DataFrame | Series`, we should generate `"DataFrame | Series"`.
return quote_annotation(parent_id, semantic, locator, stylist, generator);
return quote_annotation(parent_id, semantic, stylist);
}
}
_ => {}
}
}

// If the annotation already contains a quote, avoid attempting to re-quote it. For example:
// ```python
// from typing import Literal
//
// Set[Literal["Foo"]]
// ```
let text = locator.slice(expr);
if text.contains('\'') || text.contains('"') {
return Err(anyhow::anyhow!("Annotation already contains a quote"));
}

// Quote the entire expression.
let quote = stylist.quote();
let annotation = generator.expr(expr);
let mut quote_annotation = QuoteAnnotation::new(stylist);
quote_annotation.visit_expr(expr);

let annotation = quote_annotation.annotation;

Ok(Edit::range_replacement(
format!("{quote}{annotation}{quote}"),
Expand All @@ -304,3 +295,105 @@ pub(crate) fn filter_contained(edits: Vec<Edit>) -> Vec<Edit> {
}
filtered
}

#[derive(Copy, PartialEq, Clone)]
enum State {
Literal,
AnnotatedFirst,
AnnotatedRest,
Other,
}

pub(crate) struct QuoteAnnotation<'a> {
state: Vec<State>,
stylist: &'a Stylist<'a>,
annotation: String,
final_quote_type: Quote,
}
Glyphack marked this conversation as resolved.
Show resolved Hide resolved

impl<'a> QuoteAnnotation<'a> {
pub(crate) fn new(stylist: &'a Stylist<'a>) -> Self {
let final_quote_type = stylist.quote();
Self {
state: vec![],
stylist,
annotation: String::new(),
final_quote_type,
}
}
}

impl<'a> source_order::SourceOrderVisitor<'a> for QuoteAnnotation<'a> {
fn visit_expr(&mut self, expr: &'a Expr) {
let generator = Generator::from(self.stylist);
match expr {
Expr::Subscript(ast::ExprSubscript { value, slice, .. }) => {
if let Some(name) = value.as_name_expr() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we only look for Name node here? What about attribute nodes like pd.DataFrame or typing.Optional[int] which is more relevant in this context as it's a subscript expression.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Is it enough if we rely on an attribute with value of "typing" and attr of "Optional" or "Literal", etc.?
Or should we fully resolved and check the qualified name in this case that the symbol is resolved to that class or not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added another case for attributes.

let value = generator.expr(value);
self.annotation.push_str(&value);
self.annotation.push('[');
match name.id.as_str() {
"Literal" => self.state.push(State::Literal),
"Annotated" => self.state.push(State::AnnotatedFirst),
_ => self.state.push(State::Other),
}

self.visit_expr(slice);
self.state.pop();
self.annotation.push(']');
}
}
Expr::Tuple(ast::ExprTuple { elts, .. }) => {
let Some(first_elm) = elts.first() else {
return;
};
self.visit_expr(first_elm);
if self.state.last().copied() == Some(State::AnnotatedFirst) {
self.state.push(State::AnnotatedRest);
}
for elm in elts.iter().skip(1) {
self.annotation.push_str(", ");
self.visit_expr(elm);
}
self.state.pop();
}
Expr::BinOp(ast::ExprBinOp {
left, op, right, ..
}) => {
Glyphack marked this conversation as resolved.
Show resolved Hide resolved
self.visit_expr(left);
self.annotation.push(' ');
self.annotation.push_str(op.as_str());
self.annotation.push(' ');
self.visit_expr(right);
}
Expr::BoolOp(ast::ExprBoolOp { op, values, .. }) => {
for (i, value) in values.iter().enumerate() {
if i > 0 {
self.annotation.push(' ');
self.annotation.push_str(op.as_str());
}
self.visit_expr(value);
}
}
_ => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a branch for Expr::BoolOp although I don't think that's required. Can you confirm this? @Glyphack

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct. I was not very familiar with syntax in type annotation but I don't see anything with a boolean operation.

let source = match self.state.last().copied() {
Some(State::Literal | State::AnnotatedRest) => {
let mut source = generator.expr(expr);
source = source.replace(
self.final_quote_type.as_char(),
&self.final_quote_type.opposite().as_char().to_string(),
);
source
}
None | Some(State::AnnotatedFirst | State::Other) => {
let mut source = generator.expr(expr);
source = source.replace(self.final_quote_type.as_char(), "");
source = source.replace(self.final_quote_type.opposite().as_char(), "");
source
}
};
self.annotation.push_str(&source);
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -270,9 +270,7 @@ fn quote_imports(checker: &Checker, node_id: NodeId, imports: &[ImportBinding])
Some(quote_annotation(
reference.expression_id()?,
checker.semantic(),
checker.locator(),
checker.stylist(),
checker.generator(),
))
} else {
None
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -505,9 +505,7 @@ fn fix_imports(checker: &Checker, node_id: NodeId, imports: &[ImportBinding]) ->
Some(quote_annotation(
reference.expression_id()?,
checker.semantic(),
checker.locator(),
checker.stylist(),
checker.generator(),
))
} else {
None
Expand Down
Original file line number Diff line number Diff line change
@@ -1,24 +1,22 @@
---
source: crates/ruff_linter/src/rules/flake8_type_checking/mod.rs
---
quote.py:64:28: TCH004 [*] Quote references to `pandas.DataFrame`. Import is in a type-checking block.
quote.py:67:28: TCH004 [*] Quote references to `pandas.DataFrame`. Import is in a type-checking block.
|
63 | if TYPE_CHECKING:
64 | from pandas import DataFrame
66 | if TYPE_CHECKING:
67 | from pandas import DataFrame
| ^^^^^^^^^ TCH004
65 |
66 | def func(value: DataFrame):
68 |
69 | def func(value: DataFrame):
|
= help: Quote references

ℹ Unsafe fix
63 63 | if TYPE_CHECKING:
64 64 | from pandas import DataFrame
65 65 |
66 |- def func(value: DataFrame):
66 |+ def func(value: "DataFrame"):
67 67 | ...
66 66 | if TYPE_CHECKING:
67 67 | from pandas import DataFrame
68 68 |
69 69 |


69 |- def func(value: DataFrame):
69 |+ def func(value: "DataFrame"):
70 70 | ...
71 71 |
72 72 |
Loading
Loading