Skip to content

Commit

Permalink
Start tracking quoting style in the AST (astral-sh#10298)
Browse files Browse the repository at this point in the history
This PR modifies our AST so that nodes for string literals, bytes literals and f-strings all retain the following information:
- The quoting style used (double or single quotes)
- Whether the string is triple-quoted or not
- Whether the string is raw or not

This PR is a followup to astral-sh#10256. Like with that PR, this PR does not, in itself, fix any bugs. However, it means that we will have the necessary information to preserve quoting style and rawness of strings in the `ExprGenerator` in a followup PR, which will allow us to provide a fix for astral-sh#7799.

The information is recorded on the AST nodes using a bitflag field on each node, similarly to how we recorded the information on `Tok::String`, `Tok::FStringStart` and `Tok::FStringMiddle` tokens in astral-sh#10298. Rather than reusing the bitflag I used for the tokens, however, I decided to create a custom bitflag for each AST node.

Using different bitflags for each node allows us to make invalid states unrepresentable: it is valid to set a `u` prefix on a string literal, but not on a bytes literal or an f-string. It also allows us to have better debug representations for each AST node modified in this PR.
  • Loading branch information
AlexWaygood authored and nkxxll committed Mar 10, 2024
1 parent ad9c073 commit 4e74374
Show file tree
Hide file tree
Showing 81 changed files with 4,150 additions and 3,173 deletions.
8 changes: 4 additions & 4 deletions crates/ruff_linter/src/rules/flake8_quotes/settings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@ impl Default for Quote {
}
}

impl From<ruff_python_parser::QuoteStyle> for Quote {
fn from(value: ruff_python_parser::QuoteStyle) -> Self {
impl From<ruff_python_ast::str::QuoteStyle> for Quote {
fn from(value: ruff_python_ast::str::QuoteStyle) -> Self {
match value {
ruff_python_parser::QuoteStyle::Double => Self::Double,
ruff_python_parser::QuoteStyle::Single => Self::Single,
ruff_python_ast::str::QuoteStyle::Double => Self::Double,
ruff_python_ast::str::QuoteStyle::Single => Self::Single,
}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use ast::{StringLiteralFlags, StringLiteralPrefix};
use ruff_python_ast::{self as ast, Arguments, Expr};
use ruff_text_size::Ranged;

Expand Down Expand Up @@ -218,7 +219,13 @@ fn check_os_environ_subscript(checker: &mut Checker, expr: &Expr) {
);
let node = ast::StringLiteral {
value: capital_env_var.into_boxed_str(),
unicode: env_var.is_unicode(),
flags: StringLiteralFlags::default().with_prefix({
if env_var.is_unicode() {
StringLiteralPrefix::UString
} else {
StringLiteralPrefix::None
}
}),
..ast::StringLiteral::default()
};
let new_env_var = node.into();
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use ast::FStringFlags;
use itertools::Itertools;

use crate::fix::edits::pad;
Expand Down Expand Up @@ -97,6 +98,7 @@ fn build_fstring(joiner: &str, joinees: &[Expr]) -> Option<Expr> {
let node = ast::FString {
elements: f_string_elements,
range: TextRange::default(),
flags: FStringFlags::default(),
};
Some(node.into())
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
use anyhow::Result;

use ast::StringLiteralFlags;
use ruff_diagnostics::{AlwaysFixableViolation, Diagnostic, Fix};
use ruff_macros::{derive_message_formats, violation};
use ruff_python_ast as ast;
Expand Down Expand Up @@ -106,7 +107,7 @@ fn generate_keyword_fix(checker: &Checker, call: &ast::ExprCall) -> Fix {
.expr(&Expr::StringLiteral(ast::ExprStringLiteral {
value: ast::StringLiteralValue::single(ast::StringLiteral {
value: "locale".to_string().into_boxed_str(),
unicode: false,
flags: StringLiteralFlags::default(),
range: TextRange::default(),
}),
range: TextRange::default(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ impl AlwaysFixableViolation for UnicodeKindPrefix {

/// UP025
pub(crate) fn unicode_kind_prefix(checker: &mut Checker, string: &StringLiteral) {
if string.unicode {
if string.flags.is_u_string() {
let mut diagnostic = Diagnostic::new(UnicodeKindPrefix, string.range);
diagnostic.set_fix(Fix::safe_edit(Edit::range_deletion(TextRange::at(
string.start(),
Expand Down
6 changes: 5 additions & 1 deletion crates/ruff_python_ast/src/node.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4430,7 +4430,11 @@ impl AstNode for ast::FString {
where
V: PreorderVisitor<'a> + ?Sized,
{
let ast::FString { elements, range: _ } = self;
let ast::FString {
elements,
range: _,
flags: _,
} = self;

for fstring_element in elements {
visitor.visit_f_string_element(fstring_element);
Expand Down
Loading

0 comments on commit 4e74374

Please sign in to comment.