Implement blank_line_after_nested_stub_class preview style

[dhruvmanila]

Summary

This PR implements the blank_line_after_nested_stub_class preview style in the formatter.

The logic is divided into 3 parts:

1. In between preceding and following nodes at top level and nested suite
2. When there's a trailing comment after the class
3. When there is no following node from (1) which is the case when it's the last or the only node in a suite

We handle (3) with FormatLeadingAlternateBranchComments.

Test Plan

• Add new test cases and update existing snapshots
• Checked the typeshed diff

fixes: #8891

[github-actions]

ruff-ecosystem results

Formatter (stable)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

warning: Detected debug build without --no-cache.
error: Failed to parse examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb:3:7:8: Unexpected token 'prompt'

Formatter (preview)

ℹ️ ecosystem check detected format changes. (+5 -0 lines in 5 files in 2 projects; 1 project error; 40 projects unchanged)

python/typeshed (+4 -0 lines across 4 files)

ruff format --preview

stdlib/inspect.pyi~L430

         varargs: str | None
         keywords: str | None
         defaults: tuple[Any, ...]
+
     def getargspec(func: object) -> ArgSpec: ...
 
 class FullArgSpec(NamedTuple):

stdlib/sqlite3/dbapi2.pyi~L450

 
 @final
 class _Statement: ...
+
 class Warning(Exception): ...
 
 if sys.version_info >= (3, 11):

stubs/gevent/gevent/events.pyi~L97

 
 @implementer(IGeventWillPatchEvent)
 class GeventWillPatchEvent(GeventPatchEvent): ...
+
 class IGeventDidPatchEvent(IGeventPatchEvent): ...
 
 @implementer(IGeventWillPatchEvent)

stubs/pyserial/serial/tools/list_ports_windows.pyi~L34

         ClassGuid: ctypes._CField[Incomplete, Incomplete, Incomplete]
         DevInst: ctypes._CField[Incomplete, Incomplete, Incomplete]
         Reserved: ctypes._CField[Incomplete, Incomplete, Incomplete]
+
     PSP_DEVINFO_DATA: type[ctypes._Pointer[SP_DEVINFO_DATA]]
     PSP_DEVICE_INTERFACE_DETAIL_DATA = ctypes.c_void_p
     setupapi: ctypes.WinDLL

reflex-dev/reflex (+1 -0 lines across 1 file)

ruff format --preview

reflex/config.pyi~L47

 class Config(Base):
     class Config:
         validate_assignment: bool
+
     app_name: str
     loglevel: constants.LogLevel
     frontend_port: int

openai/openai-cookbook (error)

ruff format --preview

warning: Detected debug build without --no-cache.
error: Failed to parse examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb:3:7:8: Unexpected token 'prompt'

[MichaReiser]

@dhruvmanila what's the status of this PR? Also note that there have been some recent changes in Black. It might be worth double checking Black's tests again to ensure we match the formatting precisely.

[dhruvmanila]

Here's an update regarding this preview style implementation in Ruff:

tldr; there are bunch of edge cases where it seems to be difficult to add support ...

What's working today:

1

Cases when the class definition is followed by another statement at the same indent level. This is to say that in a suite, the preceding and following value is set. Why is this easier? Because the loop takes care of a lot of conditions.

For example, in the following code snippet there are 2 statements in a suite which is the body of the Top class. Here, the preceding would be Nested1 class and following would be the assignment statement. Because we have the context of the siblings, it's easier to find out if the class needs an empty line or not.

class Top:
	class Nested1:  # <- statement 1
		pass
	foo = 1  # <- statement 2

2

Some other cases which works today is where the class is followed by a trailing comment. This is then handled by the class formatting as then it doesn't matter if there's any other statement following the class or not as we'll always need to add an empty line between the class and the comment. For example,

class Top:
	class Nested1:
		pass
	# comment

This will be handled in empty_lines_before_trailing_comments builder.

Problematic cases

Note that some of the following cases have a workaround in the PR but I'm not fully convinced with the implementation which is why they've been put in this section.

The main type of case where there are problems are the ones where the class definition is the last statement in the suite. For example,

1

class Top:
	class Nested:
		pass
foo = 1

Here, the loop will never be executed because there's just 1 statement in the suite. This means that we need to have a special case implementation for the last statement in the suite. This needs to account for a lot of other conditions to make the decision of whether to add an empty line or not:

Is there a class in the parent suite which already added an empty line? If so, then we don't need to add another empty line.

class Top:
    class Nested11:
        class Nested12:
            pass
    class Nested21:
        pass

Here, the Nested11 class will add an empty line. Now, the Nested12 class should not be adding an empty line otherwise there'll be 2 empty lines. This is solved in this PR by thinking it in a different way. Instead of the Nested11 class being responsible for adding an empty line, we'll make Nested12 class responsible for it. This is to say that the innermost class should be responsible to add an empty line. Why? Because as a parent it's easier to traverse through the children to check this condition:

/// Checks if the last child of the given node is a class definition without a
/// trailing comment.
pub(crate) fn is_last_child_a_class_def(node: AnyNodeRef<'_>, f: &PyFormatter) -> bool {
    let comments = f.context().comments();
    std::iter::successors(node.last_child_in_body(), AnyNodeRef::last_child_in_body)
        .take_while(|last_child| !comments.has_trailing_own_line(*last_child))
        .any(|last_child| last_child.is_stmt_class_def())
}

But, this then creates another problem. Take the following code as an example:

if a:
	class Nested:
		pass
else:
	pass

Here, the problem is that the new logic will add an empty line after the Nested class which will give us:

if a:
	class Nested:
		pass

else:
	pass

Now, the same logic will try to preserve the empty line by adding the IR element in the same position. But while formatting the else block, the FormatLeadingAlternateBranchComments formatter sees that there's an empty line before the else block which it tries to maintain by adding an empty line IR element. Thus, two empty line elements are added which creates an unstable formatting.

2

When a comment is not trailing to the class definition but actually to one of the statements in the parent suite:

if top1:
    class Nested:
        pass
# comment
if top2:
    pass

3

Similar to the previous case but at end of file:

if top1:
    class Nested:
        pass
# comment

Number of empty lines

We also need to make sure that running the formatter multiple times doesn't keep on adding empty lines (at max it could go to 2). This would happen when the new logic would add an empty line on the first run and then on the second run another logic sees the empty line which it then tries to maintain. This adds multiple empty line IR elements.

Possible solutions

Instead of looking ahead, we'll always look behind which is to say that for the current node we need to query the absolute previous node. But we'll have to add the logic in multiple places - suite, if, match, try, etc. Here, absolute previous in the literal sense, not just in the same indent level. For example,

class Top:
    class Nested11:
        class Nested12:
            pass
    class Nested21:
        pass

Here, the previous statement for Nested21 would be Nested12 and not Nested11.

[dhruvmanila]

This helps when the class is at the boundary location in nested statement:

if foo:
    class Nested:
        pass
else:
    pass

The FormatLeadingAlternateBranchComments will be called before the else block formatting. Similarly for other blocks like match, case, try .. except .., etc.

[dhruvmanila]

It's difficult to merge the is_*_enabled && *_condition logic with empty_line_condition variable up above like so:

    let empty_line_condition = preceding_comments.has_trailing()
        || following_comments.has_leading()
        || !stub_suite_can_omit_empty_line(preceding, following, f)
        || (is_blank_line_after_nested_stub_class_enabled(f.context())
                    && blank_line_after_nested_stub_class_condition(
                        preceding.into(),
                        Some(following.into()),
                        f,
                    ))

The reason being that in the top level condition the boolean check is direct but in other suite kind there's an additional check of the number of empty lines. That additional check is to only preserve the empty lines if they're present while the preview style adds an empty line conditionally. So, the condition should OR the complete check which was existing earlier to complement it.

[MichaReiser]

I think what would help here is to rename empty_line_condition into preserve_empty_line which is different from your check which forces empty lines.

I recommend moving the blank_line_after_nested_stub_class_condition out of the match and next to empty_line_condition, and name the variable require_empty_line

I would rename blank_line_after_nested_stub_class_condition to requires_empty_line_between_stub_file_statements (not just preserving, enforcing an empty line).

We can then rewrite the checks in the match branches to

if requires_empty_line || preserve_empty_line {
    empty_line.fmt(f)
}

which I find easier to understand.

Edit: Maybe better. Could we move the check into stub_suite_can_omit_empty_line?

[dhruvmanila]

I think what would help here is to rename empty_line_condition into preserve_empty_line which is different from your check which forces empty lines.

I'm not sure if this is 100% correct. For top-level, if the condition is true, we always make sure that there is just one empty line. It could be that there isn't any empty line in the source code which means we're adding an additional empty line. But, for the nested cases, we'd only preserve any existing empty lines and limit it to 1.

I recommend moving the blank_line_after_nested_stub_class_condition out of the match and next to empty_line_condition, and name the variable require_empty_line

Thanks, I'll update the code accordingly.

I would rename blank_line_after_nested_stub_class_condition to requires_empty_line_between_stub_file_statements (not just preserving, enforcing an empty line).

Yeah, I'm going with is_blank_line_required_after_nested_stub_class.

[dhruvmanila]

Yeah, I'm going with is_blank_line_required_after_nested_stub_class.

Actually, with your other comment, I'm thinking of should_insert_blank_line_after_class_in_stub_file.

[dhruvmanila]

This is the core condition which checks whether we need to add an empty line between the preceding and following node or not.

[dhruvmanila]

Class with a trailing comment is handled by empty_lines_before_trailing_comments builder.

[dhruvmanila]

If the preceding node isn't a class, then we check if there's a class node as the last child node of the body of nested nodes. This nesting could be arbitrarily deep.

[MichaReiser]

The code changes overall look good to me. I think we can make some improvements to improve readability and extend documentation a bit.

I'm very surprised that there are no changed black snapshots. Do you know why that is? Are we sure we match Black's formatting?

It would help reviewers if you add an example of the expected formatting (how is it different from today) with a short explanation of the preview style to the PR. I know it's in the linked PR, but it requires multiple steps by the reviewer: Open issue, read the summary, go back to the PR, read the summary, and finally read the code.

[MichaReiser]

I think what would help here is to rename empty_line_condition into preserve_empty_line which is different from your check which forces empty lines.

I recommend moving the blank_line_after_nested_stub_class_condition out of the match and next to empty_line_condition, and name the variable require_empty_line

I would rename blank_line_after_nested_stub_class_condition to requires_empty_line_between_stub_file_statements (not just preserving, enforcing an empty line).

We can then rewrite the checks in the match branches to

if requires_empty_line || preserve_empty_line {
    empty_line.fmt(f)
}

which I find easier to understand.

Edit: Maybe better. Could we move the check into stub_suite_can_omit_empty_line?

[MichaReiser]

#9674 should ensure that black stub files with options run as stub tests.

[MichaReiser]

Can you explain why this special handling is required here but not for other cases for which stub_suite_can_omit_empty_line returns false?

[dhruvmanila]

This is to handle cases when a class definition is the last statement in a suite. The empty line formatting in suite is a loop which handles only for in between statements i.e., both preceding and following are present. This is required only before an alternate branch because otherwise it'll be handled by the top-level suite formatting.

[MichaReiser]

Awesome. Thanks for the added documentation