formatter(Jupyter): ending semicolons valid in Jupyter

[henryiii]

The Jupyter formatter has a couple of issues when running on notebooks.

One is it removes semicolons at the end of lines. black[jupyter] does not remove semicolons at the end of lines when running on a notebook, since it's used to mark output as hidden, for example when plotting. plt.plot(); will show the plot but not print out the repr for the Artists that the plot method returns.

Edit: On the last line of a cell. Related: #7300 which is for the linter. But that's rule based, so it's easy to turn it off for *.ipynb files, while there isn't a workaround for the formatter, making it a blocker.

(Fixed in main) The other is that it can't read a notebook that black and ruff-lint are able to read. The error is:

error: Failed to format docs/examples/HistDemo.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }

I'm guessing it's choking on a line magic, perhaps? The file does have this line:

%config InteractiveShell.ast_node_interactivity="last_expr_or_assign"

Which is the only % in the file.

[charliermarsh]

Thanks!

[dhruvmanila]

Re line magic, it's solved #8204, sorry for the churn!

[henryiii]

Great! Probably should have checked main. So just simicolons left.

[dhruvmanila]

Related: #7300

[MichaReiser]

Is my assumption correct that this only applies to expression statements or are there other statements for which semicolons need to be preserved?

[henryiii]

IPython has several modes. If you set:

%config InteractiveShell.ast_node_interactivity="last_expr_or_assign"

Then either val = expr or just expr will print if it's the final statement in the cell, unless there's a semicolon (see, for example, https://hist.readthedocs.io/en/latest/examples/HistDemo.html). By default, only expr will display, and val = expr won't. I'd say simply allowing a final semicolon in cells on expressions or assignment expressions would be fine.

IIRC a.b = and a[b] = statements don't display even in the enhanced mode, but could be wrong.

[MichaReiser]

Note, there's now a helper function to extract the trailing semicolon. We can use it in the assignment (including augmented and annotated) and expression statement formatting

ruff/crates/ruff_python_formatter/src/statement/mod.rs

Lines 89 to 107 in 2c84f91

	pub(super) fn trailing_semicolon(node: AnyNodeRef, source: &str) -> Option<TextRange> {
	debug_assert!(node.is_statement());
	let tokenizer = SimpleTokenizer::starts_at(node.end(), source);
	let next_token = tokenizer
	.take_while(|token| !token.kind().is_comment())
	.find(|token| !token.kind().is_trivia());
	if let Some(SimpleToken {
	kind: SimpleTokenKind::Semi,
	range,
	}) = next_token
	{
	Some(range)
	} else {
	None
	}
	}

[dhruvmanila]

Black's implementation for reference: https://github.com/psf/black/blob/c54c213d6a3132986feede0cf0525f5bae5b43d6/src/black/handle_ipynb_magics.py#L73-L102. It seems to be removing them for all cases.

[henryiii]

The function after that puts them back - not sure why you'd need to remove it and then put it back, since the normal formatter also removes them - just checking to see if it's there and then putting it back seems like it would also work. Anyway, what cases are not covered by assignments and expressions? Blocks can't end lines, for example. Simicolons aren't needed for import statements - I wonder if black keeps them, actually. :)

[MichaReiser]

Black's implementation for reference: psf/black@c54c213/src/black/handle_ipynb_magics.py#L73-L102. It seems to be removing them for all cases.

Nice find. Removing them and adding them back sounds complicated. We can just preserve them 😄