Add ability to handle magic commands in Jupyter notebook

[dhruvmanila]

Currently, if any cell contains any magic commands it'll be ignored even if the cell contains valid Python code. This might lead to false positive as some code is intentionally ignored for linting (undefined symbol).

At a high level overview what black does is the following:

1. Using IPython's transformer, convert the magic commands into the respective function calls (% ... will be converted to get_ipython().run_cell_magic)
2. Replace cell magics with tokens (string values which is a valid Python code) and keep track of such replacements
3. Replace line magics in the same way as mentioned in (2)
4. Returned the transformed code along with the replacements
5. At the end, use the replacements to get back the original content

Reference implementation in black: https://github.com/psf/black/blob/main/src/black/handle_ipynb_magics.py

[dhruvmanila]

Jupytext has a simpler implementation although lots of regex: https://github.com/mwouts/jupytext/blob/main/jupytext/magics.py

[henryiii]

There might be an easier but dumber way to do with without invoking IPython, which I would assume would be slow. You could add a configurable list of "ignored" magics. The default could contain the built-in magics that don't have much affect: %%time, %%timeit, etc. But a user could override the list via configuration. Detecting them I'd assume is already mostly done, since it can ignore them?

This wouldn't cover magics that modify stuff or change the language, but would cover a lot of common cases & is the situation that is easiest to report violations and autofix.

[dhruvmanila]

Hey, thanks for the suggestions. Yes, invoking IPython would be slow and we're not going that route. Instead, we'll use a new parser mode in which the parser will parse the magic commands at possible locations. New tokens and AST nodes are introduced which will be available only in that mode. Although the node isn't finalized, you can refer to astral-sh/RustPython-Parser#31.

[dhruvmanila]

The solution which we've implemented is to integrate it in the parser directly. The following changes were made to the parser to accommodate the requirement:

• A new Jupyter mode was added to allow lexing/parsing the magic commands only for Jupyter Notebooks and give an error otherwise.
• The lexer will recognize the magic commands and emit the MagicCommand token wherever it's allowed¹ i.e., as standalone statement (%matplotlib inline) or in an assignment statement (dir = !pwd).
  • Lex Jupyter line magic with Mode::Jupyter RustPython-Parser#23
  • Lex Jupyter Magic in assignment value position RustPython-Parser#30
• Two new nodes StmtLineMagic and ExprLineMagic are added for the parser.
  • Add line magic stmt and expr AST nodes RustPython-Parser#31

References

• All possible magic command related documentation is present on this page: https://ipython.readthedocs.io/en/stable/interactive/reference.html#interactive-use
• Original implementation in IPython codebase: https://github.com/ipython/ipython/blob/main/IPython/core/inputtransformer2.py
• sonar-python implementation: https://github.com/SonarSource/sonar-python/blob/master/python-frontend/src/main/java/org/sonar/python/api/IPythonGrammarBuilder.java

Footnotes

1. The allowed magic token positions are determined directly from the IPython implementation of the same. ↩