Allow slice syntax in e.g. Annotated[int, 1:3] and TensorType["batch":..., float]

[Zac-HD]

Consider the following trivial Python file, using the torchtyping library - if only this typechecked, we could write some fancy ML code!

from typing import Annotated, Dict
from torchtyping import TensorType  # type: ignore

a: Annotated[int, 1:2]
b: Dict[int, x:y]
c: Dict[x:y]
t: TensorType["batch":..., float]

On master, we get syntax-level errors due to the use of : slice syntax, which prevents any semantic checking - and can't even be disabled by a config file!

$ mypy --show-column-numbers t.py
t.py:4:4: error: Slice usage in type annotation is invalid
t.py:5:4: error: Slice usage in type annotation is invalid
t.py:6:4: error: Slice usage in type annotation is invalid
t.py:7:4: error: Slice usage in type annotation is invalid
Found 4 errors in 1 file (errors prevented further checking)

Following this patch, mypy identifies the problems with our Dict annotations, but is happy with Annotated and can be directed to ignore TensorType:

$ mypy --show-column-numbers t.py
t.py:5:14: error: Invalid type comment or annotation
t.py:5:14: note: did you mean to use ',' instead of ':' ?
t.py:6:4: error: "dict" expects 2 type arguments, but 1 given
t.py:6:9: error: Invalid type comment or annotation
t.py:6:9: note: did you mean to use ',' instead of ':' ?
Found 3 errors in 1 file (checked 1 source file)

This PR therefore fixes #10266; and closes #11279. Thanks to @sobolevn for pointing me in the right direction!

[sobolevn]

except that there's some weird interaction on Python < 3.9, which I don't understand well enough to track down quickly. Any suggestions?

Python3.9 has different ast.Subscript structure. This helper might be useful to understand how it changed: https://github.com/wemake-services/wemake-python-styleguide/blob/master/wemake_python_styleguide/compat/functions.py#L32-L45

[sobolevn]

I get a different ordering of messages for same-line errors on different Python versions

Try to run your tests with --show-column-numbers, I suspect that it might be the issue. Since all three items are on the same line:

  main:3: error: "dict" expects 2 type arguments, but 1 given  [type-arg]
  main:3: error: Invalid type comment or annotation  [valid-type]
  main:3: note: did you mean to use ',' instead of ':' ?

Column numbers might be different here: https://github.com/python/mypy/pull/11345/files#diff-a7d2c40f4be6322cee498432ca9dbbc3a356c3cbcb61029231452f6638bb11b5R1565-R1572

[Zac-HD]

Alright, I've fixed the pre-Python-3.9 issues: with message ordering (by propagating column numbers), and with Index nodes so that the int in Annotated[int, 3:4] is still recognised 🎉

However... this causes a problem, where on Python 3.9 mypyc complains that mypy/fastparse.py:1558: error: "Index" has no attribute "value". On one level this is fair enough - Index nodes are deprecated - but I can't even # type: ignore the problem because that's an unused ignore on older versions! Any further ideas @sobolevn?

[sobolevn]

@Zac-HD will conditional type alias work in this case? 🤔

if sys.version_info >= (3, 9):
    AstIndex = Any
else:
   AstIndex = ast.Index

[Zac-HD]

Or just imitate ASTConvertor.visit_index with return self.visit(cast(Any, n).value)? Let's see...

[Zac-HD]

Woohoo, it's all working! Thanks for your help @sobolevn 🥰

Ping @JelleZijlstra for review?