Support ParamSpec variables in type aliases #14159

ilevkivskyi · 2022-11-21T14:50:26Z

Fixes #11855
Fixes #7084
Fixes #10445
Should fix #4987

After thinking about this for some time, it looks like the best way to implement this is by switching type aliases from unbound to bound type variables. Then I can essentially simply share (or copy in one small place, to avoid cyclic imports) all the logic that currently exists for ParamSpec and Concatenate in expand_type() etc.

This will also address a big piece of tech debt, and will get some benefits (almost) for free, such as checking bounds/values for alias type variables, and much tighter handling of unbound type variables.

Note that in this PR I change logic for emitting some errors, I try to avoid showing multiple errors for the same location/reason. But this is not an essential part of this PR (it is just some test cases would otherwise fail with even more error messages), I can reconsider if there are objections.

ilevkivskyi · 2022-11-21T15:07:42Z

Oh it looks like there is a merge race with one of my other PRs, should be easy to rebase.

ilevkivskyi · 2022-11-21T16:25:13Z

On the mypy_primer:

Some remaining errors in discord.py look like mypy is struggling with type inference for unions, doesn't look like a bug in this PR (other errors are correct).
psycopg is doing Foo: "TypeAlias" = Bar[T] in two places, where TypeAlias is never imported, so technically the errors are correct
There are two things in Expression. One is exposed existing bug that "skip-level" type variables are not bound correctly in functions nested in generic functions. Second one is less clear, but unlikely to caused by this PR, probably also existing issue.

I will try to see if there are easy fixes to the things in Expression to minimize the fallout of this PR.

ilevkivskyi · 2022-11-23T13:47:24Z

@jhance @JukkaL @hauntsaninja Can one of you please take a look at this? It would be great to have a second pair of eyes, maybe I missed something (this PR includes a significant refactoring in how type aliases internals).

JukkaL · 2022-11-23T15:29:07Z

I did some manual testing, and this seems to generate a false positive when defining a type alias:

from typing import ParamSpec, Callable

P = ParamSpec("P")

# Error: The first argument to Callable must be a list of types, parameter specification, or "..."
A = list[Callable[P, None]]

f: A[[int, str]]
reveal_type(f)  # builtins.list[def (builtins.int, builtins.str)] (seems fine)

gandhis1 · 2022-11-23T16:10:38Z

As @erictraut mentioned in this comment, the tests that he put together for pyright may be useful for PRs like this one to test edge cases of ParamSpec and Concatenate. Probably only a handful of repos in mypy primer scope have even begun to use this feature. Maybe it makes sense to bring them over, see which ones fail, and then selectively disable test cases that are known to be unsupported?

JukkaL · 2022-11-23T16:28:48Z

We can now leak type lists into inferred types, which could cause problems (e.g. crashes) since type lists are not generally expected as types, I think:

from typing import ParamSpec, Callable, Concatenate, TypeVar

T = TypeVar("T")
P = ParamSpec("P")

A = Callable[Concatenate[T, P], None]

# Type argument "[int, str]" of "A" must be a subtype of "object"
f: A[[int, str], [str, float]]
reveal_type(f)  # def ([builtins.int, builtins.str], builtins.str, builtins.float)

ilevkivskyi · 2022-11-23T16:33:37Z

@JukkaL Thanks! I fixed the first one (and also a similar one with Concatenate), will push soon. For the second one there should be an easy fix, we just need to replace invalid args with Any.

ilevkivskyi · 2022-11-23T16:54:58Z

Btw this second issue affects regular instances as well (I am fixing both anyway)

JukkaL · 2022-11-23T17:13:52Z

Maybe it makes sense to bring them over, see which ones fail, and then selectively disable test cases that are known to be unsupported?

This could be helpful, since we've had a bunch of ParamSpec edge cases that we've been fixing gradually. We can do it in a separate PR, except maybe for test cases that involve type aliases.

@ilevkivskyi Do you want to have a look if there are any relevant test cases you could include in this PR?

ilevkivskyi · 2022-11-23T17:42:29Z

TBH I don't have desire to go through them, I would rather spend time working on default arguments in generics. I made this PR because I know type aliases, not because I know ParamSpec :-)

FWIW I think the best way may be if a volunteer would run that cases (and not just ParamSpec ones), and opens issues for failing ones (where it makes sense, e.g. not a duplicate).

ilevkivskyi · 2022-11-23T18:48:33Z

@JukkaL

We can now leak type lists into inferred types

Btw those are not type lists they are Parameters with a nice repr. Actually it looks like trying to replace them in semanal_typeargs conflicts with TypeVarTuples. So since it is not a new bug, I will leave it as is.

Btw @jhance it looks like semanal_typeargs is already broken for TypeVarTuples, the loop in visit_instance will compare type vars with wrong arguments and vice versa.

github-actions · 2022-11-23T19:08:01Z

Diff from mypy_primer, showing the effect of this PR on open source code:

xarray-dataclasses (https://github.com/astropenguin/xarray-dataclasses)
- xarray_dataclasses/datamodel.py:35: error: ParamSpec "PInit" is unbound  [valid-type]
- xarray_dataclasses/datamodel.py:173: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
- xarray_dataclasses/datamodel.py:173: error: Invalid location for ParamSpec "PInit"  [valid-type]
- xarray_dataclasses/datamodel.py:173: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[PInit, int]'
- xarray_dataclasses/datamodel.py:189: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
- xarray_dataclasses/datamodel.py:189: error: Invalid location for ParamSpec "PInit"  [valid-type]
- xarray_dataclasses/datamodel.py:189: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[PInit, int]'
- xarray_dataclasses/dataarray.py:29: error: ParamSpec "PInit" is unbound  [valid-type]
- xarray_dataclasses/dataset.py:29: error: ParamSpec "PInit" is unbound  [valid-type]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/_compat.py:17: error: Type variable "psycopg._compat.T" is unbound  [valid-type]
+ psycopg/psycopg/_compat.py:17: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
+ psycopg/psycopg/_compat.py:17: note: (Hint: Use "T" in function signature to bind "T" inside a function)
+ psycopg_pool/psycopg_pool/_compat.py:14: error: Type variable "psycopg_pool._compat.T" is unbound  [valid-type]
+ psycopg_pool/psycopg_pool/_compat.py:14: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
+ psycopg_pool/psycopg_pool/_compat.py:14: note: (Hint: Use "T" in function signature to bind "T" inside a function)

discord.py (https://github.com/Rapptz/discord.py)
- discord/ext/commands/_types.py:40: error: The first argument to Callable must be a list of types, parameter specification, or "..."  [valid-type]
- discord/ext/commands/_types.py:40: note: See https://mypy.readthedocs.io/en/stable/kinds_of_types.html#callable-types-and-lambdas
- discord/utils.py:152: error: The first argument to Callable must be a list of types, parameter specification, or "..."  [valid-type]
- discord/utils.py:152: note: See https://mypy.readthedocs.io/en/stable/kinds_of_types.html#callable-types-and-lambdas
- discord/utils.py:659: error: Bad number of arguments for type alias, expected: 0, given: 2  [type-arg]
- discord/utils.py:659: error: Invalid location for ParamSpec "P"  [valid-type]
- discord/utils.py:659: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
- discord/app_commands/tree.py:843: error: Bad number of arguments for type alias, expected: 0, given: 3  [type-arg]
- discord/app_commands/tree.py:843: error: Invalid location for ParamSpec "P"  [valid-type]
- discord/app_commands/tree.py:843: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
- discord/app_commands/tree.py:877: error: Bad number of arguments for type alias, expected: 0, given: 3  [type-arg]
- discord/app_commands/tree.py:877: error: Invalid location for ParamSpec "P"  [valid-type]
- discord/app_commands/tree.py:877: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
- discord/app_commands/tree.py:889: error: Need type annotation for "command"  [var-annotated]
- discord/app_commands/commands.py:113: error: The first argument to Callable must be a list of types, parameter specification, or "..."  [valid-type]
- discord/app_commands/commands.py:113: note: See https://mypy.readthedocs.io/en/stable/kinds_of_types.html#callable-types-and-lambdas
- discord/app_commands/commands.py:113: error: The last parameter to Concatenate needs to be a ParamSpec  [valid-type]
- discord/app_commands/commands.py:113: error: ParamSpec "P" is unbound  [valid-type]
- discord/app_commands/commands.py:114: error: The first argument to Callable must be a list of types, parameter specification, or "..."  [valid-type]
- discord/app_commands/commands.py:114: note: See https://mypy.readthedocs.io/en/stable/kinds_of_types.html#callable-types-and-lambdas
- discord/app_commands/commands.py:114: error: The last parameter to Concatenate needs to be a ParamSpec  [valid-type]
- discord/app_commands/commands.py:114: error: ParamSpec "P" is unbound  [valid-type]
- discord/app_commands/commands.py:139: error: Unexpected "..."  [misc]
- discord/app_commands/commands.py:139: error: Bad number of arguments for type alias, expected: 0, given: 3  [type-arg]
+ discord/app_commands/commands.py:680: error: Item "function" of "Union[Callable[[GroupT, Interaction, **P], Coroutine[Any, Any, T]], Callable[[Interaction, **P], Coroutine[Any, Any, T]]]" has no attribute "__self__"  [union-attr]
+ discord/app_commands/commands.py:681: error: Item "function" of "Union[Callable[[GroupT, Interaction, **P], Coroutine[Any, Any, T]], Callable[[Interaction, **P], Coroutine[Any, Any, T]]]" has no attribute "__func__"  [union-attr]
- discord/app_commands/commands.py:656: error: Bad number of arguments for type alias, expected: 0, given: 3  [type-arg]
- discord/app_commands/commands.py:656: error: Invalid location for ParamSpec "P"  [valid-type]
- discord/app_commands/commands.py:656: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
- discord/app_commands/commands.py:672: error: Bad number of arguments for type alias, expected: 0, given: 3  [type-arg]
- discord/app_commands/commands.py:672: error: Invalid location for ParamSpec "P"  [valid-type]
- discord/app_commands/commands.py:672: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
- discord/app_commands/commands.py:716: error: Bad number of arguments for type alias, expected: 0, given: 3  [type-arg]
- discord/app_commands/commands.py:716: error: Invalid location for ParamSpec "P"  [valid-type]
- discord/app_commands/commands.py:716: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
- discord/app_commands/commands.py:761: error: No overload variant of "TranslationContext" matches argument types "TranslationContextLocation", "Command[GroupT, P, T]"  [call-overload]
- discord/app_commands/commands.py:761: note: Possible overload variants:
- discord/app_commands/commands.py:761: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.command_name], data: Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu]) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:761: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.command_description], data: Command[Any, [VarArg(Any), KwArg(Any)], Any]) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:761: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.group_name, TranslationContextLocation.group_description], data: Group) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:761: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.parameter_name, TranslationContextLocation.parameter_description], data: Parameter) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:761: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.choice_name], data: Choice[Any]) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:761: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.other], data: Any) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:762: error: No overload variant of "TranslationContext" matches argument types "TranslationContextLocation", "Command[GroupT, P, T]"  [call-overload]
- discord/app_commands/commands.py:762: note: Possible overload variants:
- discord/app_commands/commands.py:762: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.command_name], data: Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu]) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:762: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.command_description], data: Command[Any, [VarArg(Any), KwArg(Any)], Any]) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:762: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.group_name, TranslationContextLocation.group_description], data: Group) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:762: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.parameter_name, TranslationContextLocation.parameter_description], data: Parameter) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:762: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.choice_name], data: Choice[Any]) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:762: note:     def [_L <: TranslationContextLocation, _D] __init__(self, location: Literal[TranslationContextLocation.other], data: Any) -> TranslationContext[_L, _D]
- discord/app_commands/commands.py:778: error: Argument 2 to "Parameter" has incompatible type "Command[GroupT, P, T]"; expected "Command[Any, [VarArg(Any), KwArg(Any)], Any]"  [arg-type]
- discord/app_commands/commands.py:850: error: Argument 1 to "CommandSignatureMismatch" has incompatible type "Command[GroupT, P, T]"; expected "Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu, Group]"  [arg-type]
- discord/app_commands/commands.py:875: error: Argument 1 to "CommandSignatureMismatch" has incompatible type "Command[GroupT, P, T]"; expected "Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu, Group]"  [arg-type]
- discord/app_commands/commands.py:876: error: Argument 1 to "CommandInvokeError" has incompatible type "Command[GroupT, P, T]"; expected "Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu]"  [arg-type]
- discord/app_commands/commands.py:880: error: Argument 1 to "CommandInvokeError" has incompatible type "Command[GroupT, P, T]"; expected "Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu]"  [arg-type]
- discord/app_commands/commands.py:902: error: Argument 1 to "CommandSignatureMismatch" has incompatible type "Command[GroupT, P, T]"; expected "Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu, Group]"  [arg-type]
- discord/app_commands/commands.py:905: error: Argument 1 to "CommandSignatureMismatch" has incompatible type "Command[GroupT, P, T]"; expected "Union[Command[Any, [VarArg(Any), KwArg(Any)], Any], ContextMenu, Group]"  [arg-type]
- discord/app_commands/commands.py:947: error: Argument 2 to "Parameter" has incompatible type "Command[GroupT, P, T]"; expected "Command[Any, [VarArg(Any), KwArg(Any)], Any]"  [arg-type]
- discord/app_commands/commands.py:968: error: Argument 2 to "Parameter" has incompatible type "Command[GroupT, P, T]"; expected "Command[Any, [VarArg(Any), KwArg(Any)], Any]"  [arg-type]
- discord/app_commands/commands.py:1007: error: Argument 2 to "maybe_coroutine" has incompatible type "Interaction"; expected <nothing>  [arg-type]
- discord/app_commands/commands.py:1013: error: Need type annotation for "ret"  [var-annotated]

... (truncated 127 lines) ...

Expression (https://github.com/cognitedata/Expression)
+ expression/core/fn.py:36: error: List or tuple expected as variadic arguments  [misc]
+ expression/core/fn.py:36: error: Argument after ** must be a mapping, not "Union[Any, _P.kwargs]"  [arg-type]
+ expression/core/fn.py:55: error: List or tuple expected as variadic arguments  [misc]
+ expression/core/fn.py:55: error: Argument after ** must be a mapping, not "Union[Any, _P.kwargs]"  [arg-type]
- expression/core/fn.py:22: error: ParamSpec "_P" is unbound  [valid-type]
- expression/core/fn.py:25: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
- expression/core/fn.py:25: error: Invalid location for ParamSpec "_P"  [valid-type]
- expression/core/fn.py:25: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[_P, int]'
- expression/core/fn.py:33: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
- expression/core/fn.py:33: error: Invalid location for ParamSpec "_P"  [valid-type]
- expression/core/fn.py:33: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[_P, int]'
- expression/core/fn.py:48: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
- expression/core/fn.py:48: error: Invalid location for ParamSpec "_P"  [valid-type]
- expression/core/fn.py:48: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[_P, int]'
- expression/core/fn.py:52: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
- expression/core/fn.py:52: error: Invalid location for ParamSpec "_P"  [valid-type]
- expression/core/fn.py:52: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[_P, int]'

JukkaL

LGTM now!

Clean up from #14159

ilevkivskyi added 5 commits November 20, 2022 23:52

Switch to using bound type variables for type alaises

1a673b0

Support paramspec in aliases

39e6b86

Add a test case for Concatenate

2e02ca0

Tighten unbound type variables even more

910cb38

Update some docs/tests

85ae519

ilevkivskyi requested review from jhance, JukkaL and hauntsaninja November 21, 2022 14:50

This comment has been minimized.

Sign in to view

gandhis1 mentioned this pull request Nov 21, 2022

MyPy reports false error when using Concatenate with ParamSpec #14069

Closed

ilevkivskyi added 5 commits November 21, 2022 19:32

Merge remote-tracking branch 'upstream/master' into paramspec-alias

5fb8862

Fix merge

9435ecd

Fix nested type application

57fc311

Fix some style

fdf4533

Fix Python 3.7

cc50511

This comment has been minimized.

Sign in to view

ilevkivskyi added 3 commits November 22, 2022 00:10

Remove unneeded TODO

a97ba01

Some more tests

ecc700e

Prohibit invalid replacements for ParamSpec

d7357d6

This comment has been minimized.

Sign in to view

Fix trivial parameters

c8259c5

This comment has been minimized.

Sign in to view

Fix checking alias r.h.s. as a runtime expression

36eea80

JukkaL approved these changes Nov 24, 2022

View reviewed changes

ilevkivskyi merged commit 13bd201 into python:master Nov 24, 2022

ilevkivskyi deleted the paramspec-alias branch November 24, 2022 11:22

JukkaL mentioned this pull request Jan 20, 2023

Unbound type variable false positive with six.with_metaclass #14475

Closed

This was referenced Jan 31, 2023

New "Must not defer during final iteration" crash #14565

Closed

Remove dead branch when analysing type aliases #14566

Merged

hauntsaninja added a commit that referenced this pull request Jan 31, 2023

Remove dead branch when analysing type aliases (#14566)

1a781e7

Clean up from #14159

A5rocks mentioned this pull request Feb 25, 2023

Can't use Callable[..., T] | Callable[..., Awaitable[T]] as a function argument #14669

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support ParamSpec variables in type aliases #14159

Support ParamSpec variables in type aliases #14159

ilevkivskyi commented Nov 21, 2022

ilevkivskyi commented Nov 21, 2022

This comment has been minimized.

ilevkivskyi commented Nov 21, 2022 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented Nov 23, 2022

JukkaL commented Nov 23, 2022

gandhis1 commented Nov 23, 2022

JukkaL commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

JukkaL commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

github-actions bot commented Nov 23, 2022

JukkaL left a comment

Support ParamSpec variables in type aliases #14159

Support ParamSpec variables in type aliases #14159

Conversation

ilevkivskyi commented Nov 21, 2022

ilevkivskyi commented Nov 21, 2022

This comment has been minimized.

ilevkivskyi commented Nov 21, 2022 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented Nov 23, 2022

JukkaL commented Nov 23, 2022

gandhis1 commented Nov 23, 2022

JukkaL commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

JukkaL commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

ilevkivskyi commented Nov 23, 2022

github-actions bot commented Nov 23, 2022

JukkaL left a comment

Choose a reason for hiding this comment

ilevkivskyi commented Nov 21, 2022 •

edited

Loading