Basic ParamSpec Concatenate and literal support #11847

A5rocks · 2021-12-26T05:59:24Z

Description

I was planning on adding a ParamSpec-based interface to something I'm making but... it turns out I need Concatenate and Z[[int]] syntax.

Closes #11833
Closes #12276 (needs test)
Closes #12257 (needs test; not sure how to test cache issues)
Refs #8645
External ref python/typeshed#4827

This PR adds a new Parameters proper type to represent parameters (more later in this description), along with adding a Concatenate operator.

Test Plan

I copied all (seemingly relevant) tests from PEP 612. I plan on copying test cases from issues and thinking of a few of my own later.

Notes

(not a real section but seemed important enough)

Concatenate support does not use a ProperType. That would be too much new code for an operator, in my opinion. Thus, it prepends to a ParamSpecType's .prefix.
I would really really like CallableType to take in a parameters: ParamSpecType | Parameters -- this would simplify some code, and it fits better in your head. However, this would be a breaking change, so I did not do this (along with the fact that making basic changes in this direction led to 300 type errors when self-checking).
~~- Concatenate was super unspecified by the PEP in my opinion. I ended up making it work wherever ParamSpec could be (or so I think)... Which means Concatenate[int, Concatenate[str, P]] works!~~
Error messages could be improved, but I'm not sure where I can get the context to improve those.

Now this program works: ```py from typing_extensions import ParamSpec from typing import Generic, Callable P = ParamSpec("P") class Z(Generic[P]): def call(self, *args: P.args, **kwargs: P.kwargs) -> None: pass n: Z[[int]] reveal_type(n) # N: Revealed type is "repro.Z[[builtins.int]]" def f(n: Z[P]) -> Z[P]: ... reveal_type(f) # N: Revealed type is "def [P] (n: repro.Z[P`-1]) -> repro.Z[P`-1]" reveal_type(f(n)) # N: Revealed type is "repro.Z[[builtins.int]]" ```

This program: ```py from typing_extensions import ParamSpec, Concatenate from typing import Generic P = ParamSpec("P") class Z(Generic[P]): ... n: Z[[int]] reveal_type(n) def f(n: Z[P]) -> Z[Concatenate[bool, Concatenate[str, P]]]: ... reveal_type(f) reveal_type(f(n)) ``` outputs: ``` repro.py:10: note: Revealed type is "repro.Z[[builtins.int]]" repro.py:14: note: Revealed type is "def [P] (n: repro.Z[P`-1]) -> repro.Z[Concatenate[builtins.bool, builtins.str, P`-1]]" repro.py:15: note: Revealed type is "repro.Z[[builtins.bool, builtins.str, builtins.int]]" ``` Next up: checking inputs match prefixes and why does it only work when cache exists?

It wasn't actually cache...

mypy/typeanal.py

test-data/unit/check-parameter-specification.test

jhance · 2022-03-17T04:48:53Z

mypy/meet.py

@@ -506,6 +506,17 @@ def visit_param_spec(self, t: ParamSpecType) -> ProperType:
        else:
            return self.default(self.s)

+    def visit_parameters(self, t: Parameters) -> ProperType:
+        # TODO: is this the right variance?


I think it is correct, we need any instance of the return type to be compatible iwth both, which seems to be the intersection of their arguments e.g. meet_types.

Also, if its possible to test these independently (I assume we can add tests in the semanal section that simply test this, if not thats a problem in itself) the "Add all the visitors" section of the diff should probably be migrated to a separate PR with its own semanal tests.

jhance · 2022-03-17T04:54:12Z

mypy/typeanal.py

+
+        # This is a heuristic: it will be checked later anyways but the error
+        # message may be worse.
+        with self.set_allow_param_spec_literals(info.has_param_spec_type):


Existing precedent seems to be that anal_type does its own pushing/popping.

IMO (and we can do this in a separate PR so that this one is a minor tweak) we should do something like having the parameters to anal_type be some sort of dataclass, and then allow passing mutators to the dataclass to anal_array at which point it will push/pop the options. This avoids having N things that are pushing/popping to some sort of stack.

Its also really confusing how the state set by set_allow_param_spec_literals compares to the value allow_param_spec send to self.anal_array and self.anal_type. If the distinction is necessary it should be documented somewhere what the difference is.

ygale · 2022-03-22T05:28:55Z

@A5rocks Great work, thanks! I also hit the same roadblock, and I'm glad to see that you are working on it.

Your idea to generalize where Concatenate can occur is interesting. Do you have any code samples where this would be useful? The thing is, both PEP 612 and the Python Library docs explicitly forbid this. The CPython interpreter rejects your sample usage of that syntax with

TypeError: The last parameter to Concatenate should be a ParamSpec variable.

So to do this, we would need to submit PRs to change the CPython source code, the docs, and the PEP. For those to have a chance of being accepted, we would need convincing justifications of the need for the change, not just "because it was there". mypy would still need to reject that syntax for now, but we could guard the rejection by version numbers, in anticipation of removing the restriction in future versions of Python after the PRs are accepted.

Truthfully, I'm not really sure why this syntax is needed, other than aesthetics, because Concatenate[int, Concatenate[str, P]] seems to be equivalent to Concatenate[int, str, P]. But if you have an idea about why it is needed, I'd be happy to help out with the PRs.

EDIT: To be clear, the simplest solution is to stick with the current syntax: Concatenate is allowed only as the first parameter of Callable, nowhere else. And the last parameter of Concatenate must be a ParamSpec variable. Programs that violate either of those rules are rejected by the type checker.

JukkaL · 2022-03-22T12:51:11Z

@A5rocks This close to being ready to be merged! If you can fix the merge conflicts and CI failures, and address feedback in the next 1 week or so, we could probably include this in the next mypy feature release (planned to be out in the first half of April).

A5rocks · 2022-03-26T01:52:24Z

@ygale

The thing is, both PEP 612 and the Python Library docs explicitly forbid this.

Yep I just somehow misread the syntax definition in PEP 612, fixed.

@cdce8p

In my last comment I mentioned one more issue I wasn't able to isolate. Since then I did take another look at it. From the looks of it, it seems to be caused by a combination of overload, a Generic class, and passing an invalid type with type: ignore. Maybe you've an idea.

Looked a bit at it: so the overload resolution is failing for exactly what you predicted: the type ignore. In a nutshell, the overloads are checked through until the first without type errors upon calling it -- this yields 'repro.py:39: error: Argument 2 to "run_job" has incompatible type "str"; expected "int"' for the second overload due to a lacking type: ignore in the synthesized check. Thus, mypy thinks none of the overloads work and then chooses the first (arbitrary choice, just for typing reasons).

I'm not sure a solution would be in scope for this PR (as a proper solution would involve propagating type ignores)

EDIT: filed #12454

github-actions · 2022-03-26T02:16:02Z

Diff from mypy_primer, showing the effect of this PR on open source code:

core (https://github.com/home-assistant/core)
+ homeassistant/helpers/event.py:116: error: Unused "type: ignore" comment
+ homeassistant/helpers/event.py:117: error: Unused "type: ignore" comment
+ homeassistant/components/tplink/entity.py:22: error: Unused "type: ignore" comment
+ homeassistant/components/tplink/entity.py:23: error: Unused "type: ignore" comment
+ homeassistant/components/sonarr/sensor.py:112: error: Unused "type: ignore" comment
+ homeassistant/components/sonarr/sensor.py:113: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:18: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:19: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:27: error: Unused "type: ignore" comment
+ homeassistant/components/zabbix/__init__.py:93: error: Argument 3 has incompatible type "Callable[[datetime], bool]"; expected "Union[HassJob[Optional[Awaitable[None]]], Callable[[datetime], Optional[Awaitable[None]]]]"  [arg-type]
+ homeassistant/components/zabbix/__init__.py:93: error: Incompatible return value type (got "bool", expected "Optional[Awaitable[None]]")  [return-value]
+ homeassistant/components/plugwise/util.py:18: error: Unused "type: ignore" comment
+ homeassistant/components/plugwise/util.py:19: error: Unused "type: ignore" comment
+ homeassistant/components/roku/helpers.py:32: error: Unused "type: ignore" comment
+ homeassistant/components/roku/helpers.py:33: error: Unused "type: ignore" comment
+ homeassistant/components/webostv/media_player.py:101: error: Unused "type: ignore" comment
+ homeassistant/components/webostv/media_player.py:102: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:38: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:47: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:55: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:61: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:62: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/media.py:138: error: Argument 1 to "int" has incompatible type "Optional[Any]"; expected "Union[str, bytes, SupportsInt, SupportsIndex, SupportsTrunc]"  [arg-type]
+ homeassistant/components/vlc_telnet/media_player.py:79: error: Unused "type: ignore" comment
+ homeassistant/components/vlc_telnet/media_player.py:80: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:82: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:83: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:95: error: Unused "type: ignore" comment

cdce8p · 2022-03-27T22:19:35Z

Looked a bit at it: so the overload resolution is failing for exactly what you predicted: the type ignore. In a nutshell, the overloads are checked through until the first without type errors upon calling it -- this yields 'repro.py:39: error: Argument 2 to "run_job" has incompatible type "str"; expected "int"' for the second overload due to a lacking type: ignore in the synthesized check. Thus, mypy thinks none of the overloads work and then chooses the first (arbitrary choice, just for typing reasons).

Thanks for taking the time! I already left a #12454 (comment), but the gist is that I should just fix the argument types itself. So not an issue for this PR.

--
Not completely sure what's left here. All major issues seem to be fixed and the primer output looks good. There is the TypeAlias support that's still missing #11855 and some other (minor) issues which might not be fixed yet: topic-paramspec PEP 612, ParamSpec, Concatenate
As you said though, these can be addressed later. It would be awesome if the PR would be part of the next release.

The failed windows CI tests seem unrelated. They fail on the master branch, too.

cdce8p · 2022-03-31T06:01:34Z

Windows tests have been fixed in #12477

test-data/unit/check-parameter-specification.test

Co-authored-by: Marc Mueller <[email protected]>

pros: - more readable IMO - more concise - standardization of output cons: - can be easy to miss double backets - I feel like this is a small deviation from normal mypy representations... but I have nothing to back that up. Overall, I think it's a good thing to try at least.

github-actions · 2022-04-05T01:45:33Z

Diff from mypy_primer, showing the effect of this PR on open source code:

core (https://github.com/home-assistant/core)
+ homeassistant/helpers/event.py:116: error: Unused "type: ignore" comment
+ homeassistant/helpers/event.py:117: error: Unused "type: ignore" comment
+ homeassistant/components/tplink/entity.py:22: error: Unused "type: ignore" comment
+ homeassistant/components/tplink/entity.py:23: error: Unused "type: ignore" comment
+ homeassistant/components/sonarr/sensor.py:112: error: Unused "type: ignore" comment
+ homeassistant/components/sonarr/sensor.py:113: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:18: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:19: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:27: error: Unused "type: ignore" comment
+ homeassistant/components/zabbix/__init__.py:93: error: Argument 3 has incompatible type "Callable[[datetime], bool]"; expected "Union[HassJob[Optional[Awaitable[None]]], Callable[[datetime], Optional[Awaitable[None]]]]"  [arg-type]
+ homeassistant/components/zabbix/__init__.py:93: error: Incompatible return value type (got "bool", expected "Optional[Awaitable[None]]")  [return-value]
+ homeassistant/components/plugwise/util.py:18: error: Unused "type: ignore" comment
+ homeassistant/components/plugwise/util.py:19: error: Unused "type: ignore" comment
+ homeassistant/components/roku/helpers.py:32: error: Unused "type: ignore" comment
+ homeassistant/components/roku/helpers.py:33: error: Unused "type: ignore" comment
+ homeassistant/components/webostv/media_player.py:101: error: Unused "type: ignore" comment
+ homeassistant/components/webostv/media_player.py:102: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:38: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:47: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:55: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:61: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:62: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/media.py:138: error: Argument 1 to "int" has incompatible type "Optional[Any]"; expected "Union[str, bytes, SupportsInt, SupportsIndex, SupportsTrunc]"  [arg-type]
+ homeassistant/components/vlc_telnet/media_player.py:79: error: Unused "type: ignore" comment
+ homeassistant/components/vlc_telnet/media_player.py:80: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:82: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:83: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:95: error: Unused "type: ignore" comment

github-actions · 2022-04-05T01:59:33Z

Diff from mypy_primer, showing the effect of this PR on open source code:

core (https://github.com/home-assistant/core)
+ homeassistant/helpers/event.py:116: error: Unused "type: ignore" comment
+ homeassistant/helpers/event.py:117: error: Unused "type: ignore" comment
+ homeassistant/components/tplink/entity.py:22: error: Unused "type: ignore" comment
+ homeassistant/components/tplink/entity.py:23: error: Unused "type: ignore" comment
+ homeassistant/components/sonarr/sensor.py:112: error: Unused "type: ignore" comment
+ homeassistant/components/sonarr/sensor.py:113: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:18: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:19: error: Unused "type: ignore" comment
+ homeassistant/components/evil_genius_labs/util.py:27: error: Unused "type: ignore" comment
+ homeassistant/components/zabbix/__init__.py:93: error: Argument 3 has incompatible type "Callable[[datetime], bool]"; expected "Union[HassJob[Optional[Awaitable[None]]], Callable[[datetime], Optional[Awaitable[None]]]]"  [arg-type]
+ homeassistant/components/zabbix/__init__.py:93: error: Incompatible return value type (got "bool", expected "Optional[Awaitable[None]]")  [return-value]
+ homeassistant/components/plugwise/util.py:18: error: Unused "type: ignore" comment
+ homeassistant/components/plugwise/util.py:19: error: Unused "type: ignore" comment
+ homeassistant/components/roku/helpers.py:32: error: Unused "type: ignore" comment
+ homeassistant/components/roku/helpers.py:33: error: Unused "type: ignore" comment
+ homeassistant/components/webostv/media_player.py:101: error: Unused "type: ignore" comment
+ homeassistant/components/webostv/media_player.py:102: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:38: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:47: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:55: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:61: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/helpers.py:62: error: Unused "type: ignore" comment
+ homeassistant/components/sonos/media.py:138: error: Argument 1 to "int" has incompatible type "Optional[Any]"; expected "Union[str, bytes, SupportsInt, SupportsIndex, SupportsTrunc]"  [arg-type]
+ homeassistant/components/vlc_telnet/media_player.py:79: error: Unused "type: ignore" comment
+ homeassistant/components/vlc_telnet/media_player.py:80: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:82: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:83: error: Unused "type: ignore" comment
+ homeassistant/components/dlna_dmr/media_player.py:95: error: Unused "type: ignore" comment

JukkaL

This looks good now! The core functionality looks solid, and it will be great to have this in the next release. Remaining corner cases and TODOs can be worked on in separate PRs.

Thank you for working on this -- this is the biggest remaining 3.10 feature that mypy has been missing.

A5rocks · 2022-04-07T10:54:30Z

Thanks!

FYI, #12096 should be checked out (I put similar code in the else branch rather than a new elif, not realizing that typevarlikes other than typevar could be invariant, like typevartuple)

AlexWaygood · 2022-04-07T11:07:02Z

Just want to say that I'm really excited to see this get merged — thanks for all your work on this @A5rocks!

A5rocks added 10 commits December 24, 2021 16:29

Add ParamSpec literals

ef32680

Fix "cache" bug

58e6dbe

It wasn't actually cache...

Check Concatenate prefixes

51ba4ea

Polish work

24432ee

Merge branch 'master' into paramspec-literals

c507152

Tests for literals

d202d1e

Tests for Concatenate

9ed9830

Appease CI

3ffc343

A5rocks commented Dec 26, 2021

View reviewed changes

mypy/typeanal.py Outdated Show resolved Hide resolved

Forgot to comment out the directives...

ae8ac73

A5rocks force-pushed the paramspec-literals branch from 1e06238 to ae8ac73 Compare December 26, 2021 06:46

Improve literal TODOs

9c849cc

A5rocks marked this pull request as ready for review December 27, 2021 01:53

A5rocks added 6 commits December 28, 2021 12:33

Add more tests

d9dcc76

Allow TypeVars in Concatenate

0e2b207

Fix a couple of dumb oversights

bd445e5

Allow Callables along with Parameters

604c304

Fix tests

f8004ec

Misc changes

9e75481

A5rocks commented Dec 31, 2021

View reviewed changes

test-data/unit/check-parameter-specification.test Show resolved Hide resolved

A5rocks added 2 commits January 1, 2022 14:14

Solve with self types

7b89f06

Add fallback return to meeting paramspec literals

f24cf4f

FasterSpeeding mentioned this pull request Jan 2, 2022

Improve typing around callbacks with dynamic keyword-arguments FasterSpeeding/Tanjun#176

Merged

2 tasks

A5rocks and others added 5 commits January 3, 2022 10:30

Type application of ParamSpec literals

472b20c

Ellipsis paramspec literals

14ecfb9

Merge branch 'master' into paramspec-literals

5e0ae49

Appease flake8

45c8057

Merge branch 'master' into paramspec-literals

10966ea

jhance reviewed Mar 17, 2022

View reviewed changes

A5rocks added 3 commits March 26, 2022 08:54

Some of the PR feedback

4475515

Merge branch 'master' into paramspec-literals

0091762

Prepare for GitHub actions

81994f1

A5rocks mentioned this pull request Mar 26, 2022

Overloads do not take type: ignore into account #12454

Closed

clokep mentioned this pull request Mar 28, 2022

_do_execute (database function) could be more precisely typed with better support for ParamSpec matrix-org/synapse#12312

Closed

cdce8p reviewed Mar 31, 2022

View reviewed changes

test-data/unit/check-parameter-specification.test Outdated Show resolved Hide resolved

A5rocks and others added 3 commits April 5, 2022 10:09

Merge branch 'master' into paramspec-literals

1ff96c1

Fix tests to latest output

c79918e

Co-authored-by: Marc Mueller <[email protected]>

JukkaL approved these changes Apr 7, 2022

View reviewed changes

JukkaL merged commit 07d8878 into python:master Apr 7, 2022

This was referenced Apr 7, 2022

Decorated function leaks type variables #11816

Closed

"Parameters cannot be constrained" with generic ParamSpec #12542

Closed

AlexWaygood mentioned this pull request Apr 7, 2022

@final and ParamSpec #12033

Closed

PIG208 mentioned this pull request Apr 13, 2022

typing: Strengthen Type Signature of Decorators with ParamSpec. zulip/zulip#21782

Merged

emmatyping mentioned this pull request May 21, 2022

PEP 612 tracking issue #8645

Closed

A5rocks deleted the paramspec-literals branch May 15, 2023 21:58

brianschubert mentioned this pull request Sep 25, 2024

Crash when using ParamSpec #17825

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic ParamSpec Concatenate and literal support #11847

Basic ParamSpec Concatenate and literal support #11847

A5rocks commented Dec 26, 2021 •

edited

Loading

jhance Mar 17, 2022

jhance Mar 17, 2022

jhance Mar 17, 2022 •

edited

Loading

jhance Mar 17, 2022

ygale commented Mar 22, 2022 •

edited

Loading

JukkaL commented Mar 22, 2022

A5rocks commented Mar 26, 2022 •

edited

Loading

github-actions bot commented Mar 26, 2022

cdce8p commented Mar 27, 2022

cdce8p commented Mar 31, 2022

github-actions bot commented Apr 5, 2022

github-actions bot commented Apr 5, 2022

JukkaL left a comment

A5rocks commented Apr 7, 2022

AlexWaygood commented Apr 7, 2022

Basic ParamSpec Concatenate and literal support #11847

Basic ParamSpec Concatenate and literal support #11847

Conversation

A5rocks commented Dec 26, 2021 • edited Loading

Description

Test Plan

Notes

jhance Mar 17, 2022

Choose a reason for hiding this comment

jhance Mar 17, 2022

Choose a reason for hiding this comment

jhance Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

jhance Mar 17, 2022

Choose a reason for hiding this comment

ygale commented Mar 22, 2022 • edited Loading

JukkaL commented Mar 22, 2022

A5rocks commented Mar 26, 2022 • edited Loading

github-actions bot commented Mar 26, 2022

cdce8p commented Mar 27, 2022

cdce8p commented Mar 31, 2022

github-actions bot commented Apr 5, 2022

github-actions bot commented Apr 5, 2022

JukkaL left a comment

Choose a reason for hiding this comment

A5rocks commented Apr 7, 2022

AlexWaygood commented Apr 7, 2022

A5rocks commented Dec 26, 2021 •

edited

Loading

jhance Mar 17, 2022 •

edited

Loading

ygale commented Mar 22, 2022 •

edited

Loading

A5rocks commented Mar 26, 2022 •

edited

Loading