TypeVar inside ParamSpec doesn't get inferred

[squahtx]

Bug Report

To Reproduce

from typing import Callable, ParamSpec, TypeVar

T = TypeVar("T")

P = ParamSpec("P")
R = TypeVar("R")

def call(f: Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R:
    return f(*args, **kwargs)

def identity(x: T) -> T:
    return x

call(identity, 2)  # error: Argument 2 to "call" has incompatible type "int"; expected "T"

y: int = call(identity, 2)  # error: Incompatible types in assignment (expression has type "T", variable has type "int")

Expected Behavior

Code passes type checking.

Actual Behavior

Mypy emits errors.

Your Environment

• Mypy version used: 0.931
• Python version used: 3.10

[erictraut]

I wouldn't expect this to work. The TypeVar T in the is scoped to the identity function. It therefore needs to be "solved" in the context of that function. Once you assign the input parameters for identity to a ParamSpec, its context is lost, and the TypeVar loses all meaning. There is no longer any association between the TypeVar captured by P and the TypeVar captured by R.

Edit: I take back what I said above. This should be solvable.

[squahtx]

That's unfortunate. My naive expectations were informed by TypeScript, which does figure out the T.

More concretely, I ran into the issue with code along the lines of run_on_thread(shutil.copyfile, src, dst), where shutil.copyfile has a TypeVar.

[flisboac]

I got into a very similar scenario these days, the only difference being that I was not using this double level of TypeVars (ie. identity would have only concrete types).

So I tried to exercise this in a different lens, to see if I can understand why this is happening (because I may fall into this same use case in the future):

from typing import Callable, ParamSpec, TypeVar, Generic, Final, Protocol

T = TypeVar("T")

P = ParamSpec("P")
R = TypeVar("R")
C = TypeVar("C", covariant=True)

class Builder(Protocol[P, C]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> C: ...

class Call:
    def __getitem__(self, callable: Callable[P, R]) -> Build[P, R]:
        def wrapper(*args: P.args, **kwargs: P.kwargs):
            nonlocal callable
            return callable(*args, **kwargs)
        return wrapper

call: Final = Call()

def identity(x: T) -> T:
    return x


# Instead of trying to do everything at once, I tried to break the
# operation in two steps, statically (a waste, runtime-wise, I admit).


# According to the revealed type, the __getitem__ is properly generic
# (because of the `def [P, R] ...`). I'm not sure what  "P`71" and "R`72" mean;
# is this the notation for a "locally" bound type variable (I suppose it is,
# and the numbers refer to the instantiations)?
reveal_type(call.__getitem__)  # ==> note: Revealed type is "def [P, R] (def (*P.args, **P.kwargs) -> R`72) -> __main__.Builder[P`71, R`72]"


# Question here is: what does mypy generate when we parameterize a generic
# function with another generic function?
wrapper = call[identity]
reveal_type(wrapper)  # ==> note: Revealed type is "__main__.Builder[def [T] (x: T`-1), T`-1]"


# It seems that, with a generic protocol (and maybe with other generic types
# as well), the ParamSpec is propagated as a generic, but the return type is
# dissociated from the protocol's second type parameter -- even though the
# first argument and the return type were inferred to be the same.
# Going forward, the two inferred types in Builder are effectively distinct.
# For this idiom to work, ParamSpec should be able to pass along the whole
# initial function signature, ie. including the return type, which PEP-612 does
# not seem to support.
#
# I also tried a non-protocol (ie. returning "Callable[P, R]") variant for Call.__getitem__;
# the revealed type for "wrapper" is "def (x: T`-1) -> T`-1", which is not generic.
# I'm not sure if the same reasoning could be applied in this case.


wrapper(2)  # error: Argument 1 to "call" has incompatible type "int"; expected "T"

y: int = call[identity](2)  # error: Incompatible types in assignment (expression has type "T", variable has type "int")

@erictraut Any chance a scenario like this would be supported in the future (in a new PEP or otherwise)?

[erictraut]

It's probably best to ask the authors of PEP 612. They're members of the team responsible for the pyre type checker, and they might have additional insights here. cc @pradeep90

[JukkaL]

I'm hoping that we can support this. There seems to be no particular reason why this shouldn't work, but we may need a bit of special logic for this particular use case.

[ilevkivskyi]

Note that although #15896 will fix this, you will still likely need to use --new-type-inference for the fix to work.