Callables like operator.call produce arg-type error with generic or overloaded functions

[gschaffner]

Bug Report

operator.call and similarly-typed callables produce an arg-type error when used with many generic functions.

Additionally, if the arg-type error is # type: ignore'd, the inferred type is a TypeVar (removed from its scope!) rather than Any.

To Reproduce

import operator
from collections.abc import Callable
from typing import TypeVar
from typing import assert_type
from typing import reveal_type

T = TypeVar("T")
T0 = TypeVar("T0")
U = TypeVar("U")  # this is redundant, but it helps understand mypy's output


def call2(cb: Callable[[T0], T], arg0: T0, /) -> T:
    # similar to operator.call, but only takes one posarg (doesn't use ParamSpec)
    return cb(arg0)


def ident(x: U) -> U:
    return x


assert_type(operator.call(ident, "foo"), str)
# error: Expression is of type "U", not "str"  [assert-type]
# error: Argument 2 to "call" has incompatible type "str"; expected "U"  [arg-type]

assert_type(call2(ident, "foo"), str)
# error: Expression is of type "U", not "str"  [assert-type]
# error: Argument 1 to "call" has incompatible type "Callable[[U], U]"; expected "Callable[[str], U]"  [arg-type]

x = operator.call(ident, "foo")  # type: ignore[arg-type]
reveal_type(x)  # should be inferred as Any
# note: Revealed type is "U`-1"

(https://mypy-play.net/?mypy=latest&python=3.11&gist=a66524b9632714157bd87ef422725e41)

Expected Behavior

The inferred type of operator.call(ident, "foo") should match the inferred type of ident("foo"). (Aside: should this be str or Literal["foo"]?)

Comparing to other type checkers, call2(ident, "foo") works with pyre (inferred as Literal["foo"]), sort-of-works with pyright (inferred as U@ident | str), and doesn't work with pytype (inferred as Any).

Also, if # type: ignore[arg-type] is used on the arg-type error, the inferred type should by Any, not a TypeVar removed from its scope.

Actual Behavior

See comments in the reproducer above.

Your Environment

• Mypy version used: 0.991 (compiled) and latest master (uncompiled)
• Mypy command-line flags: none required to reproduce
• Mypy configuration options from mypy.ini (and other config files): none required to reproduce
• Python version used: 3.11, 3.10, 3.9, 3.8, 3.7

[tmke8]

It works when the definition of call2 only takes a single TypeVar:

T = TypeVar("T")
U = TypeVar("U")

def call2(cb: Callable[[T], T], arg0: T, /) -> T:
    return cb(arg0)

def ident(x: U) -> U:
    return x

assert_type(call2(ident, "foo"), str)

So, I guess mypy has trouble realizing it should assign U to both T0 and T in your example?

[gschaffner]

yeah, the problem seems to be that Mypy isn't binding TypeVars of generic functions when they are callback functions (rather than being the function being called).

[gschaffner]

overloads aren't getting resolved either:

from operator import call
from typing import Literal
from typing import overload


@overload
def foo(*, flag: Literal[True], extra_if_flag: int = ...) -> None:
    ...


@overload
def foo(*, flag: Literal[False] = ...) -> None:
    ...


def foo(*, flag: bool = False, extra_if_flag: int = 0) -> None:
    ...


foo()
call(foo)  # error: Missing named argument "flag" for "call"  [call-arg]

foo(flag=False)
call(foo, flag=False)  # error: Argument "flag" to "call" has incompatible type "Literal[False]"; expected "Literal[True]"  [arg-type]

foo(flag=True)
call(foo, flag=True)

foo(flag=True, extra_if_flag=0)
call(foo, flag=True, extra_if_flag=0)

(https://mypy-play.net/?mypy=1.1.1&python=3.11&gist=590d1fbbfdc8afd03a82d7877bbea61b)

[ilevkivskyi]

The original example (with generics) passes on current master with --new-type-inference. The other example from comments (with overloads) still fails some calls, with errors like:

test.py:24: error: Argument 1 to "call" has incompatible type overloaded function; expected "Callable[[bool], None]"  [arg-type]
test.py:27: error: Argument 1 to "call" has incompatible type overloaded function; expected "Callable[[bool], None]"  [arg-type]
test.py:30: error: Argument 1 to "call" has incompatible type overloaded function; expected "Callable[[bool, int], None]"  [arg-type]

One can argue that mypy is overly strict here (the error goes away if you use regular position-or-name arguments in definition of foo()). But handling argument kinds is tricky with ParamSpec (unless we admit some loss of type safety in similar but incorrect examples).

I recommend opening a specific separate issue about this remaining edge case if you think it is important. (cc @JukkaL this is precisely the corner case I mentioned in #15896)