Make any callable compatible with (*args: Any, **kwargs: Any)?

[JukkaL]

Perhaps a callable that accepts arbitrary arguments by having (*args: Any, **kwargs: Any) could be treated similarly to Callable[..., X] (with explicit ...), i.e. a callable with arbitrary arguments would be compatible with it.

The reason for this is that there's no easy way to annotate a function so that the type would be Callable[..., X], even though this seems to be a somewhat common need. Users sometimes assume that using (*args: Any, **kwargs: Any) will work for this purpose and are confused when it doesn't. python/typing#264 (comment) is a recent example.

Example:

from typing import Any

class A:
    # A method that accepts unspecified arguments and returns int
    def f(self, *args: Any, **kwargs: Any) -> int: pass

class B(A):
    def f(self, x: int) -> int: pass  # Should be okay

We could perhaps extend this to handle decorators as well:

from typing import Any, Callable, TypeVar

T = TypeVar('T', bound=Callable[..., Any])

def deco(f: T) -> T:
    def wrapper(*args: Any, **kwargs: Any) -> Any:
        print('called')
        return f(*args, **kwargs)
    return wrapper  # Maybe this should be okay

[ilevkivskyi]

I actually like this idea, I have seen this confusion several times, and although it is a bit unsafe, most of the time when people write (*args, **kwargs) it means "don't care", rather than "should work for all calls".

[gvanrossum]

Agreed, this is a case where practicality beats purity.

[jonapich]

I just stumbled upon this trying to build a Protocol to require a callable that takes an Event as the first parameter:

class SubprocessTarget(Protocol):
    def __call__(self, quit_event: MultiprocessEvent, *args: Any, **kwargs: Any) -> int: ...

Just throwing this out here, but maybe a variant of Any could be useful for defining callback signatures:

...*args: AnyOrMissing, **kwargs: AnyOrMissing (sorry for the poor choice of name)

Or maybe the call could be decorated with something like @OptionalVarArgs to indicate that we really don't care about args/kwargs? Or the other way around, @PartialSignature to indicate that the method may include additional arguments, then the protocol signature only lists the required arguments (it doesn't have to include args/kwargs anymore).

Another similar (but different) use case would be to require the method to include a specific kwarg (or **kwargs), regardless of its position. I feel these will be hard to implement without the help of some decorators?

[ckyoog]

I do read the following words in mypy docs

Mypy recognizes a special form Callable[..., T] (with a literal ...) which can be used in less typical cases. It is compatible with arbitrary callable objects that return a type compatible with T, independent of the number, types or kinds of arguments. Mypy lets you call such callable values with arbitrary arguments, without any checking – in this respect they are treated similar to a (*args: Any, **kwargs: Any) function signature.

Does it mean this issue has been resolved? Correct me if I am wrong.

[jonapich]

@ckyogg: no, the issue here is to be able to enforce partial signatures to improve duck typing possibilities. Your excerpt is asking for "any callback whatsoever" but i want to define "a callback with a quit flag as first argument and anything else after"

[jlaine]

Has anyone found a good solution for this? My usecase is exactly the same as @jonapich I want to enforce the first argument and allow anything afterwards.

[chadrik]

+1. It seems like the only solution right now is to create a union type alias covering all cases, which is obviously something I'd like to avoid doing!

class CallbackWithKwargs1(Protocol[T_contra, T_co]):
    def __call__(self, __element: T_contra, __arg1=...) -> T_co:
        pass

class CallbackWithKwargs2(Protocol[T_contra, T_co]):
    def __call__(self, __element: T_contra, __arg1=..., __arg2=...) -> T_co:
        pass

...

CallbackWithKwargs=Union[
    CallbackWithKwargs1, CallbackWithKwargs2, ...]

[antonagestam]

Another common real-world example is trying to create a type for a functional view in Django, where the first argument is always a Request and there might be more arguments for URL parameters.

[rmorshea]

@antonagestam perhaps #8263 could help resolve this issue?

[jmehnle]

I really just care about typing a function (in a prototype) with arbitrary arguments (except for perhaps self), and def foo(self, *args: Any, **kwargs: Any) -> Any: ... just doesn't work. What am I to do?

[JelleZijlstra]

@jmehnle does something like foo: Callable[..., Any] work?

[rmorshea]

Tip for achieving this at present:

from typing import Callable, Any


class Parent:
    method: Callable[..., int]
    def method(  # type: ignore
        self, *args: Any, **kwargs: Any,
    ) -> int:
        ...

class Child(Parent):
    def method(self, x: str) -> int:
        ...

Try it: https://mypy-play.net/?mypy=latest&python=3.8&gist=ba254920ee62608c5696c29610b2a379

[jmehnle]

@jmehnle does something like foo: Callable[..., Any] work?

Alas, no. I get: note: Protocol member Connection.cursor expected settable variable, got read-only attribute.

And unfortunately @rmorshea's trick still generates that very same message (see also #9560).

[NeilGirdhar]

Doesn't the recently-added ParamSpec cover the decorator case in the original issue? Is there still a significant need for this?

[hauntsaninja]

It's still an issue for subclasses, which is the main thing in the original post. This is one of mypy's most reacted-to open issues, so definitely some real usability issues here.

[NeilGirdhar]

It's still an issue for subclasses, which is the main thing in the original post.

Are there good examples of when a base class would want to expose a member that accepts arbitrary parameters, but has a derived class that doesn't?

This is one of mypy's most reacted-to open issues, so definitely some real usability issues here.

Some of that reaction may have preceded the addition of ParamSpec, and may be focused on the decorator example, which is extremely important. There's no way to gauge how many people really want base methods with arbitrary parameters that are constrained in derived classes. To my eyes, that is a bad pattern that should be avoided.

[hauntsaninja]

Check out some of the issues that link to here. Yes, this is obviously unsafe if the parent actually accepts everything, which is why this hasn't been done yet. The change itself is trivial, see #11203

[NeilGirdhar]

@hauntsaninja Yeah, I looked at a few. I think MyPy is correct. These are LSP violations and indicative of poor code. It would be better in my opinion to just expand the documentation to explain workarounds. For example, in #9174, the methods do_something on each class should simply have three different names. There was no reason to give them the same name since they can't be used polymorphically.

[hauntsaninja]

Since this has been merged and I got a question about it already, if your parent class actually accepts everything, use object instead of Any to ensure you get LSP errors (*args: object, **kwargs: object)

[gandhis1]

I've got the same issue here, currently writing an interface that accepts a callback that has at least one specific parameter, the others are optional and can be handed to a registering function beforehand. I thus defined my protocol as follows:

from typing import Any

from typing_extensions import Protocol

class Callback(Protocol):
    def __call__(self, gain: bool, *args: Any, **kwargs: Any) -> None:
        ...

But now, every function passed with type Callback needs to explicitly have the structure

def func(param: bool, *args: Any, **kwargs: Any)

although a function

def func(param: bool)

is totally valid as long as no additional arguments were supplied to the registering function before. Is there any solution to this problem yet?

How did you end up resolving this issue? The PR appears to have only solved the case when the entirety of your function signature is args/kwargs, not when you have a partial signature as your example is.

Here is an example of code I am attempting to annotate, with no success. I was able to use ParamSpec/Concatenate to annotate this with pyright, but not with MyPy. But either one should be able to handle Protocol here:

from functools import wraps
from typing import TypeVar, Any, Protocol, Callable


class A():
    pass

class B(A):
    pass


A_co = TypeVar("A_co", bound=A, covariant=True)


class PreFunc(Protocol[A_co]):
    def __call__(self, length_table: int, *args: Any, **kwargs: Any) -> A_co:
        ...


PostFunc = Callable[..., int]

def add_table_to_args(table_name: str) -> Callable[[PreFunc[A_co]], PostFunc]:
    def decorator(func: PreFunc[A_co]) -> PostFunc:
        @wraps(func)
        def wrapped_func(*args: Any, **kwargs: Any) -> int:
            test = func(len(table_name), *args, **kwargs)
            return 4
        return wrapped_func
    return decorator


@add_table_to_args("random string")  # Argument 1 has incompatible type "Callable[[int], B]"; expected "PreFunc[<nothing>]"
def test_function(length_table: int) -> B:
    return B()

[NeilGirdhar]

How did you end up resolving this issue?

For your problem, is it possible to use ParamSpec?

---

The pull request that closed this issue was really about providing some notation for Callable[..., Any]. (Personally, the purist in me wishes a decorator had been required in addition to the function accepting args/kwargs.)

In more general cases where you want to have a generic method that accepts some required parameters, it may be better to choose a design that doesn't have LSP errors, like:

class PreFuncExtra: ...

class PreFunc(Protocol[A_co]):
    def __call__(self, length_table: int, pre_func_extra: PreFuncExtra) -> A_co:
        ...

And then you can simply derive from PreFuncExtra in your derived classes to add data members. Your derive classes should all accept PreFuncExtra, but assert that the correct subtype has been passed in.

Even if you were able to use the arbitrary args/kwargs solution that you want, you would have to assert on types, and you would end up asserting on many more types rather than a single instance of PreFuncExtra.

[gandhis1]

For your problem, is it possible to use ParamSpec?

It certainly is possible - as I noted above, using ParamSpec and Concatenate to do this works as expected in Pyright. It does not work in MyPy. Does MyPy have robust ParamSpec/Concatenate support? It was difficult to ascertain this from the various issues/PRs in this repo.

Making a separate sub-class annotation for every variant of arguments is not really a practical solution for me. This is a decorator that needs to be usable by anyone, anywhere, as long as their function accepts an int first argument and returns a sub-class of A. All other arguments to the decorated function are passed down verbatim. My actual code uses totally different types and has a real purpose behind it, the example I included above was just a minimal test case that paraphrased the issue.

[NeilGirdhar]

This is a decorator that needs to be usable by anyone, anywhere, as long as their function accepts an int first argument

In the case of annotating a decorator, you should use ParamSpec. I think MyPy support is advancing quickly, but it might not be complete yet. You can track its progress using its topic: https://github.com/python/mypy/issues?q=is%3Aissue+is%3Aopen+label%3Atopic-paramspec+

Using PreFuncExtra is just a design pattern to avoid LSP errors and the accompanying design problems in your own class hierarchy.