Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better type inference for decorators. #774

Closed
giyyapan opened this issue Jun 29, 2020 · 9 comments
Closed

Better type inference for decorators. #774

giyyapan opened this issue Jun 29, 2020 · 9 comments
Labels
as designed Not a bug, working as intended

Comments

@giyyapan
Copy link

Is your feature request related to a problem? Please describe.
Decorated functions will lose all of its signatures if used with decorators. I know some decorators might change the signature of the original function, but most of them are just using *args and **kwargs to proxy all the parameters.

Please see this example:
image

from functools import wraps

def some_decorator_generator():
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            return func(*args, **kwargs)
        return wrapper
    return decorator

def some_decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@some_decorator_generator()
def fn1(a: int):
    pass

@some_decorator
def fn2(a: int):
    pass

fn1() # parameter a is not provided, but no error reported
fn2() # parameter a is not provided, but no error reported

reveal_type(fn2)

Describe the solution you'd like
Not sure if it's easy, but I hope a function decorated by decorators that only proxy all the parameters can keep its original signatures. this will make decorators a lot safer to use.

Thanks!

@giyyapan giyyapan added the enhancement request New feature or request label Jun 29, 2020
@erictraut
Copy link
Collaborator

Pyright is doing the right thing in this case. It is inferring as much as it can given the information it is provided within your code plus the stdlib type stubs.

The wraps decorator in functools.pyi is declared as follows:

_AnyCallable = Callable[..., Any]

def wraps(wrapped: _AnyCallable, assigned: Sequence[str] = ..., updated: Sequence[str] = ...) -> Callable[[_AnyCallable], _AnyCallable]: ...

Note that _AnyCallable is not a TypeVar, so there's no generic type matching here. _AnyCallable is simply a type alias for Callable[..., Any]. Because of this definition, the @wraps(func) always results in a decorated function type of Callable[..., Any]. Pyright has nothing to infer in this case because a return type is provided. Inference is used only in cases where a type is omitted and Pyright needs to infer it from other information.

I don't know enough about functools or the wraps decorator method to say whether this is a bug in the type stub. Should it be using a TypeVar that is bound to an _AnyCallable? If you think this is a bug, please report it in the typeshed repo.

In any case, there's a simple workaround that you can apply to your sample. Define a TypeVar that is bound to a function type, and use it in the declaration of decorator and some_decorator.

_TFunc = TypeVar("_TFunc", bound=Callable[..., Any])

def some_decorator_generator():
    def decorator(func: _TFunc) -> _TFunc:
       ...

def some_decorator(func: _TFunc) -> _TFunc:
 ...

@erictraut erictraut added as designed Not a bug, working as intended and removed enhancement request New feature or request labels Jun 30, 2020
@giyyapan
Copy link
Author

giyyapan commented Jul 1, 2020

Thank you so much for your clear explanation and brilliant suggestion! That's exactly what I need. ❤️

@moi90
Copy link

moi90 commented Jan 6, 2021

If I do this:

def decorator(f: _TFunc) -> _TFunc:
    def wrapper(*args, **kwargs):
        print("wrapper", *args, **kwargs)
        return f(*args, **kwargs)

    return wrapper

I get the following error (at the return wrapper line):

Expression of type "(*args: Unknown, **kwargs: Unknown) -> Any" cannot be assigned to return type "_TFunc"
  Type "(*args: Unknown, **kwargs: Unknown) -> Any" cannot be assigned to type "_TFunc" Pylance(reportGeneralTypeIssues)

Did I do something wrong? Can this error be removed?

@erictraut
Copy link
Collaborator

In Python 3.9, there's not a great solution to this problem. The best workaround I can offer is the following:

    return cast(_TFunc, wrapper)

In Python 3.10, there's a new facility called a ParamSpec. It's described in PEP 612. You can use it in older versions of Python through the typing_extensions package.

Here's how it would look with the new capability:

from typing import Callable, TypeVar
from typing_extensions import ParamSpec

_P = ParamSpec("_P")
_R = TypeVar("_R")


def decorator(f: Callable[_P, _R]) -> Callable[_P, _R]:
    def wrapper(*args: _P.args, **kwargs: _P.kwargs):
        print("wrapper", *args, **kwargs)
        return f(*args, **kwargs)

    return wrapper

@moi90
Copy link

moi90 commented Jan 7, 2021

Thanks for the explanation and the hint to PEP 612! It's great to see how the language keeps evolving.

@cmpute
Copy link

cmpute commented Mar 29, 2021

Just side note here: support for PEP 612 in typing_extensions is still in progress (python/typing#774)

@tarheels100
Copy link

tarheels100 commented Jan 18, 2023

In Python 3.9, there's not a great solution to this problem. The best workaround I can offer is the following:

    return cast(_TFunc, wrapper)

In Python 3.10, there's a new facility called a ParamSpec. It's described in PEP 612. You can use it in older versions of Python through the typing_extensions package.

Here's how it would look with the new capability:

from typing import Callable, TypeVar
from typing_extensions import ParamSpec

_P = ParamSpec("_P")
_R = TypeVar("_R")


def decorator(f: Callable[_P, _R]) -> Callable[_P, _R]:
    def wrapper(*args: _P.args, **kwargs: _P.kwargs):
        print("wrapper", *args, **kwargs)
        return f(*args, **kwargs)

    return wrapper

Is there a way to apply this for class decorators as well? I lose intellisense documentation and autosuggestions when I decorate a class. I'm in python 3.10.5 by the way.

@erictraut
Copy link
Collaborator

If you'd like help annotating your class decorator, please post a question to the discussions forum and include the unannotated code.

@liambob77
Copy link

In Python 3.9, there's not a great solution to this problem. The best workaround I can offer is the following:

    return cast(_TFunc, wrapper)

In Python 3.10, there's a new facility called a ParamSpec. It's described in PEP 612. You can use it in older versions of Python through the typing_extensions package.
Here's how it would look with the new capability:

from typing import Callable, TypeVar
from typing_extensions import ParamSpec

_P = ParamSpec("_P")
_R = TypeVar("_R")


def decorator(f: Callable[_P, _R]) -> Callable[_P, _R]:
    def wrapper(*args: _P.args, **kwargs: _P.kwargs):
        print("wrapper", *args, **kwargs)
        return f(*args, **kwargs)

    return wrapper

Is there a way to apply this for class decorators as well? I lose intellisense documentation and autosuggestions when I decorate a class. I'm in python 3.10.5 by the way.

I'm in python 3.12.2, I did similar modification on my class base decoreator like following

def __call__(self, func: Callable[_P, _R]) -> Callable[_P, _R]:
    @functools.wraps(func)
    def reporter(*_args: _P.args, **_kwargs: _P.kwargs) -> _R | Result:

Pylance and mypy are happy, before this modification, Pylance is reporting "reportMissingParameterType"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
as designed Not a bug, working as intended
Projects
None yet
Development

No branches or pull requests

6 participants