Inferring the fields of a dataclass type from its statically-typed __init__ signature.

[rsokl]

I would love for pyright (and all type checkers) to be able to infer the fields on a dynamically-generated dataclass type, based on its statically-typed __init__ signature. E.g.

from typing import Any, ClassVar, ParamSpec, Protocol, Callable


P = ParamSpec("P")

# Proposal: expose a dataclass protocol whose fields can be inferred from its `__init__` signature.
class DataClass(Protocol[P]):
    __dataclass_fields__: ClassVar[dict[str, Any]]
    def __init__(self, *a: P.args, **k:P.kwargs) -> None:
        ...


def make_dataclass_from_func_sig(fn: Callable[P, Any]) -> type[DataClass[P]]:
    # returns some call to `dataclasses.make_dataclass`
    ...

def foo_fn(x: int, y: str = "a"):
    ...

Foo = make_dataclass_from_func_sig(foo_fn)

foo = Foo(x=1)

foo.x  # pyright should see `foo.x` as int
foo.y  # pyright should see `foo.y` as str
foo.z  # pyright should report Member "z" is unknown

As you well know, PEP 681 enables you to statically declare fields on a class to inform the __init__ of that class in the eyes of type checkers. What I am proposing is enabling this to go in the opposite direction, so to speak.

Motivation

Many experiment / configuration framework libraries (including hydra-zen) have gravitated towards using dataclass types to represent "configs" for functions. The following pattern is quite common:

from dataclasses import dataclass

@dataclass
class Config:
    x: int
    y: str


def func(config: Config):
    # this might be a very expensive function
    ...

Config is acting like a floating, mutable signature for func. Its defaults can be modified/overwritten via inheritance, and you can create and store many instances of Config as a way to call func in various ways later. These are the upsides to this approach. The downsides, however, are numerous - such as decoupling Config from the implementation func that you care about, and no longer being able to call func in an ergonomic way. Soon, your code base is littered with this pattern instead of having simple functions.

This pattern can be seen throughout nerfstudio and many many ML code bases.

Far preferable would be to leave func as a normal function and then generate the Config type dynamically, based off of its signature.

def func(x: int, y: str):
    ...

Config = make_dataclass_from_func_sig(func)

then it is simple enough to write a wrapper that lets you do call_via_config(foo)(Config(x=1, y="b")) to call the function using your config instance. Now, the simple function foo is the star player in your code base once again and your desire to create a type that configure it can be done automatically and as an after thought.

Indeed this is the pattern that hydra-zen enables. But a major downside to this dynamic approach is the lack of type-checking/autocomplete support for the fields on the generated type.

I have to imagine that libraries like pydantic and other heavy users of PEP 681 would have use for this feature as well.

Limitations

In this proposal, P is only valid in this context if there are no *args, **kw, or positional-only entries in the function's signature - these are not supported by dataclass types.

dataclass features not supported:

• declaring InitVar entries in the signature
• declaring no-init fields

[erictraut]

I understand the problem you're trying to solve, but this solution won't work.

First, a solution of this sort would need to go through a spec'ing and standardization process. It's not something that I'd implement unilaterally in pyright without it becoming part of the type system.

Second, there are many problems with the proposed solution.

• Inferring the existence of attributes from a ParamSpec would be completely inappropriate in the general case and would lead to many false positives for existing code that uses ParamSpec in protocols.
• While it's possible to deterministically translate a list of dataclass field definitions into a synthesized __init__ method, the reverse is not possible. There are many details that are lossy.
• Even if that were't a problem, it's not clear what would happen if the signature captured by the ParamSpec has positional-only parameters, *args, or **kwargs.

If you would like to discuss this problem and solicit ideas for solutions, I recommend opening an issue in the python/typing discussion forum or the python.org typing discourse forum.

[rsokl]

Thanks for the feedback Eric! I appreciate you taking the time.

I agree that this would likely need to be specified by a PEP. I will go ahead and open a discussion in one of the forums that you linked.

[BrandonNav]

On the point of being able to extract a function signature and manipulate it, I have a similar need with wanting to extra a function signature into a TypedDict: python/typing#1501