Pick up Type Annotations From .pyi Stub Files (PEP-561)

[ravenexp]

Problem Description

I'm building a native code Python extension module in Rust with PyO3.
It can attach docstrings and function signatures to the module items,
but AFAIK currently there is no way to add Python type information to the extension modules.

PEP-561 was created to help to solve this issue.
It defines type information "stubs" (.pyi files), which only carry type hints and signatures,
but not the implementation or docstrings.

I've created a PEP-561 compliant stubs package foo-stubs for my extension module foo,
which contains the module type hints in its only __init__.pyi source file.
This scheme was needed because PEP-561 does not support distributing type information
as a part of module-only distributions.

I have also verified that both mypy and PyCharm are able to extract and use type information
distributed in this manner.

Proposal

pdoc is currently able to extract inline type hints from conventional .py sources.
It would be great if pdoc was also able to pull the type information from .pyi
stubs that are distributed alongside with the Python package sources or
as a separate "-stubs" package.

Alternatives

The native code extension modules may gain the ability to encode the required type information
at some point in the future, making .pyi type stubs obsolete.

Additional context

PyCharm is able to combine the docstrings and function signatures extracted from the extension module
via introspection with the type hints extracted from the separately distributed .pyi files.

[mhils]

My initial reaction reading this was that this should just be solved by extension modules supporting __annotations__, but reading through the PyO3 and Python issue trackers this seems to be a non-trivial undertaking and may not happen anytime soon. I if we fully want to support PyO3 with type hints short-term, PEP-561 support proabably needs to be added to pdoc.

Some observations:

• I naively hoped that pyi files would at least be importable, but that's not the case. pathlib, inspect and re from https://github.com/python/typeshed all bork on import. The only viable way seems to be to ast.parse them, which is going to be relatively painful.
• This module has some example code showing how to detect pyi files: https://github.com/ethanhs/check_typedpkg

I think a reasonable way to implement this would be to override Namespace.members in Module.members and add a doc_ast.patch_in_typehints_from_stubs(self, members) call. This to-be-implemented method then takes the pyi file, parses it, and recursively applies type annotations to all Doc objects in the module. pdoc makes heavy use of cached_property, which makes patching attributes straightforward - they just need to be reassigned. Walking the AST will be messy by nature, but not overly complicated.

PRs are welcome, I'm happy to help if you get stuck somewhere! 😃

[ravenexp]

I'll have a look at it, thought it would probably take a while...

I don't actually program in Python, except for writing some trivial unit tests, so my Python language knowledge is superficial at best.

[mhils]

An update on my previous comment: It seems like .pyi files are importable now for the most part, which makes this undertaking a bit more reasonable. :)

[mhils]

Update: pdoc now picks up .pyi files and integrates the type info. 🎉
Combined with PyO3/maturin#886 it now works quite well to document PyO3 projects. 😃 You author the docstring in Rust, add the .pyi file with type annotations, and pdoc picks up both:

example.pyi

async def start_server(
    host: str,
    port: int,
    private_key: str,
    peer_public_keys: list[str],
    handle_connection: Callable[[asyncio.StreamReader, asyncio.StreamWriter], Awaitable[None]],
    receive_datagram: Callable[[bytes, tuple[str, int], tuple[str, int]], None],
) -> WireguardServer: ...

lib.rs

/// Start a WireGuard server.
#[pyfunction]
fn start_server(
    py: Python<'_>,
    host: String,
    port: u16,
    private_key: String,
    peer_public_keys: Vec<String>,
    handle_connection: PyObject,
    receive_datagram: PyObject,
) -> PyResult<&PyAny> {
    /* ... */
}

⬇️

output.html

---

One very nice aspect about this is that you don't need to add text_signature in PyO3, which was rather tedious. :)

[ravenexp]

Thanks!