Optionally build type information stub packages

[ravenexp]

PEP 561 allows to distribute the Python type information for
the extension modules by creating separate stub-only packages.

Mixed language projects can embed the type information stubs (.pyi)
into the package by simply adding a py.typed marker file.

maturin stubs command accomplishes this also for the module-only extensions
by automatically building and publishing an associated foo-stubs package
when the foo extension is being built and the foo.pyi type information
stubs file is found in the project directory.

maturin build and maturin publish commands build and publish
the stubs package automatically unless --no-stubs flag is passed.

[konstin]

I really want to support stub files in maturin and I appreciate the initiative, but I'm not sure whether extra stub packages are the way to go (for these cases it can also be helpful to open an issue to discuss the implementation before a pull request).

PEP 484 says:

The easiest form of stub file storage and distribution is to put them alongside Python modules in the same directory.

The mypy docs add:

Use the normal Python file name conventions for modules, e.g. csv.pyi for module csv. Use a subdirectory with __init__.pyi for packages. Note that PEP 561 stub-only packages must be installed, and may not be pointed at through the MYPYPATH (see PEP 561 support).

Would this be an option for maturin? I wouldn't mind changing the pure-rust layout for that to the following:

mymodule
├── __init__.py
├── __init__.pyi
└── mymodule.abi3.so

__init__.py would only contain from .mymodule import * and __init__.pyi gets renamed from the stub file the user provides.

CC @ijl any thoughts on this? I remember that you have a stub file for orjson but I don't know how you ship it.

[ravenexp]

That would work, but it's almost the same as using a mixed-language project layout from the start.
It doesn't require special support from maturin, because the users can just drop __init__.pyi and py.typed into the python sources directory and call it a day.

I wouldn't mind changing the pure-rust layout for that to the following:

Wouldn't implicitly converting module-only packages to directory packages cause backwards compatibility issues on the Python side?
I've only built module-only packages so far, so I'm not sure.
I don't really want to convert everything to directory packages just to ship the optional type information.

This PR simply encodes the build flow I'm currently using, except I was scraping Cargo.toml for the package metadata from the stubs package setup.py script and then pushing it with maturin upload. I then tried to hack build and publish commands to do this for me.

[konstin]

I've implemented the new layout in #558, please have a look. As far as I can tell, this means no changes for the user:

$ python -c "from pyo3_pure import DummyClass; print(DummyClass.get_42(), DummyClass.__qualname__, DummyClass.__module__, DummyClass.__class__)"
42 DummyClass builtins <class 'type'>

This is the same for both layouts, so afaik not even the string representations changed.

I'll check whether mypy likes this once #558 is merged (I'd prefer that layout either way).

[ravenexp]

#558 allows to embed the type info directly into the package, but it isn't clear yet how the user can include __init__.pyi and py.typed files for pure rust projects.

[messense]

#558 allows to embed the type info directly into the package, but it isn't clear yet how the user can include __init__.pyi and py.typed files for pure rust projects.

It should be easy to add __init__.pyi or maybe <module_name>.pyi with auto-generated py.typed file into the package if it is found relative to cargo manifest path.

[ravenexp]

<module_name>.pyi would be nice. __init__.pyi makes the project directory look somewhat like a python package, which it isn't.

[ijl]

@konstin I had matched the behavior of #558 using a mixed project, so that looks good.

[messense]

<module_name>.pyi would be nice. __init__.pyi makes the project directory look somewhat like a python package, which it isn't.

Opened #567