Support finding "editable" type annotated packages made editable through direct_url.json

[hashstat]

Feature

Add the ability for mypy to discover the path of "editable" packages installed with direct_url.json files, as described in PEP-610.

Pitch

As the Python ecosystem continues to move toward PEP-517-compliant build systems, the way editable packages are installed is also changing. PEP-660 uses the editables package as an example of how to install editable wheels using import hooks that act as a proxy for editable packages. Editables uses .pth files to load a module finder to proxy the loading of modules from outside of the packages directory without modifying the Python path. Because the path is left unmodified, the methods described in PEP-561, and currently employed by mypy, fail to find these packages. But pip leaves behind a hint about the package path in the direct_url.json file it installs in the .dist-info directory when the -e/--editable flag is used.

The hatchling build backend uses editables to install editable packages. The problem is that mypy cannot determine that such packages are type annotated because of the limitations imposed by its current PEP-561-based discovery methods. Packages that are not stubs cannot be discovered because they aren't named appropriately. And py.typed files cannot be found because the project is proxied by a single python module and is missed by the current mypy search algorithm because the package files are not found on the python path.

Here is an example editable installation that might be installed using pip install -e path/to/package:

venv/lib/python3.9/site-packages/editable_pkg_typed.pth

import _editable_pkg_typed

venv/lib/python3.9/site-packages/_editable_pkg_typed.py

from editables.redirector import RedirectingFinder as F
F.install()
F.map_module('editable_pkg_typed', '/path/to/package/editable_package_typed/__init__.py')

venv/lib/python3.9/site-packages/editable_pkg_typed-0.dist-info/

INSTALLER
LICENSE
METADATA
RECORD
REQUESTED
WHEEL

venv/lib/python3.9/site-packages/editable_pkg_typed-0.dist-info/direct_url.json

{
    "url": "file:///path/to/package",
    "dir_info": {
        "editable": true
    }
}

/path/to/package/

LICENSE
README.md

/path/to/package/pyproject.toml

[build-system]
requires = ['hatchling']
build-backend = 'hatchling.build'

[project]
name = 'editable_pkg_typed'
version = '0'

/path/to/package/editable_pkg_typed/

__init__.py
py.typed

Adding the path from direct_url.json to those discovered in .pth files should be sufficient for mypy to find these editable packages. I will submit a pull request soon with the suggested functionality.

[abravalheri]

Adding the path from direct_url.json to those discovered in .pth files should be sufficient for mypy to find these editable packages. I will submit a pull request soon with the suggested functionality.

Hi @hashstat, although this approach might work for some packages, it may not generalise well...

For example, it is a wide spread practice in the community to organise a project in what is called src-layout (as shown in the PyPA tutorial). This means that between the directory shown in direct_url.json and the actual package directory, there might be a src folder, e.g.:

/path/to/project/
├── LICENSE
├── README.md
├── pyproject.toml
└── src
    └── package
        ├── __init__.py
        └── py.typed

(Of course there might be all sorts of creative ways of organizing a project layout, but I think that at least it is important to consider the src-layout, since it has a non-negligible adoption).

[hashstat]

it is a wide spread practice in the community to organise a project in what is called src-layout

Hmm. @abravalheri makes a good point.

We could add a search for modules/packages in the src directory. That addition would likely handle 90% of the cases out there. But now we are making assumptions about the layout of project source code that may or may not be correct.

Unfortunately, there doesn't appear to be a simple and reliable method to determine the path of editable packages installed using import hooks. Looks like we might be stuck using MYPYPATH, unless someone has another idea?