Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reportMissingModuleSource warnings when importing submodules from extension modules (pybind11) #5950

Closed
tonywu6 opened this issue Sep 15, 2023 · 8 comments
Assignees
Labels
as designed Not a bug, working as intended bug Something isn't working

Comments

@tonywu6
Copy link

tonywu6 commented Sep 15, 2023

Describe the bug

I have a Python package with a C extension module (built with pybind11) that also declares a submodule. I also have the corresponding type stubs (generated with mypy). Assuming the following layout after installed:

example.so
example/
  __init__.pyi
  submodule.pyi
  py.typed

I attempt to import from the submodule, and then type-check with Pyright, assuming default configuration, and that the stubs are correct:

from example.submodule import func

Expected result

Pyright reports no warnings. (MyPy doesn't)

Actual result

Pyright warns about Import "example.submodule" could not be resolved from source (but also finds the type stubs correctly).

Screenshot 2023-09-15 at 4 47 15 PM
Verbose output
$ npm exec pyright main.py
Search paths for ...
  .../node_modules/.pnpm/[email protected]/node_modules/pyright/dist/typeshed-fallback/stdlib
  ...
  .../typings
  .../node_modules/.pnpm/[email protected]/node_modules/pyright/dist/typeshed-fallback/stubs/...
  .../.venv/lib/python3.8
  .../.venv/lib/python3.8/lib-dynload
  .../.venv/lib/python3.8/site-packages
Found 1 source file
Could not resolve source for 'example.submodule' in file '.../main.py'
  Looking in root directory of execution environment '...'
  Attempting to resolve using root path '...'
  Finding python search paths
  Executing interpreter: 'python3'
  Skipping '.../.venv/lib/python38.zip' because it is not a valid directory
  Received 3 paths from interpreter
    .../.venv/lib/python3.8
    .../.venv/lib/python3.8/lib-dynload
    .../.venv/lib/python3.8/site-packages
  Looking in python search path '.../.venv/lib/python3.8'
  Attempting to resolve using root path '.../.venv/lib/python3.8'
  Looking in python search path '.../.venv/lib/python3.8/lib-dynload'
  Attempting to resolve using root path '.../.venv/lib/python3.8/lib-dynload'
  Looking in python search path '.../.venv/lib/python3.8/site-packages'
  Attempting to resolve using root path '.../.venv/lib/python3.8/site-packages'
pyright 1.1.327
.../main.py
  .../main.py:2:6 - warning: Import "example.submodule" could not be resolved from source (reportMissingModuleSource)
0 errors, 1 warning, 0 informations 
Completed in 0.512sec

Reproducible example

https://github.com/tonywu6/pyright-pybind11

(Just started learning about bindings stuff so it probably looks like a mess)

Environment

  • Python 3.8.17 (conda 23.5.2)
  • Pyright 1.1.327
  • Pylance 2023.9.10 (issue happens both in VS Code and on the command line)

The extension module was built with:

  • macOS 13.5.1 (aarch64)
  • Python 3.8.17
  • pybind11 2.11.1
@tonywu6 tonywu6 added the bug Something isn't working label Sep 15, 2023
@erictraut
Copy link
Collaborator

@heejaechang , IIRC, you initially implemented the code for reportMissingModuleSource and are probably the most familiar with it. Could you take this bug?

@heejaechang
Copy link
Collaborator

@tonywu6 this is happening since we don't know example.so also declares submodule. this kind of dynamic behavior is not something we can statically figure out.

if you had submodule.so along with submodule.pyi we won't show the warning (it basically telling you, you have pyi file but we couldn't find any corresponding implementation (py or native modules))

you probably need # type: ignore for this one.

@erictraut erictraut added the as designed Not a bug, working as intended label Sep 20, 2023
@erictraut erictraut closed this as not planned Won't fix, can't repro, duplicate, stale Sep 20, 2023
@wojdyr
Copy link

wojdyr commented Dec 28, 2023

Is there a way to disable this warning in type stubs? (I'm only starting to use type checking, maybe the answer is obvious).

I have a similar configuration as OP and I'd like to package my python library together with stubs.
Adding # type: ignore in python code that imports my module works, but it would need to be added by all users of the library. Is there a way to disable the warning in my pyi files?

@erictraut
Copy link
Collaborator

@wojdyr, I'm a bit unclear on your situation. Are you running pyright on your .pyi files to type check them? A consumer of your library would not run pyright on your .pyi files, so I presume you're asking for your own usage (e.g. in your CI pipeline).

Is your library written in Python? If so, I recommend putting type information directly in your Python code rather than shipping parallel ".pyi" files. If you keep the type information separate, it's much tougher to maintain because the type information will inevitably diverge from the implementation over time. If your library (or parts of your library) are written in another language, like the OP, then ".pyi" files are the recommended approach for those submodules.

Which submodules are you importing from your type stubs? Are they part of your library? Are those submodules written in Python or some other language? If it's the latter, do you have ".pyi" files for those submodules as well?

@wojdyr
Copy link

wojdyr commented Dec 28, 2023

Thanks for quick response. To clarify, I have the same setup as OP. Python extension module is written entirely in C/C++. Python C API allows to have submodules in a single python extension file (.so/.pyd). The layout is (copying from OP):

example.so
example/
  __init__.pyi
  submodule.pyi
  py.typed

The extension (which including submodules) is in one file example.so, but pyi files are separate.

When a user of this library writes a program and imports the submodule:

import example.submodule

such a warnings is printed:

$ pyright foo.py
/path/to/foo.py:2:8 - warning: Import "example.submodule" could not be resolved from source

The user may add # type: ignore in their program, but ideally, I'd like to disable the warning inside my stubs.

@erictraut
Copy link
Collaborator

You will need to change the structure of your package to conform to PEP 561 if you want this to work with static type checkers. PEP 561 (which introduced the concept of a "py.typed" marker file) requires that the package be in a directory — in your case, the top-level directory named example. Your ".so" file would need to be located within this top-level directory, which would also require a minimal __init__.py file that imports (and re-exports) the symbols from your native library. If all of the symbols exported by your native file are part of the submodule namespace, then I recommend renaming it submodule.so so it sits alongside the submodule.pyi file.

@wojdyr
Copy link

wojdyr commented Dec 28, 2023

The authors of this PEP didn't think much about extension modules at that time. Having intermediary __init__.py has drawbacks, as discussed in python/typing#1333; for now I don't plan to use it.
The current setup works fine with mypy, and in pyright it's only a warning, so I can live with it.

@wojdyr
Copy link

wojdyr commented Dec 29, 2023

FTR, PEP 561 won't be updated to clarify the structure of extension packages, because it's already accepted, but having extension module (.so) directly in site-packages is considered correct: python/peps#2318

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
as designed Not a bug, working as intended bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants