Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyLance not recognizing imports from PEP-660 editable installs #3473

Closed
rzats opened this issue Oct 13, 2022 · 27 comments
Closed

PyLance not recognizing imports from PEP-660 editable installs #3473

rzats opened this issue Oct 13, 2022 · 27 comments
Assignees
Labels
by design pep 660 Issues related to PEP 660 import hooks

Comments

@rzats
Copy link

rzats commented Oct 13, 2022

Environment data

  • Language Server version: 2022.10.20
  • OS and version: darwin x64
  • Python version (and distribution if applicable, e.g. Anaconda): 3.8.15
  • python.analysis.indexing: false
  • python.analysis.typeCheckingMode: off

Code Snippet

I'm currently trying to get static analysis working for a project with a nonstandard directory structure (more at #3454). Turning it into an editable install seem like the best option so far.

https://github.com/rzats/setuptools-repro is a small repo that mirrors the structure of that project. It contains this pyproject.toml:

[build-system]
requires = ["setuptools>=65", "wheel"]
build-backend = "setuptools.build_meta"

setup.cfg:

[metadata]
name = Local Package

[options]
packages =
    root.package

package_dir =
    root.package = root/package/src

as well as a Python file at root/package/src/test.py, and a main.py that tries to import test.py.

Repro Steps

git clone https://github.com/rzats/setuptools-repro.git
cd setuptools-repro
python -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
.venv/bin/python -c 'import root.package.test; print(root.package.test)'
<module 'root.package.test' from '/Users/work/Documents/setuptools-repro/root/package/src/test.py'>

Expected behavior

root.package.test should show up as a valid import in VSCode too.

Actual behavior

That doesn't happen:

screenshot of source code inside main.py that reads "import root.package.test as test" overlaid by an error message that reads "Import "root.package.test" could not be resolved (reportMissingImports)"

Unfortunately, reverting to legacy editable packages as suggested by #3265 (comment) does not work for me, since that has an unrelated issue where advanced package_dir bindings don't work at all - the newer PEP-660 implementation does support that.

Logs

Python
LSP Notebooks experiment is enabled
LSP Notebooks interactive window support is enabled
Python interpreter path: ./.venv/bin/python
> conda info --json
> ./.venv/bin/python ~/.vscode/extensions/ms-python.python-2022.16.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.16.0/pythonFiles/interpreterInfo.py
> /usr/bin/python3 ~/.vscode/extensions/ms-python.python-2022.16.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.16.0/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3.8 ~/.vscode/extensions/ms-python.python-2022.16.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.16.0/pythonFiles/interpreterInfo.py
> . ./.venv/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode/extensions/ms-python.python-2022.16.0/pythonFiles/printEnvVariables.py
Starting Pylance language server.
Python Language Server
[Info  - 6:15:09 PM] (17998) Pylance language server 2022.10.20 (pyright e2d13ab1) starting
[Info  - 6:15:09 PM] (17998) Server root directory: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist
[Info  - 6:15:09 PM] (17998) Starting service instance "setuptools-repro"
[Info  - 6:15:09 PM] (17998) Notebook support: LSP
[Info  - 6:15:09 PM] (17998) Interactive window support: LSP
[Info  - 6:15:09 PM] (17998) No configuration file found.
[Info  - 6:15:09 PM] (17998) pyproject.toml file found at /Users/work/Documents/setuptools-repro.
[Info  - 6:15:09 PM] (17998) Setting pythonPath for service "setuptools-repro": "/Users/work/Documents/setuptools-repro/.venv/bin/python"
[Info  - 6:15:09 PM] (17998) Loading pyproject.toml file at /Users/work/Documents/setuptools-repro/pyproject.toml
[Error - 6:15:09 PM] (17998) Pyproject file "/Users/work/Documents/setuptools-repro/pyproject.toml" is missing "[tool.pyright]" section.
[Warn  - 6:15:09 PM] (17998) stubPath /Users/work/Documents/setuptools-repro/typings is not a valid directory.
[Info  - 6:15:09 PM] (17998) Assuming Python version 3.8
[Info  - 6:15:09 PM] (17998) Assuming Python platform Darwin
[Info  - 6:15:09 PM] (17998) Search paths for /Users/work/Documents/setuptools-repro
[Info  - 6:15:09 PM] (17998)   /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib
[Info  - 6:15:09 PM] (17998)   /Users/work/Documents/setuptools-repro
[Info  - 6:15:09 PM] (17998)   /Users/work/Documents/setuptools-repro/typings
[Info  - 6:15:09 PM] (17998)   /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stubs/...
[Info  - 6:15:09 PM] (17998)   /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/bundled/stubs
[Info  - 6:15:09 PM] (17998)   /usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8
[Info  - 6:15:09 PM] (17998)   /usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload
[Info  - 6:15:09 PM] (17998)   /Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages
[Info  - 6:15:09 PM] (17998) Adding fs watcher for library directories:
 /usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8
/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload
/Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages
[Info  - 6:15:09 PM] (17998) Adding fs watcher for directories:
 /Users/work/Documents/setuptools-repro
[Info  - 6:15:09 PM] (17998) Searching for source files
[Info  - 6:15:09 PM] (17998) Found 2 source files
(17998) [IDX(FG)] index libraries /Users/work/Documents/setuptools-repro (index) ...
(17998) [IDX(FG)]   read stdlib indices (27ms)
(17998) [IDX(FG)] index libraries /Users/work/Documents/setuptools-repro (index) [succeed] (29ms)
(17998) [FG] parsing: /Users/work/Documents/setuptools-repro/main.py (13ms)
[Info  - 6:15:09 PM] (17998) Could not import 'root.package.test' in file '/Users/work/Documents/setuptools-repro/main.py'
[Info  - 6:15:09 PM] (17998)   Looking in stubPath '/Users/work/Documents/setuptools-repro/typings'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/Documents/setuptools-repro/typings'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve using root path '/Users/work/Documents/setuptools-repro/typings'
[Info  - 6:15:09 PM] (17998)   Looking in root directory of execution environment '/Users/work/Documents/setuptools-repro'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/Documents/setuptools-repro'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve using root path '/Users/work/Documents/setuptools-repro'
[Info  - 6:15:09 PM] (17998)   Looking in python search path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve stub package using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
[Info  - 6:15:09 PM] (17998)   Looking in python search path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve stub package using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload'
[Info  - 6:15:09 PM] (17998)   Looking in python search path '/Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve using root path '/Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages'
[Info  - 6:15:09 PM] (17998)   Looking in bundled stubs path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/bundled/stubs'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/bundled/stubs'
[Info  - 6:15:09 PM] (17998)   Attempting to resolve using root path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/bundled/stubs'
[Info  - 6:15:09 PM] (17998)   Looking for typeshed stdlib path
[Info  - 6:15:09 PM] (17998)   Looking for typeshed stdlib path
[Info  - 6:15:09 PM] (17998)   Attempting to resolve using root path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib'
[Info  - 6:15:09 PM] (17998)   Typeshed path not found
[Info  - 6:15:09 PM] (17998)   Looking for typeshed third-party path
[Info  - 6:15:09 PM] (17998)   Looking for typeshed stubs path
[Info  - 6:15:09 PM] (17998)   Typeshed path not found
(17998) [FG] parsing: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 3ms] (70ms)
(17998) [FG] binding: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/builtins.pyi (24ms)
(17998) [FG] binding: /Users/work/Documents/setuptools-repro/main.py (2ms)
[Info  - 6:15:10 PM] (17998) Background analysis(1) root directory: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist
[Info  - 6:15:10 PM] (17998) Background analysis(1) started
(17998) Background analysis message: setConfigOptions
(17998) Background analysis message: setImportResolver
(17998) Background analysis message: ensurePartialStubPackages
(17998) Background analysis message: setTrackedFiles
(17998) Background analysis message: markAllFilesDirty
(17998) Background analysis message: setFileOpened
(17998) Background analysis message: getDiagnosticsForRange
(17998) Background analysis message: getSemanticTokens full
(17998) [BG(1)] getSemanticTokens full at /Users/work/Documents/setuptools-repro/main.py ...
(17998) [BG(1)]   parsing: /Users/work/Documents/setuptools-repro/main.py (12ms)
[Info  - 6:15:10 PM] (17998) Could not import 'root.package.test' in file '/Users/work/Documents/setuptools-repro/main.py'
[Info  - 6:15:10 PM] (17998)   Looking in stubPath '/Users/work/Documents/setuptools-repro/typings'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/Documents/setuptools-repro/typings'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve using root path '/Users/work/Documents/setuptools-repro/typings'
[Info  - 6:15:10 PM] (17998)   Looking in root directory of execution environment '/Users/work/Documents/setuptools-repro'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/Documents/setuptools-repro'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve using root path '/Users/work/Documents/setuptools-repro'
[Info  - 6:15:10 PM] (17998)   Looking in python search path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve stub package using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
[Info  - 6:15:10 PM] (17998)   Looking in python search path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve stub package using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve using root path '/usr/local/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload'
[Info  - 6:15:10 PM] (17998)   Looking in python search path '/Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve using root path '/Users/work/Documents/setuptools-repro/.venv/lib/python3.8/site-packages'
[Info  - 6:15:10 PM] (17998)   Looking in bundled stubs path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/bundled/stubs'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve stub package using root path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/bundled/stubs'
[Info  - 6:15:10 PM] (17998)   Attempting to resolve using root path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/bundled/stubs'
[Info  - 6:15:10 PM] (17998)   Looking for typeshed stdlib path
[Info  - 6:15:10 PM] (17998)   Looking for typeshed stdlib path
[Info  - 6:15:10 PM] (17998)   Attempting to resolve using root path '/Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib'
[Info  - 6:15:10 PM] (17998)   Typeshed path not found
[Info  - 6:15:10 PM] (17998)   Looking for typeshed third-party path
[Info  - 6:15:10 PM] (17998)   Looking for typeshed stubs path
[Info  - 6:15:10 PM] (17998)   Typeshed path not found
(17998) [BG(1)]   parsing: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 2ms] (79ms)
(17998) [BG(1)]   binding: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/builtins.pyi (29ms)
(17998) [BG(1)]   binding: /Users/work/Documents/setuptools-repro/main.py (2ms)
(17998) [BG(1)] getSemanticTokens full at /Users/work/Documents/setuptools-repro/main.py (126ms)
(17998) Background analysis message: getDiagnosticsForRange
(17998) Background analysis message: getSemanticTokens range
(17998) [BG(1)] getSemanticTokens range 0:0 - 0:32 at /Users/work/Documents/setuptools-repro/main.py (1ms)
(17998) Background analysis message: analyze
(17998) [BG(1)] analyzing: /Users/work/Documents/setuptools-repro/main.py ...
(17998) [BG(1)]   checking: /Users/work/Documents/setuptools-repro/main.py ...
(17998) [BG(1)]     parsing: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/typing.pyi [fs read 0ms] (19ms)
(17998) [BG(1)]     binding: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/typing.pyi (5ms)
(17998) [BG(1)]     parsing: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/typing_extensions.pyi [fs read 1ms] (3ms)
(17998) [BG(1)]     binding: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/typing_extensions.pyi (2ms)
(17998) [BG(1)]     parsing: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi [fs read 0ms] (7ms)
(17998) [BG(1)]     binding: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi (2ms)
(17998) [BG(1)]     parsing: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/abc.pyi [fs read 0ms] (1ms)
(17998) [BG(1)]     binding: /Users/work/.vscode/extensions/ms-python.vscode-pylance-2022.10.20/dist/typeshed-fallback/stdlib/abc.pyi (1ms)
(17998) [BG(1)]   checking: /Users/work/Documents/setuptools-repro/main.py (73ms)
(17998) [BG(1)] analyzing: /Users/work/Documents/setuptools-repro/main.py (73ms)
(17998) Background analysis message: resumeAnalysis
(17998) Background analysis message: getDiagnosticsForRange
(17998) Background analysis message: getDiagnosticsForRange
(17998) Background analysis message: getDiagnosticsForRange
(17998) Background analysis message: getDiagnosticsForRange
@debonte debonte self-assigned this Oct 13, 2022
@judej judej added the needs investigation Could be an issue - needs investigation label Oct 13, 2022
@debonte debonte added the pep 660 Issues related to PEP 660 import hooks label Oct 13, 2022
@debonte
Copy link
Contributor

debonte commented Oct 13, 2022

reverting to legacy editable packages as suggested by #3265 (comment) does not work for me

Can you try this instead of SETUPTOOLS_ENABLE_FEATURES="legacy-editable"?:

pip install -e . --config-settings editable_mode=compat

Also, for context, see our docs on PEP 660 editable installs.

@debonte debonte added waiting for user response Requires more information from user and removed needs investigation Could be an issue - needs investigation labels Oct 13, 2022
@rzats
Copy link
Author

rzats commented Oct 13, 2022

Perfect, editable_mode=compat editable_mode=strict resolved the issue! Feel free to close.

@debonte
Copy link
Contributor

debonte commented Oct 13, 2022

@rzats, that's good to know, because setuptools is planning to remove compat mode in the near future and I was under the impression until today that it was equivalent to SETUPTOOLS_ENABLE_FEATURES="legacy-editable".

I'll open an issue with setuptools asking them to keep compat mode around longer, ideally permanently. They indicated that they would be willing to do that if there was a strong argument for it.

I'll keep this issue open until I file the issue with setuptools and then provide a link here.

@rzats
Copy link
Author

rzats commented Oct 13, 2022

fwiw, we ended up selecting a slightly different mode, pip install -e . --config-settings editable_mode=strict - this seems to be an implementation exclusive to the setuptools backend that creates symlinks to the working directory in the build subfolder.

@debonte
Copy link
Contributor

debonte commented Oct 13, 2022

Ok, but to be clear, both compat mode and strict mode solve your problem?

@rzats
Copy link
Author

rzats commented Oct 13, 2022

Just strict mode - sorry, that's my oversight. I assumed it was identical to compat mode, but in fact it's a separate option that falls back to the non-PEP-660 implementation; and this runs into package_dir issue I've mentioned.

@jborean93
Copy link

Can I ask why this issue was closed, not supporting the default PEP-660 editable install mode is going to confuse a lot of users who can't find this page. Is there a fundamental reason why pylance can't use these libraries when installed in the default mode.

@debonte
Copy link
Contributor

debonte commented Jul 3, 2023

Can I ask why this issue was closed, not supporting the default PEP-660 editable install mode is going to confuse a lot of users who can't find this page. Is there a fundamental reason why pylance can't use these libraries when installed in the default mode.

Unfortunately PEP 660 was approved without discussion with the Python typing community. We cannot run import hooks. But there has been discussion about alternative ways to solve this problem. See this typing-sig thread. If a solution is found, agreed upon by typing-sig, and standardized, Pylance will consider supporting it.

There's more about this in our docs on PEP 660 editable installs.

@jborean93
Copy link

Thanks for the further info, hopefully a decent resolution comes out of it as trying to remember this issue everytime I do an editable install of a package has bitten me a few times.

@qci-amos
Copy link

It took me a fair amount of searching after a lot of mucking with setting interpreters to find this issue and the --config-settings editable_mode=strict solution... I've been a python developer for many years and I never "knew" there were different flavors of editable installs and wouldn't have guessed that pylance would care about which one I use! Nor would I have guessed that there would be persistent sticking issues on a 2.5 year old pep! It's very confusing! For years now, I've just accepted the "orange squiggles" on my python files as my fault.

I wonder if Pylance could add some text or a link to the reportMissingImports popup?

I just want to add that this solution is not currently sufficient for me. If I do "Go to Definition F12", it takes me to the /build/__editable__. folder, not to the directory in my vscode workspace. I assume this is very similar to the linked issue here, but in this case, I'm just using vscode (not a debugger). I don't want to mess with pathMappings, partly because I don't want to hand-edit that on principle, but also because I have many editable repos.

I found that --config-settings editable_mode=compat seems to solve both problems (pylance and f12).

I'm using

WSL
conda 23.7.4
py3.8.18
setuptools                  68.2.2
pip                         23.2.1

@debonte
Copy link
Contributor

debonte commented Sep 19, 2023

I wonder if Pylance could add some text or a link to the reportMissingImports popup?

@qci-amos, can you open a separate issue on this?

I just want to add that this solution is not currently sufficient for me...I found that --config-settings editable_mode=compat seems to solve both problems (pylance and f12).

There have been several similar issues filed and it doesn't surprise me that the user's chosen solution might differ in each. Our primary reference on PEP 660 import resolution is this doc page, which mentions both compat and strict.

@ThiefMaster
Copy link

I don't understand why PyLance cannot use a simple logic like this to handle PEP660-style editable install

import importlib
import re
import sys
from itertools import chain
from pathlib import Path


def get_editable_package_mappings():
    # XXX not using `ast` to parse the import from the pth file since it's way easier with a simple regex
    mappings = [
        importlib.import_module(m.group(1)).MAPPING
        for f in chain.from_iterable(
            Path(p).glob('__editable__.*.pth') for p in sys.path
        )
        if (m := re.search(r'import (__editable___\S+_finder)', f.read_text()))
    ]
    return mappings

This works perfectly fine to map package names to their source locations. Sure, it makes certain assumptions about the structure of those files, but since it's probably unlikely that those files will significantly chance any time soon (even more so if popular tools like PyLance use them), but it does the job.

In the typing mailing list discussion linked by @debonte a year ago, there's a mention of not being able to use anything but certain stdlib modules. My example above uses some more stdlib modules, but it would be fairly trivial to adapt the code and e.g. avoid importing the module but instead uses regex (and/or the ast module) to extract the mapping dict.

bpkroth added a commit to microsoft/MLOS that referenced this issue Jul 10, 2024
1. Temporarily avoid recent asyncssh version that breaks the tests
2. Workaround pylance issue with pyproject.toml related changes and pip
editable modules install format (#768)
   See Also: 
   - microsoft/pylance-release#3473
   May also affect `mypy`:
   - python/mypy#16988
   - python/mypy#12313
bpkroth added a commit to bpkroth/MLOS that referenced this issue Jul 10, 2024
1. Temporarily avoid recent asyncssh version that breaks the tests
2. Workaround pylance issue with pyproject.toml related changes and pip
editable modules install format (microsoft#768)
   See Also:
   - microsoft/pylance-release#3473
   May also affect `mypy`:
   - python/mypy#16988
   - python/mypy#12313
DelphianCalamity pushed a commit to DelphianCalamity/MLOS that referenced this issue Jul 12, 2024
1. Temporarily avoid recent asyncssh version that breaks the tests
2. Workaround pylance issue with pyproject.toml related changes and pip
editable modules install format (microsoft#768)
   See Also: 
   - microsoft/pylance-release#3473
   May also affect `mypy`:
   - python/mypy#16988
   - python/mypy#12313
@link89
Copy link

link89 commented Jul 15, 2024

pip install -e . --config-settings editable_mode=strict not works for me.

image

@debonte
Copy link
Contributor

debonte commented Jul 15, 2024

pip install -e . --config-settings editable_mode=strict not works for me.

Does compat mode work for you? See https://microsoft.github.io/pyright/#/import-resolution?id=editable-installs for links to the pip/setuptools documentation.

If neither compat mode nor strict mode works, please file a new issue and we can investigate. Please include links to a repo that reproduces the issue you are seeing.

@link89
Copy link

link89 commented Jul 16, 2024

@debonte compat works!

@stellaraccident
Copy link

This still appears to be broken as of September 2024.

@ThiefMaster
Copy link

TBH I think the easiest solution, if it's your own package, is to just drop setuptools in favor of hatchling. AFAICT it only has advantages :)

@stellaraccident
Copy link

TBH I think the easiest solution, if it's your own package, is to just drop setuptools in favor of hatchling. AFAICT it only has advantages :)

Lots of things on the internet presume that tools work as advertised. I don't just need projects I control to work.

@stellaraccident
Copy link

stellaraccident commented Sep 13, 2024

The most reliable thing is just to stop expecting that any of it works and use a .env file.

@debonte
Copy link
Contributor

debonte commented Sep 13, 2024

We currently have no plans to make changes here. See my comment above for an explanation of how we got where we are today. If the packaging and typing communities can come to an agreement on a way that static analyzers (emphasis on static) can work better here, we would likely adopt that approach.

Also see Eric Traut's recent comment here: #6384 (reply in thread)

@stellaraccident
Copy link

TBH I think the easiest solution, if it's your own package, is to just drop setuptools in favor of hatchling. AFAICT it only has advantages :)

For anyone else who would like to skip this research. From hatchling's readme: "If building extension modules is required then it is recommended that you continue using setuptools, or even other backends that specialize in interfacing with compilers."

@stellaraccident
Copy link

I ended up just writing a script that I can run in any repo to introspect the python environment and produce a .env file that gets the job done.

Such a thing could be better if the IDE had some config options that let me set a list of package_prefix->path mappings for a project vs the single flat list of paths.

This all seems like options that the static analyzer should support and then that the ide plug-in should be pragmatic about attempting to auto detect during environment setup.

Sure, I would have done the packaging specification differently, but I have little sympathy for the argument here that has produced a user hostile outcome for years from an IDE.

@stellaraccident
Copy link

I don't understand why PyLance cannot use a simple logic like this to handle PEP660-style editable install

import importlib
import re
import sys
from itertools import chain
from pathlib import Path


def get_editable_package_mappings():
    # XXX not using `ast` to parse the import from the pth file since it's way easier with a simple regex
    mappings = [
        importlib.import_module(m.group(1)).MAPPING
        for f in chain.from_iterable(
            Path(p).glob('__editable__.*.pth') for p in sys.path
        )
        if (m := re.search(r'import (__editable___\S+_finder)', f.read_text()))
    ]
    return mappings

This works perfectly fine to map package names to their source locations. Sure, it makes certain assumptions about the structure of those files, but since it's probably unlikely that those files will significantly chance any time soon (even more so if popular tools like PyLance use them), but it does the job.

In the typing mailing list discussion linked by @debonte a year ago, there's a mention of not being able to use anything but certain stdlib modules. My example above uses some more stdlib modules, but it would be fairly trivial to adapt the code and e.g. avoid importing the module but instead uses regex (and/or the ast module) to extract the mapping dict.

I realize that this style of solution can offend certain sensibilities, but if it were turned into a real workaround and recommended in a faq, it would have saved many people countless hours.

If there were a vscode setting that just took a json mapping of package prefix/path mappings, a small script like this could generate that. Then copy/paste, done.

Then maybe a couple of years from now the conversation resolves some declarative way to... and everyone is happy. I know this packaging stuff is a mess. Better workarounds and do-it-myself config options are the soup-du-joure.

FTR, there are significant usability hurdles to the strict mode recommendation -- so much so that I have spent a lot of time living with the orange squiggles as the better option. I've spent many hours trying to use it, and it is just better to slog on without ide integration.

@heejaechang
Copy link
Contributor

other easy workarounds are either just setting PYTHONPATH env using .env file or using extraPaths settings that points to where source of editable install is. that would make them discoverable inside of pylance.

@stellaraccident
Copy link

Yeah, I think I've given up on this working in any sane way by default and am just sticking to .env files. Would be really nice if the IDE itself, as part of python environment selection could advertise a list of package/paths to plugins.

@luckylinux
Copy link

@stellaraccident: may I know how you managed to solve this ?

I have a Project Namespace in a company.private.package.subpackage Format.

In my case, either I use:
a. Native NameSpace Packages so only one __init__.py in the Lowest Subfolder Level -> sphinx /pydoctor get the Namespace Wrong (subpackage only), PyLance works correctly and detects the other Packages/Modules
b. PKGUtil Namespace Packages -> sphinx / pydoctor get the Namespace Right (company.private.package.subpackage) but PyLance refuses to find other Packages/Modules located OUTSIDE of the current Workspace (does NOT matter if I set python.analysis.extraPaths to include basically everything I could think of)

So, for (b) as long as other Folders (besides last subpackage Subfolder) "down the chain" features an __init__.py File, PyLance will refuse to look elsewhere.

There seem to be other Issues related to "Editable Installs" (I usually just run poetry install for the other Packages).

I also tried these two but nothing changes:

  1. pip install -e . --config-settings editable_mode=strict
  2. pip install -e . --config-settings editable_mode=compat

Is there something that could make both work correctly ?

Is seems to me like there is a "Package" merging that is not working correctly (at all).

@luckylinux
Copy link

The other dirty workaround is to write a helper Script / hook to:

  1. When generating Documentation using sphinx / pydoctor (and maybe running Tests): create __init__.py in all parent Folders (subpackage and upwards until src - excluding this last one)
  2. After that has been done and we are back to code, delete all of these extra __init__.py so the Project essentially "looks" like a PKGUtil Namespace Package

Any other Ideas ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
by design pep 660 Issues related to PEP 660 import hooks
Projects
None yet
Development

No branches or pull requests

10 participants