Does not resolve pyproject.toml (using WSL2)

[TimChild]

What happened?

Pyright runs, doesn't produce any obvious errors (is there a log somewhere that I can check for more details?) but doesn't seem to be using the pyproject.toml settings that are picked up when running pyright from wsl2 terminal directly.

I went down a whole rabbit whole thinking it was just a filename conversion issue related to wsl2, but I'm not sure now...
@InSyncWithFoo I'm hoping that you'll just be able to spot obvious to you given your recent work here where you "overcame a nightmare".

Steps to reproduce

Repository and poetry env live in WSL2, pycharm is being run from windows. Pyright is installed in poetry env.

First thing I noticed was that packages installed in the poetry env weren't being recognized, but further testing revealed that the whole pyproject.toml settings are being ignored (i.e. setting reportXXX = false in [tool.pyright] has no effect on warnings shown.

Setup and things I tried:

Also tried setting the configuration file to the pyroject.toml or a pyrightconfig.json file but whatever I selected wouldn't save (maybe that is actually a bug?). I don't think it should be necessary though since the pyproject.toml should have been detected automatically by pyright (as it was when running from wsl terminal).

In pyproject.toml

I don't think the language server part does anything with Command line mode selected (right?).

Things I also tried:

• LSP4IJ mode with the wsl -e /.... and with just /home/... and with C:\\... (and pyright installed in windows), but either it would fail starting up, or I would get an error related to path starting with // (which I think came from running pyright from cmd/powershell, and the filenames being passed as //wsl.localhost/... )
• Native LSP client mode with the different options above -- Seemed to ignore the settings, but errored because zsh:1: command not found: C:\Users\...\pyright-langserver.exe and Command line: C:\Users\...\pyright-langserver.exe --stdio\nStdIO connection closed\nExit code: 127

Tried these things before I realised the pyproject.toml was not being used at all...

• Changing venvPath -- tried starting with //wsl.localhost/Ubuntu/home/...., //wsl$/Ubuntu/home/...
• Adding extraPaths -- with path to the site-packages directly (similar path styles to above)

All in vain...

Seems like the Native LSP client mode could work given than the first command tried to run from zsh, if only I could point that to the pyright in wsl instead of the windows path, but I couldn't figure out where to configure that.

Or is there a better solution I am missing altogether?

System:
Win 11
PyCharm 2024.3 (Professional Edition)
Build #PY-243.21565.199, built on November 12, 2024
pyright 1.1.389 (both in windows and poetry env)

Sorry to bring up PyCharm/WSL nightmare stuff again!

Relevant log output or stack trace

Where can I find these?

Operating system

Windows

[gerhardadler]

I think I'm facing the same issue on a computer running native Linux (openSUSE Tumbleweed). I cannot save the configuration file. (pyproject.toml)

[InSyncWithFoo]

I'll look into this by the end of this week.

Thank you both for reporting the problems. It is really nice to know that something I maintain have such dedicated users. So nice, in fact, that the debugging nightmares don't seem too bad after all.

[TimChild]

@InSyncWithFoo -- Honestly, your plugin is the last hope Intellij has of keeping me as a user of pycharm at all. Their built in type checking/inference is hopeless.
It's frustrating that they don't make you're life as a plugin maintainer a little easier too!

[InSyncWithFoo]

Currently, only the Native LSP client supports WSL. This is the recommended mode if you are on PyCharm Professional.

The cause of the bug is a failed check for the specified executable's existence. Given a path of the form \\wsl$\Ubuntu\home\you\... (the expected format), Kotlin's Path.exists(), which uses Java's java.nio.file.Files.exists() rather than java.io.File.exists(), always returns false.

I couldn't, for the life of me, create a proper Poetry project, but I suppose that is unimportant. Please install the build manually (see installation guide here) and let me know if it works for you.

Small note: This mode (and LSP4IJ) doesn't respect the configuration file field at all (there are reasons for that), only workspaceFolders, from which the configuration file will be detected automatically.

[TimChild]

@InSyncWithFoo
Followed your instructions above... It looks as though the Pyright LSP is running (pie icon in the bottom bar), but I'm not seeing any diagnostics show up in the main window.

When opening files, I see error notifications like this:

Here are screenshots of what I think are the relevant configurations:

All overrides disabled in the project specific settings for simplicity.

In pyproject.toml

Also tried without the venvPath and venv for simplicity.

Looks like the LSP is running

Super simple test file that has several typing errors

The warning squiggly is only from pycharm, not pyright.

[InSyncWithFoo]

@TimChild What do you see in the logs? You might have to enable language server logging to make the most out of it. The entries containing com.insyncwithfoo.pyright are those of interest.

If you don't want to post your logs publicly, you can send an email to <my username>@gmail.com.

[TimChild]

@InSyncWithFoo Just sent you and email with some logs.

[InSyncWithFoo]

@TimChild It seems that the server was hanging ("No response from the server in 10000ms"). I will need everything you can send me; please set "Log level" to "Trace" so that we know what Pyright has to say as well.

I also found these issues over @microsoft/pylance-releases:

• "Received redundant open text document command" when working off mapped network drive
• Symlinks lead to "Received redundant open text document command"
• Rename symbol in a symbolic py file opens source file

This means either Pyright itself is not handling the paths correctly, or I'm sending it malformed paths.

[TimChild]

Editied this comment because I didn't realise I was relpying to a github email... Just sent an email to your gmail address with the log file at trace level from IDE startup to seeing the error notification in the IDE.

[InSyncWithFoo]

@TimChild I think I know what the culprit is. You seem to have the LSP4IJ plugin installed (it used to be a required dependency before 0.7.0, but no longer). Try disabling or uninstalling it, then restart and try again.

LSP4IJ and the native client are incompatible with each other, since they share a dependency that (due to Java's limitation or whatever) can only be loaded once. This is what causing the weird errors that would otherwise be undebuggable ("class jdk.proxy2.$Proxy160 cannot be cast to class org.eclipse.lsp4j.services.LanguageServer").

[TimChild]

@InSyncWithFoo Removing LSP4IJ seems to have fixed the error notification, but I'm still not seeing any squiggles...

I noticed that the inspections for pyright got disabled at some point (maybe when removing LSP4IJ?), but re-enabling didn't make any difference... I've emailed you a new set of logs.

Thanks a lot for all this work you're doing!

[InSyncWithFoo]

The problems are quite evident:

1. Pyright could find neither pyrightconfig.json nor pyproject.toml:

   INFO - #c.i.p.l.i.LspServerImpl - PyrightServerDescriptor@...(Running;0): window/logMessage Log: No configuration file found.
   INFO - #c.i.p.l.i.LspServerImpl - PyrightServerDescriptor@...(Running;0): window/logMessage Log: No pyproject.toml file found.
   

2. Pyright did publish diagnostics, but PyCharm could not find the files to apply those to:

   FINE - #c.i.p.l.i.c.Lsp4jServerConnector - <-- PyrightServerDescriptor@...: {"jsonrpc":"2.0","method":"textDocument/publishDiagnostics", ...}
   WARN - #c.i.p.l.i.h.LspDiagnosticsCache - Could not find file with diagnostics: file:///.../file.py
   

Regarding the first problem, this is probably because your pyproject.toml isn't placed directly inside the project root. The solution is to mark its parent directory as source root and then change "Workplace folders" to "Source roots".

As for the second, I could only guess that the WSL distribution somehow wasn't detected properly. Please try out the new build at #96, which falls back to detecting the distribution from the project's UNC path.

[TimChild]

@InSyncWithFoo -- Ah yes, I do have my pyproject.toml file in a different folder.. Using Source roots does fix finding the pyproject.toml file.

And yes, I see now that the logs show that pyright is doing the right thing.

Unfortunately, I'm still seeing the error for

2024-12-02 17:25:56,669 [  78794]   WARN - #c.i.p.l.i.h.LspDiagnosticsCache - Could not find file with diagnostics: file:///home/tim/.../file.py

That's after uninstalling the plugin and installing the one at #96

I assume that part works find for you using WSL2?
Is the problem that I am running pycharm from windows? I don't know if it's possible to install it in wsl2 directly...

Everything about the path looks fine to me assuming that the file:/// part should somehow be recognized as \\wsl$\Ubuntu.
I don't understand what service is responsible for interpreting that part, but seems like that is the final hurdle here...

I don't see anything in the log lines following that to suggest that any other filepaths are being looked for btw (if that's related to what you just implemented), it just goes onto the next pair of <-- PyrightServerDescriptor and Could not find file... pairs.

[TimChild]

Ok, just read into file URIs a little.. I see that file:///home... just means /home... with no hostname.

So I guess from there, pycharm needs to somehow know that the path refers to a file within wsl2... I don't know kotlin, so struggling to read through your changes...

I just noticed you added some logging though, so here is what I see for that in the logs:

2024-12-02 17:36:51,637 [ 733762]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - Executable: \\wsl$\Ubuntu\home\tim\.cache\pypoetry\virtualenvs\athena-vic8bzTz-py3.12\bin\pyright-langserver
2024-12-02 17:36:51,637 [ 733762]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - WSL distro: Ubuntu

[InSyncWithFoo]

I'm also running PyCharm on the Windows side. This part is responsible for converting a LSP file URI to a Windows path, which PyCharm can use to search for the file in its virtual file system. getFileUri does the same thing but the other way around.

This conversion is necessary because both the project and Pyright itself live in Linux realm. The file URI represents a Linux path; it doesn't know anything about Windows. To convert that to an UNC path, it is necessary to know the distribution.

private fun findFileByUri(fileUri: URI): VirtualFile? {
    val virtualFileUri = when {
        wslDistribution == null || fileUri.pathIsAbsoluteDos -> fileUri
        else -> Path.of(wslDistribution!!.getWindowsPath(fileUri.path)).toUri()
    }
    
    return super.findFileByUri(virtualFileUri.toString())
}

wslDistribution is certainly not null (as have been confirmed by "WSL distro:" log entries), and the path is not an absolute DOS path either (it doesn't start with a drive letter). This means the second branch was taken, resulting in virtualFileUri.toString() being exactly file:////wsl$/Ubuntu/home/tim/.../file.py (four slashes).

Assuming all of that are correct, this URI is then passed to the default implementation by JetBrains, which eventually concluded it leads to nowhere. What that does boils down to:

// .path: //wsl$/Ubuntu/home/tim/../file.py
LocalFileSystem.getInstance().findFileByPath(URI(virtualFileUri.toString()).path)

The latest commit I just pushed should help us confirm this, but if even this call is unable to find your file, then I'm not sure what else I can do.

[TimChild]

Well, here's the new logs:

2024-12-02 18:50:20,943 [  10125]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - Getting file for URI: file:///home/tim/github/AthenaInstruments/app/tests/pyright_tests.py (false)
2024-12-02 18:50:20,944 [  10126]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - Resolved virtual file URI: file://wsl.localhost/Ubuntu/home/tim/github/AthenaInstruments/app/tests/pyright_tests.py
2024-12-02 18:50:20,944 [  10126]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - File system: LocalFileSystem
2024-12-02 18:50:20,944 [  10126]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - Virtual file: null
2024-12-02 18:50:20,944 [  10126]   WARN - #c.i.p.l.i.h.LspDiagnosticsCache - Could not find file with diagnostics: file:///home/tim/github/AthenaInstruments/app/tests/pyright_tests.py

Resolved virtual file URI looks good... But still not working.

Anything there enlightening to you?

[InSyncWithFoo]

@TimChild There is, actually. The virtual file URI has //wsl.localhost instead of ////wsl$. I'm not certain fixing this will change anything though, so I also ported WSL support to the Command line mode as a fallback. Could you try both of them and see if they work?

[TimChild]

@InSyncWithFoo
In Native LSP client mode, same behaviour as before:

2024-12-02 21:49:58,310 [ 308065]   FINE - #c.i.p.l.i.c.Lsp4jServerConnector - <-- [email protected]: {"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"uri":"file:///home/tim/github/AthenaInstruments/app/tests/pyright_tests.py","version":9,"diagnostics":[{"range":{"start":{"line":9,"character":21},"end":{"line":9,"character":24}},"message":"Argument of type \"Literal['5']\" cannot be assigned to parameter \"var\" of type \"int\" in function \"temp\"\n  \"Literal['5']\" is not assignable to \"int\"","severity":1,"code":"reportArgumentType","source":"Pyright","codeDescription":{"href":"https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportArgumentType"}},{"range":{"start":{"line":9,"character":16},"end":{"line":9,"character":25}},"message":"Type \"str | None\" is not assignable to declared type \"str\"\n  Type \"str | None\" is not assignable to type \"str\"\n    \"None\" is not assignable to \"str\"","severity":1,"code":"reportAssignmentType","source":"Pyright","codeDescription":{"href":"https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportAssignmentType"}},{"range":{"start":{"line":13,"character":20},"end":{"line":13,"character":27}},"message":"Type \"str | None\" is not assignable to declared type \"str\"\n  Type \"str | None\" is not assignable to type \"str\"\n    \"None\" is not assignable to \"str\"","severity":1,"code":"reportAssignmentType","source":"Pyright","codeDescription":{"href":"https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportAssignmentType"}}]}}
2024-12-02 21:49:58,311 [ 308066]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - Getting file for URI: file:///home/tim/github/AthenaInstruments/app/tests/pyright_tests.py (false)
2024-12-02 21:49:58,311 [ 308066]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - Resolved virtual file URI: file://wsl.localhost/Ubuntu/home/tim/github/AthenaInstruments/app/tests/pyright_tests.py
2024-12-02 21:49:58,311 [ 308066]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - File system: LocalFileSystem
2024-12-02 21:49:58,311 [ 308066]   INFO - #com.insyncwithfoo.pyright.lsp.PyrightServerDescriptor - Virtual file: null
2024-12-02 21:49:58,311 [ 308066]   WARN - #c.i.p.l.i.h.LspDiagnosticsCache - Could not find file with diagnostics: file:///home/tim/github/AthenaInstruments/app/tests/pyright_tests.py

In Command line mode it now appears to work! Squigglies where I would expect with Pyright hints.

Weird that my wsl2 behaviour is different to yours, but great that you managed to get the command line mode working.

Unfortunately, I now have to decide between having pyright, or having my ruff formatting run correctly...
I currently run all my ruff formatting in file watcher tasks because there is another pycharm issue that breaks import sorting with the ruff plugin..., but now the code keeps jumping around as I'm typing, I'm assuming because the pyright command line mode causes frequent file saving in order to re-trigger pyright.
I'll think about my setup more in the morning...

Thanks for all the help!

FYI my wsl2 ubuntu version is:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

And windows version:

Is that the same as you?

[TimChild]

Might also be related to this...
https://youtrack.jetbrains.com/issue/PY-63825/IDE-confused-with-wsl.localhost-and-wsl-paths-and-treat-them-as-two-different-projects

I'm now wondering if there is a common theme with some of the other issues I have with Pycharm (like the debugger not always respecting breakpoints).

[InSyncWithFoo]

(I'm on Windows 10 and Ubuntu 24.04.) I'm glad to hear you got it working, and I'm sorry for not being able to resolve this more cleanly. I'll get the PR merged after a little refactoring. Thanks again for being such an awesome user!

[angelozerr]

@InSyncWithFoo do you know how LSP4IJ should be improved to support ws2l like lsp native does?

Please create an issue in lsp4ij repo to explains what you need. My first impression is that you need a new LSP api but perhaps I am wrong?