Editable installs with pip and VS Code

[hoechenberger]

Since we adopted PEP 660 by moving to pyproject.toml, I'm having trouble with Pylance (VS Code) when working with editable installs.

Usually I do:

pip install -e .

from the MNE-Python directory. I can then import mne, and Pylance will recognize the package and provide tab completion etc.

Since we adopted pyproject.toml, I had to install in non-editable mode to have Pylance recognize the package.

I started digging a little and found out it's a pain point for static type checkers (like Pylance) introduced through PEP 660.

The proposed solution is to run setuptools in "legacy" or "compat" mode. However, recent releases of pip don't support this anymore.

My current solution is using the new "strict" mode:

pip install --config-settings editable-mode=strict -e .

With that, everything works as expected, with the only limitation that newly created files don't show up in the editable install – one has to "re-install" in that case. But I can live with that.

We should probably document this behavior.

Another alternative is switching our build backend to Hatch, like Napari did recently to work around this issue. But this step seems to be a bit much.

[cbrnr]

Wait, I've been using editable installs and Visual Studio Code forever, and I haven't noticed any regressions recently. Can you give an example what's not working anymore?

[hoechenberger]

I've had this issue for months, it only occurs with new versions of setuptools and pip

❯ mamba create -c conda-forge -n mne-base-test mne-base ipykernel
❯ mamba activate mne-base-test
❯ mamba uninstall --force mne-base
❯ pip --version
pip 23.3.1 from /Users/richardhochenberger/mambaforge/envs/mne-base-test/lib/python3.12/site-packages/pip (python 3.12)
❯ pip install --quiet -e ~/Development/mne-python
❯ code foo.py

Ensure VS Code is using the correct Python environment

Add the following content to the file:

# %%
import mne

Save the file

You will get red squiggly lines under mne:

Import "mne" could not be resolved Pylance reportMissingImports

Close VS Code

Run:

❯ pip install --quiet --config-settings editable-mode=strict -e ~/Development/mne-python

Start VS Code:

❯ code foo.py

Now Pylance doesn't complain about the missing import anymore.

• Can only be replicated if using new (the latest?) versions of setuptools and pip
• When trying to replicate, ensure that you're not inside the mne-python directory, as Pylance may pick up the nested mne directory there (I'm not 100% sure about this but it's better to exclude this possibility) -> May explain why this problem is not an issue when developing MNE-Python itself; but I often want to use the devel version from an editable install in other projects, where I then run into this limitation

[cbrnr]

I can confirm that this is indeed not working. However, as you suspected, it does work if your working directory is actually mne-python (i.e. the base directory of the editable install). So for developing MNE-Python, the editable install works as expected.

The solutions/workarounds you point out don't really sound convincing to me. Why do you need an editable install? If you have to re-install even in editable strict mode when adding new files, this seems a bit pointless (meaning you might as well install in non-editable mode and re-install every time you make a change). What exactly is your use case?

I'm fine with documenting this special case, but as we've mentioned previously, this does not affect you when you're developing MNE-Python, so I'm not even sure where we should document it.

[hoechenberger]

If you have to re-install even in editable strict mode when adding new files, this seems a bit pointless (meaning you might as well install in non-editable mode and re-install every time you make a change).

Normally I edit existing files only; I don't add new ones. Changes to files that existed by the time the "strict" editable install was performed will be reflected when importing the package. Only newly-created files don't show up.

What exactly is your use case?

For example, when I'm developing MNE-BIDS-Pipeline and want to improve mne.Report, I have my workspace / environment with MNE-BIDS-Pipeline open in VS code, and an editable install of MNE-Python into that environment, such that every time I change something on my MNE-Python development branch, the Pipeline will pick it up upon a re-run.

I had a similar issue recently when developing MNE-BIDS.

[drammock]

Another alternative is switching our build backend to Hatch, like Napari did recently to work around this issue. But this step seems to be a bit much.

It actually should be fairly simple to try this out: napari/npe2#280 (comment) / https://github.com/napari/npe2/pull/282/files

@hoechenberger can you see if that minimal change to pyproject.toml (and installation of hatchling) is enough to fix the issue?

One thing we'd need to watch out for is whether we can still get setuptools_scm-like behavior for our version string.

[cbrnr]

A bit more info is also available here: https://learn.scientific-python.org/development/guides/packaging-simple/

[hoechenberger]

Excellent resource, @cbrnr!

@drammock I can look into this tomorrow

[hoechenberger]

Just a note to self, I just ran into the same issue when trying to use an editable install of MNE-BIDS.

With autoreject, I receive:

❯ pip install --quiet --config-settings editable-mode=strict -e ../autoreject
  DEPRECATION: Config settings are ignored for project file:///Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/Development/autoreject. pip 24.0 will enforce this behaviour change. A possible replacement is to use --use-pep517 or add a pyproject.toml file to the project.

cc @sappelhoff

[cbrnr]

I'd be interested in what adopting hatchling as a build backend means for users. It will probably not affect normal pip install mne, right? But I'm assuming pip install -e /some/path/to/mne-python will require the hatch package. Is this correct?

[hoechenberger]

@cbrnr The nice thing about pyproject.toml and PEP 660 (?) is that we can specify a build system:

mne-python/pyproject.toml

Lines 45 to 47 in fcb5926

	[build-system]
	requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2", "wheel"]
	build-backend = "setuptools.build_meta"

This will be installed automatically before the build process starts.

Users don't have to worry about this.

[cbrnr]

But what is "the build process"?

[hoechenberger]

But what is "the build process"?

The package is first built, then installed, when you run pip install ./my-package or pip install ./my-sdist.zip

[cbrnr]

So it's the process of building a wheel, more or less, right? Which means that this change does not affect pip install mne at all, because the wheel has already been built.

[drammock]

One thing we'd need to watch out for is whether we can still get setuptools_scm-like behavior for our version string.

Looks like that won't be a problem... from the scientific-python page that @cbrnr linked to:

You can tell hatchling to get the version from VCS. Add hatch-vcs to your build-backend.requires, then add the following configuration:

[tool.hatch]
version.source = "vcs"
build.hooks.vcs.version-file = "src/<package>/version.py"

[hoechenberger]

So it's the process of building a wheel, more or less, right?

Yes

[ofek]

Maintainer of Hatch/Hatchling here! Let me know if I can be of assistance 🙂