Python: editable installation mode for multiple packages

[FRidh]

By default Python packages are built and installed in the Nix store. buildPythonPackage has a shellHook to install a package in editable mode (pip install -e or python setup.py develop) .
This requires invoking nix-shell in the root of the package source tree. A shell.nix file is required in which buildPythonPackage is called. If the root contains setup.py an editable install is done, bypassing the different phases.

This works fine when developing a single package. However, when you're developing multiple packages that need to be accessible from within a single environment, then you cannot install these packages in editable mode. Therefore, what is needed is a way to create a Python environment in which multiple packages can installed in editable mode

Interface

How to define now whether a package should be installed in editable mode? Without considering implementation details yet I thought of the two following ways:

1. add an editable parameter to buildPythonPackage. That way, you have maximum flexibility; you simply override a package and set this flag if you want to have an editable installation.
2. add an editableLibs parameter to buildEnv. Because you will generally want a Python environment, it might make sense to have it here, as it gives you directly an overview of editable installations. A downside would be that contrary to option 1 this doesn't allow any editable installations of packages when used as library in an application.

Implementation

editable parameter for buildPythonPackage

During a normal installation this parameter would be ignored; the parameter is only relevant for the shellHook. However, the shellHook is only used when buildPythonPackage is called directly in the Nix expression. Therefore a shellHook in buildPythonPackage is not a solution. Perhaps use the installPhase for installing in editable mode in case editable=true?

editableLibs parameter for buildEnv

This is perhaps the easiest solution. Add an editableLibs parameter which calls for each item in the list a function that installs the respective item in editable mode.

Nice to have

Editable installations for all packages, not only for those you have the source locally. Obviously, we would need a place then outside of the store to put the editable sources.

---

Python to-do list: #1819

[copumpkin]

I understand the motivation, but don't understand the proposed implementation. The Nix store is conceptually immutable, and both buildEnv and buildPythonPackage produce outputs in the Nix store. As soon as you start talking about mutable values, you get into the hairy notion of identity, which I don't think we have a good answer for.

[pSub]

I don't understand the purposed implementation neither. @FRidh Do you have any prototype or can you explain how this is supposed to work? If currently don't have a solution I would close the issue and reopen if something is "on the way" (even if it is only conceptually).

[FRidh]

Yeah, this is something I really need a solution for but never actually continued with trying to fix.

My idea is to have shellHook in python.buildEnv that, when editableLibs is passed, will install those specified libs in editable mode, in the same way as buildPythonPackage does it now for an individual package. Packages that were not passed to editableLibs will still go via the store whereas those that are specified would not.

[FRidh]

Need to think about this more.

[teto]

I used to be adept of develop mode on ubuntu. I overrode the shellHook of a python package (a few months ago) and just now realized it removed the develop mode.
Shouldn't we add a trace/warning when shellHook is overriden here:
pkgs/development/interpreters/python/build-python-package-setuptools.nix
or is there a legitimate usecase for it ?

[FRidh]

There is a discussion ongoing about standardization of editable installations across backends. One main idea is to build a tiny wheel that points to your project and install that. In our case, we could easily integrate that by adding a editable ? false parameter. This should also allow one to install multiple projects as editable.
https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098/77

[raj-magesh]

Sorry for bumping this thread, but this is the only discussion I could find referencing multiple editable installs of Python packages.

Has there been any progress on this issue or is it still only possible to install one package as an editable one? If I'm not mistaken, the installation of editable packages has been standardized by PEP 660.

[con-f-use]

Yes, PEP 660 was accepted and is widely implemented, e.g. in pip, poetry, hatch and setuptools. Can we re-open this issue? I've seen a bunch of stuff with hatch and poetry2nix regarding develop/editable mode, but there's no documentation or discoverability.