Fixing / documenting how editable installs with --install-dir=$path work

[dwt]

Hi there,

since pypa/pip#9636 got merged, setuptools is now next in line. :-)

I'd like some input on how to proceed here, as I am still quite new to setuptools.

Some immediate thoughts, I think it would be safe to document that right now (i.e. with the next version of pip) it will work to use pip install --target $target --editable path/to/package which is then equivalent to… I'm actually not quite sure yet, python setup.py develop --install-dir=$targetI think? And to activate the packages in that directory code like this can be used:

def add_site(path):
    """This adds a path to as proper site packages to all associated import
    systems.  Primarily it invokes `site.addsitedir` and also configures
    pkg_resources' metadata accordingly.
    """
    site.addsitedir(path)
    ws = pkg_resources.working_set
    ws.entry_keys.setdefault(path, [])
    ws.entries.append(path)
    for dist in pkg_resources.find_distributions(path, False):
        ws.add(dist, path, insert=True)

Then, second would be the question how to best proceed with this in setuptools? A simple thing could be to add code like this in setuptools to ensure .egg-link's in directories on the python path are recognised? I have actually not yet started looking into how setuptools can actually do something there.

I'm not sure yet how backwards compatibility concerns play a role here, but flit allows to just symlink the installed package or allows using .pth files to achieve the same effect.

It is my understanding that even Windows supports symlinks nowadays, so maybe that is a really attractive option to change the install and simplify all the code in question?

Well, some code reading is probably next, but I wanted to open a space for discussion on how to proceed here, so: feedback welcome.

[dwt]

Linking to pypa/pip#4390 to continue the discussion from there.

[dwt]

@jaraco I would really appreciate some input on how to approach this issue.

[jaraco]

Absolutely. Thanks for the ping.

First, some observations mostly off the top of my head:

• New use of pkg_resources is discouraged. If possible, whatever solution we come up with should work without setuptools (and thus pkg_resources) being installed.
• Although pkg_resources recognizes .egg-link files, Python itself does not, and pkg_resources is being replaced by other libraries (importlib.metadata and importlib.resources) that don't recognize egg-links. For that reason, I consider egg-link to be largely deprecated.
• pth files are normally not honored in directories except site-packages, but they can be honored if something calls addsitedir() on that directory early (in site, sitecustomize, or usercustomize modules).
• pth files (easy-install.pth in particular) is how regular develop installs work today. Setuptools can install to --user or the system site-packages path, and since these paths are implicitly inclided in site.addsitedir(), pth files are honored there.
• Although Symlinks are supported on Windows, they still require enabling developer mode or running as Administrator or running as a user with special privileges. Last I checked, it doesn't work out of the box, so can't be relied upon in the general case.
• In Development environments using nix-shell no longer work #2612, we see where other environments have struggled with develop installs to non-site directories when setuptools removed its implicit 'site.py' hack, but the effort there illustrated a simple workaround that nix was able to use to make those prefix directories develop-installable. If we can find a solution that also works for nix's use-case, that'd be great.
• The solution might be as simple as making sure a sitecustomize exists in the target directory that makes sure that addsitedir is added for the target dir and install a .pth file.
• The situation might be complicated if we need to support more than one such install target in the same environment, but we can figure that out later.