Installing local dependencies with develop = true causes unexpected behaviour which was not present in older versions

[manulera]

• Poetry version: 1.6.1
• Python version: 3.9
• OS version and name: macOS 12.6.7 and Ubuntu 20.04
• pyproject.toml: https://github.com/manulera/example_local_python_module_poetry/blob/master/pyproject.toml

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

First of all let me say that I love poetry, and that I recommend it to anyone working with python, so a big thanks for all the team for the great package manager.

This issue is related to discussion #5276 and issues #34, and #8513 but describes a different problem (I think). Tagging @neersighted since they closed #34 and @adam-grant-hendry since they created the discussion, and @francoisWeber since they created #8513.

In this repo I have a minimal example to highlight the problem, with a directory structure like below:

.
├── mymodule
│   ├── mymodule
│   │   ├── __init__.py
│   │   └── functions.py
│   └── setup.py
├── poetry.lock
├── poetry.toml
├── pyproject.toml
└── script.py

Where mymodule is a local python module. During development of an analysis pipeline, you may want to have code organised like this in a repository where you want to import common functions from mymodule in scripts located in subfolders. I would expect that the way to import a function present in mymodule/mymodule/functions.py from another script (./script.py) would be from mymodule.functions import print_hello. poetry 1.6.1 gives an unexpected behaviour as described below, tested with github actions. The github action runs reproduce.sh, which:

• Installs dependencies with poetry and runs script.py
• Removes the mymodule dependency, installs it with pip install -e . and runs script.py again.

Installing module with poetry 1.6.1

See this action run

• from mymodule.functions import print_hello works, but does not show the go-to-function in VSCode, as described in Local developping modules ill referenced by VSCode's "go to definition" #8513.
• from mymodule.mymodule.functions import print_hello (mymodule twice) also works (I don't think it should, hence the bug🐛), and has go-to-function in VSCode.

Installing module with pip install -e . or poetry 1.2.2

See this action run

• from mymodule.functions import print_hello works, and shows the go-to-function in VSCode (expected behaviour in Local developping modules ill referenced by VSCode's "go to definition" #8513).
• from mymodule.mymodule.functions import print_hello does not work (expected behaviour).

Conclusion

I think something must have changed to between poetry 1.2.2 and current version in how local dependencies are handled, that produces an unexpected behaviour and prevents VScode intellisense. Happy to help to fix this if given some guidance.

[dimbleby]

the difference between poetry install and pip install is likely that poetry will install mymodule as editable where pip won't - because afaik there's no way for the poetry-core backend to communicate to pip through wheel metadata that it wants mymodule to be editable

from mymodule.mymodule.functions import print_hello of course also works if you don't do any installing at all...

I doubt that any of this represents a bug, just quirks of how things work.

[manulera]

Hi @dimbleby thanks for the quick response. Just a clarification on what you mentioned and why I still think this is a bug, or at least unintended behaviour.

pip install -e . does install the module as editable, that's why mymodule = { path = "mymodule", develop = true } was added as its equivalent, as mentioned in #34.

I understand why from mymodule.mymodule.functions import print_hello works without installing, but it should not work if there is an installed module with the same name. You can test this with your favourite library, I did it with regex (I installed the module with poetry, then created a directory as below).

regex/
├── regex
│   ├── __init__.py
│   └── blah.py
└── setup.py

If I use from regex.regex.blah import blah, it does not work, and tells me No module named 'regex.regex.blah'; 'regex.regex' is not a package. I think this is the intended behaviour when a local directory name clashes with an installed package, and this is how poetry works for modules installed the "normal" way poetry add xyz as per my example. pip works like this for both normal and editable folders, and that's how poetry 1.2.2 used to work, as seen in this action. This means that at least at some point in the past this was considered the intended behaviour in poetry as well.

[dimbleby]

pip install -e . does not install mymodule as editable. It installs the top-level project as editable, but not the "editable" mymodule dependency of that project.

poetry has a new installer since 1.2.2 - https://github.com/pypa/installer - it might very well behave differently than the old one but I am very not convinced that this is a bug.

Even if it is you probably want to go that project with it.

[manulera]

Hi @dimbleby, sorry if this was not clear, but I run pip install -e inside the mymodule folder:

https://github.com/manulera/example_local_python_module_poetry/blob/73436f965c682c712646b77cdd6dfedcbffb85ab/reproduce.sh#L20-L21

This installs mymodule as a package, as seen in the output of pip list:

Package    Version   Editable project location
---------- --------- -------------------------------------------------------------------------------
mymodule   0.0.0     /.../example_local_python_module_poetry/mymodule
pip        22.1.2
regex      2023.10.3
setuptools 62.3.2
wheel      0.37.1

This is the pip -e option documentation.

With this clarified, don't you think that the previous example (behaviour of regex vs. mymodule in poetry 1.6.1 vs. pip and poetry 1.2.2) is unintended behaviour?

[dimbleby]

Then I don't reproduce, from mymodule.mymodule.functions import print_hello works fine with an editable pip install too:

$ pip --version
pip 23.3.1 from /home/dch/.virtualenvs/foo/lib/python3.10/site-packages/pip (python 3.10)

$ (cd mymodule ; pip install -e .)
Obtaining file:///home/dch/example_local_python_module_poetry/mymodule
  Installing build dependencies ... -^[[\^[[done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: mymodule
  Building editable for mymodule (pyproject.toml) ... done
  Created wheel for mymodule: filename=mymodule-0.0.0-0.editable-py3-none-any.whl size=2462 sha256=7f3a82f1163ece52f9ab085b7af9fb84ec24974b919b55dcd0cfc7486ad4406e
  Stored in directory: /tmp/pip-ephem-wheel-cache-5d3stkdu/wheels/ed/ff/fe/53c2d4c1bc0a2f162151c1faaa1708acc9cdd1e912e122883d
Successfully built mymodule
Installing collected packages: mymodule
Successfully installed mymodule-0.0.0

$ python -c 'from mymodule.mymodule.functions import print_hello; print_hello()'
hello

[manulera]

Hi @dimbleby, it seems the behaviour of pip has changed. I was using pip 21.2.3 (🚨2 years old🚨) in my computer. I ran your small test on the new and old version of pip, and the new version of pip does not give an error indeed.

• New pip: https://github.com/manulera/example_local_python_module_poetry/actions/runs/6722799034
• Old pip: https://github.com/manulera/example_local_python_module_poetry/actions/runs/6722754370

17c17
<           pip install --upgrade pip
---
>           pip install --upgrade pip==21.2.3

This was the only difference between the two actions. Not sure what to make of this, I would have not expected such a significant change in pip behaviour. I guess then poetry is mimicking pip behaviour and this can be closed.

Thanks for taking the time to reproduce, and for maintaining poetry, which is really great.

[dimbleby]

no worries, please close

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.