RE: Issue #34 - pip install -e . equivalent?

[adam-grant-hendry]

Beginning this discussion here to continue questions and answers regarding issue #34.

[adam-grant-hendry]

@NeilGirdhar In continuation of our discussion, let's assume I wish to add matplotlib as a dependency to my project. Let's also assume I need to edit it to meet my user's requirements, but it wouldn't make sense for the matplotlib developers to add the edit in a PR because it wouldn't serve other matplotlib users. Hence, I would need to install this in development mode.

If I run

poetry run py -m pip install -e git+https://github.com/matplotlib/matplotlib.git#egg=matplotlib

this installs the package in development mode (under .venv/src/), but doesn't add matplotlib to my pyproject.toml (at least with poetry 1.1).

What should I put in my pyproject.toml to track it? This?:

[too.poetry]
# ...
packages = [
    { include = "my_package" },
    { include = "matplotlib", from = ".venv/src" },
]

[NeilGirdhar]

So your user requires a modified version of matplotlib? Why not fork matplotlib into a new github repo, and then depend on that repo

poetry add my_matplotlib --git git://github.com/myorganization/matplotlib.git

[adam-grant-hendry]

@NeilGirdhar This is a theoretical example. No, I do not need a forked version of matplotlib.

If that is your answer, then that is what the git submodule add command is for. Please read my original question.

[NeilGirdhar]

I'm trying to understand your workflow. In the workflow you describe above, this is how I would do it. How does this solution not satisfy your workflow?

Incidentally, aren't you getting tired of downvoting people's attempts at helping you?

[adam-grant-hendry]

I will always downvote an answer that is not helpful, no I am not tired of it. Voting gives other readers an idea of what users found helpful. I downvoted it because my original question contained this solution in the form of git submodule add. This tells me you didn't read my question clearly, which caused the issue to be litered with superfluous discussion, and consequently further frustrate others coming to the issue trying to find an answer.

[finswimmer]

Hello @adam-grant-hendry,

in your original question you was asking for best practices. For the case you need a modified version of another project, I agree with @NeilGirdhar: Fork this project, modify it and add it as a git dependency.

Until now I cannot see your use case for editable installs. While them see quite practical at a first glance, they made more problems than they solve. There are only a few cases - maybe just one case - where they are the solution of choice: You have a monorepo with multiple projects that depend on each other and even after deployment the paths hasn't change and you don't plan to build wheels or sdist packages.

Being able to work on multiple projects, that depend on each other, at the same time, is another often mentioned use case. That's what I was uses earlier as well. But to my experiences this makes lot of headache as well, when working in a team. If these project are really that tight coupled, that one cannot wait until a function is implemented and deployed the software architecture might be not optimal. Instead start developing these projects as subpackages and if you feel later this is something that is needed by other, split them out into its own package.

[adam-grant-hendry]

@finswimmer

Thank you, and I agree, both with you and @NeilGirdhar: forking is the right way to go.

[arogozhnikov]

I find editable installs a critical feature. So I've made a test what happens if project in poetry-managed, but installed with pip install -e . and it seems to work.

How does it work and should I consider that supported? Is there any benefit of using that over pip + pyproject.toml?

[NeilGirdhar]

Did you set the backend in your pyproject.toml? build-backend = 'poetry.core.masonry.api' There's a way to grab the one off github, but you'll have to find that.

[arogozhnikov]

I did

[jonananas]

@arogozhnikov I think you may be using the default pip 20.0.2 that comes with Ubuntu instead of pip >= 21.3
That's what happens if I do

apt-get update
apt-get install python3
apt-get install python3-pip
pip3 install poetry
poetry new foo
pip3 install -e foo

What does pip3 --version report?

[abn]

Curious, why not just use:

# poetry <= 1.1.13
poetry install --no-dev

# poetry >= 1.2.0a2
poetry install --only default

The root project is installed as an editable project.

[phaseOne]

Looks like you need pip >= 21.3.1, which is the first stable version that supports PEP 660, which introduces support for editable installs for pyproject.toml based builds. Sadly this version of pip is not available on Raspberry Pi OS buster, so I'll have to upgrade to bullseye.

[jonananas]

I believe it's the beauty of using a standard, i.e. that the fact that both pip and poetry supports pyproject.toml is the reason it works. But I am also not 100% certain.

[fredrikaverpil]

@adam-grant-hendry I would clone down the matplotlib repo locally. Then I would depend on it (in editable mode) via the path directive in my project:

git clone https://github.com/matplotlib/matplotlib.git ../matplotlib

[tool.poetry.dependencies]
matplotlib = {path = "../matplotlib", develop = true}

Note: for matplotlib, you need to also install setuptool-scm, as that is required for building from source:

poetry add --dev setuptools-scm

After a poetry install in your project, your project is using the code from your local ../matplotlib folder. Edit the files in ../matplotlib as you wish. Normally, you wouldn't have to re-run poetry install for making changes take effect, but with matplotlib I'm not sure how this works...