Simple workflow for overriding "dependency of dependency" versions?

[kkom]

What's the problem this feature will solve?

Sometimes I need to use an unreleased version of a Python package, to get a fix for an issue that's blocking me.

While it's pretty straightforward with direct dependencies (thanks to vsc support), it's not as easy for fixes in the dependencies of my dependencies.

Describe the solution you'd like

Well, I don't know! Asking for help - maybe it's already possible.

I can tell you what doesn't work.

In this case aiogremlin is a dependency of goblin:

https://github.com/goblin-ogm/goblin/blob/e45c32956eb93be8fa55eba15a3a08919626528b/setup.py#L31-L33

I would like to use an unreleased VCS checkout of aiogremlin:

# requirements.in
git+git://github.com/kkom/aiogremlin.git@301c9d1dd0cf07b50934c8df2b51084acea86cd7
goblin

I was hoping that explicitly specifying a version of aiogremlin will result in just a single entry for this package in requirements.txt, but it wasn't the case:

# requirements.txt
aenum==2.2.3              # via aiogremlin, gremlinpython
aiogremlin==3.3.4         # via goblin
git+git://github.com/kkom/aiogremlin.git@301c9d1dd0cf07b50934c8df2b51084acea86cd7  # via -r requirements.in
aiohttp==3.6.2            # via aiogremlin
async-timeout==3.0.1      # via aiohttp
attrs==19.3.0             # via aiohttp
chardet==3.0.4            # via aiohttp
goblin==2.2.3             # via -r requirements.in
gremlinpython==3.4.3      # via aiogremlin
idna==2.9                 # via yarl
inflection==0.4.0         # via aiogremlin
isodate==0.6.0            # via gremlinpython
multidict==4.7.5          # via aiohttp, yarl
pyyaml==5.3.1             # via aiogremlin
six==1.14.0               # via aiogremlin, gremlinpython, isodate
tornado==4.5.3            # via gremlinpython
yarl==1.4.2               # via aiohttp

Alternative Solutions

I could modify the goblin package too, but this sounds like a lot of boilerplate work...

[kkom]

As a small update, I tried to modify the goblin package to specify as a dependency a patched version of aiogremlin, which didn't work either unfortunately: pypa/pip#8105

Thanks in advance for any help from the pip-tools community! :)

[kkom]

Solution

Specifying the requirements.in file in this way does exactly what I wanted:

# requirements.in
aiogremlin @ git+https://github.com/kkom/aiogremlin.git@301c9d1dd0cf07b50934c8df2b51084acea86cd7
goblin

requirements.txt ends up this way:

# requirements.txt
aenum==2.2.3              # via aiogremlin, gremlinpython
git+https://github.com/kkom/aiogremlin.git@301c9d1dd0cf07b50934c8df2b51084acea86cd7  # via -r requirements.in, goblin
aiohttp==3.6.2            # via aiogremlin
async-timeout==3.0.1      # via aiohttp
attrs==19.3.0             # via aiohttp
chardet==3.0.4            # via aiohttp
goblin==2.2.3             # via -r requirements.in
gremlinpython==3.4.3      # via aiogremlin
idna==2.9                 # via yarl
inflection==0.4.0         # via aiogremlin
isodate==0.6.0            # via gremlinpython
multidict==4.7.5          # via aiohttp, yarl
pyyaml==5.3.1             # via aiogremlin
six==1.14.0               # via aiogremlin, gremlinpython, isodate
tornado==4.5.3            # via gremlinpython
yarl==1.4.2               # via aiohttp

Thanks @McSinyx for the hint on the other issue!

[atugushev]

The aiogremlin example from the first message is missed egg fragment. Have you tried this?

git+https://github.com/kkom/aiogremlin.git@301c9d1#egg=aiogremlin

[kkom]

Thanks @atugushev – that works as well

I thought that #egg=aiogremlin was optional and could be inferred from setup.py in most cases, but it turns out it wasn't the case.

Since we're already talking... Is there any difference between those two syntaxes? Should I prefer one in certain situations?

[kkom]

I can think of one reason to prefer #egg=... in this particular case

It propagates the package name to the generated requirements.txt file:

(...)
aenum==2.2.3              # via aiogremlin, gremlinpython
git+https://github.com/kkom/aiogremlin.git@301c9d1dd0cf07b50934c8df2b51084acea86cd7#egg=aiogremlin  # via -r requirements.in, goblin
aiohttp==3.6.2            # via aiogremlin
(...)

And we've just established that the package name is not always inferred from the VCS checkout

[atugushev]

Pip recommends git+https://git.example.com/MyProject#egg=MyProject, see examples https://pip.pypa.io/en/stable/reference/pip_install/#git.

pip @ vcs_url format usually is used in setup.py to specify VCS requirement and support by pip for compatibility (i guess).

[atugushev]

We have a bug btw with this format in setup.py, see #902.

[kkom]

We have a bug btw with this format in setup.py, see #902.

Ah yes! That is exactly what my comment #1111 (comment) referred to. Well, bug or not bug, it's a practical reason to stick with #egg= if using pip-tools.

Thanks @atugushev!

[sambarluc]

I am having issues using a private URL with pip-compile, as even if my requirements.in looks like:

https://my.server/python_pkg-1.2.3.tar.gz#egg=python_pkg==1.4.0

the generated requirements.txt looks like:

python_pkg @ https://my.server/python_pkg-1.2.3.tar.gz

that is with the egg-fragment removed. This is rejected by pip install, when --generate-hashes is used, which is my current approach. Is this expected? Is there any known workaround to allow pulling a package from a private URL and still include the hashes? Should I file a separate bug report?