editable cannot be installed when requiring hashes

[jedie]

• Pip version: 9.0.1
• Python version: 3.5.2
• Operating system: Ubuntu 16.04.3 LTS

Description:

I added --hash= information to my requirements.txt, but it seems that hash usage can't be used in combination with editables?!?

Running pip3 install --exists-action s -r requirements.txt creates this message:

The editable requirement from git+https://github.com/... (from -r requirements.txt (line )) cannot be installed when requiring hashes, because there is no single file to hash.

Is this a "known limitation" or a bug?

[ghost]

This issue appear to be invalid, since there's no reasonable improvement to the error message. Out of curiosity, why did you not understand the error message and what do you think it should be?

[pradyunsg]

Hi @jedie!

pip does hash checking on a single file, the wheel archive or source distribution archive, which are basically a complete representation of what you're about to install. It is easy to just compute the hash of a single file and verify that the hash checks out when getting the same file over the network.

The one case where it doesn't use a wheel archive or a source distribution is editable installations. These installations are really just doing setup.py develop. In this case, there's no single file like a source archive, that is representative of the entire package -- there's nothing to compute the hash of for doing useful hash-checking.

This is really a limitation due to the nature of editable installs. They just are mutually exclusive to hash-checking by virtue of how they work.

---

I hope that clarifies your doubt. I'm curious as to what you think the error message was missing to be able to communicate the same.

[jedie]

I'm aware of the fact that hash compare doesn't work on editables ;)
(Of course, pip could create a hash over all checked out files. But that is not the topic here.)

I have a requirements.txt with some editables and with normal PyPi packages.
The hashes are only attached to the normal package names.

I would assume that pip check the hashes for the normal PyPi downloads.
But pip simply aborts with the message from above and doesn't do anything more.

So: I can't use the hash feature for normal PyPi packages if i have one or more editables in requirements.txt ?!?

[pradyunsg]

Ah, I think what you'll want to do is break up the requirements.txt into 2 and invoke pip 2 times - once without the editable requirements in hash-checking mode and then once with editable requirements. I'm actually not sure which ones you'll want to install first. I imagine the editables with --no-deps followed by the hash checked dependencies via requirements.txt would be a good process.

…

On Fri, 26 Jan 2018, 21:11 Jens Diemer, ***@***.***> wrote: I'm aware of the fact that hash compare doesn't work on editables ;) (Of course, pip could create a hash over all checked out files. But that is not the topic here.) I have a *requirements.txt* with some editables and with normal PyPi packages. The hashes are only attached to the normal package names. I would assume that pip check the hashes for the normal PyPi downloads. But pip simply aborts with the message from above and doesn't do anything more. So: I can't use the hash feature for normal PyPi packages if i have one or more editables in *requirements.txt* ?!? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4995 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADH7SRAQOKUuGjajlz4dVACil63LnHwJks5tOfI9gaJpZM4RshLV> .

[earthboundkid]

This is completely baffling and bizarre. Why should one requirement having a hash affect any other line in the requirements file? What if you only want hashes on some requirements?

[rooterkyberian]

Could we add something like --dont-require-hashes? or disable automatic enabling hash checking for every direct dependency and force use of --dont-require-hashes if someones desires it?

[brandsimon]

The example here is git, isnt it safe to use the hash of the last commit?

[pradyunsg]

If someone has a suggestion on how to fix this, you're welcome to make the suggestion and file a PR.

It'll go through the regular review process as every other PR in pip. If you want to help out and have an idea for how to fix this, please feel free to ping me. 🙃

[pradyunsg]

The example here is git, isnt it safe to use the hash of the last commit?

Editable installs can be from directories that are not under git or even any kind of version control.

I don't think special casing VCS setups is the way to go here.

[Qu4tro]

@pradyunsg if possible I would like your opinion on this subject.
(Disclaimer: I'm moving things to Poetry and tbh this is my vision as an end-user for Poetry)

From my reading there are two main schools of thoughts here:

• Allow pip to install editable packages alongside hashed packages.

  • Pro: Most friction-less approach for users
  • Pro: Doesn't require changes in pip wrappers. (i.e. Poetry).
  • Cons: Require changes in pip

• Use a separate requirements.txt to install editable packages.

  • Pro: Doesn't require changes in pip
  • Pro: Splits "safe" packages, from "unsafe" packages.
  • Cons: Wrappers will be required to output multiple requirements.txt

I might be missing others, possibly important, concerns.

I'm partial to having a separate requirements.txt, due to the splitting between "safe" and "unsafe". I think an application should strive to have "safe" packages and only install "unsafe" explicitly.

[sbidoul]

I personally think editable installs must continue to be rejected when --require-hashes is used, because the first thing pip does with these is update the src directory which may have local changes.

I'm in favor of #6469, however.

@Qu4tro assuming #6469 gets traction, a third approach will be to try to understand why pip wrappers such as poetry or pip-tools use editable installs. Since I'm working towards implementing #609, I'm curious to understand if pip-tools and poetry would benefit from it and/or PEP 610. My general goal with this work is to reduce the need of using --editable for working with VCS requirements.

[kapilt]

there are many reasons for editable, most commonly is when referencing the package under development, most commonly for local development (avoiding extra copy of src into site-packages) to facilitate editing source, and ci/testing setup (ie I run into this issue with tox), but also arising in more complex cases of a mono repo with several packages and interdependencies.

of the two variants, @Qu4tro suggested, I'd like to see the allow pip to install editable packages alongside hashed packages. unfortunately the other variant isn't really compatible with other tools in the ecosystem (tox), which want to generate a single invocation to pip for install with all configured requirement files/dep strings.

@sbidoul afaics, your conflating editable with vcs when its just a directory vs a vcs url, ie vcs urls are editable, but editable does not imply vcs url it may just be a path/dir

[uranusjr]

Personally I would prefer to keep the current default; editable installs are by definition cannot be pinned, so it does not make sense to ask for it when hashes are required. Making it a special case feels to me like creating a loophole. The two-step solution not working for tools sounds like a toolchain problem to me; tools should provide a way to run multiple commands in the installation phase, instead of pip (or any other installer command) needing to cram steps into one command.

I do feel though it would be acceptable to have some way for the user to explicitly say “hey I know I said I want hashes, but this one is special”. The flag can be added to the command (i.e. the --no-require-hashes proposal) or applied on a per-package bases (i.e. add something like --no-hash or --hash=none to a line in requirements.txt). It would be a conscious decision if the user decides to supply that, and if that’s the case, fine (we’re grown-ups here).

[petergaultney]

I'm not trying to weigh in on anything else that's being discussed, but I agree with @carlmjohnson that having --hash on a single line in a requirements.txt fundamentally 'coupled' to every other line in that same file both violates the principle of least surprise and in general feels like incorrect software design. In fact, it's basically impossible to imagine an implementation of this that isn't more complex than the straightforward implementation where each line of a requirements.txt handles its flags individually.

FWIW, I have written a workaround wrapper for this that works well for our internal use case but likely could be improved if other people wanted to start using it and trying it out. But it takes 'advantage' of the odd situation by basically attempting a --no-deps install of every requirement individually, assuming that dependency resolution has already been accomplished separately, so it probably isn't a truly general purpose solution.

[pfmoore]

having --hash on a single line in a requirements.txt fundamentally 'coupled' to every other line in that same file both violates the principle of least surprise and in general feels like incorrect software design.

I believe that it was a deliberate decision, based on the wishes of the people who asked for hash checking mode in the first place - the idea being to avoid a security loophole where you are hash-checking everything, but accidentally add a dependency without a hash, leaving your application insecure when you think it's secure. I do agree that explicitly specifying --require-hashes seems like a reasonable thing in that case, though.

My memory is very vague on the details here, and I could easily be completely wrong. I encourage anyone who wants to argue for dropping the implicit enabling of --require-hashes whenever a hash is specified, to do some research to confirm (or otherwise) the original intent here. (And please do add a link here, to help others - including me - confirm the details).

[mikenerone]

Just my 2 cents as to the behavior that I would personally naturally expect:

• Default: All deps are valid, hash or not, but any that specify hashes have them enforced. (Unsurprising behavior: use all information present, don't make additional assumptions.)
• --require-hashes: The input is invalid and pip aborts immediately unless all deps specify hashes. (Unsurprising behavior: all deps "require hashes" just like the option says.)
• --ignore-hashes: All deps are valid, hash or not, and any hashes that are present are not enforced. (Unsurprising behavior: pip "ignores hashes" just like the option says.)

[mitar]

I think this issue could be resolved in backwards compatible way by:

• Keeping current behavior that one hash forces hashes on all entries in requirements txt. I think we all agree that this is a more secure behavior.
• We define and support hash on editable installations. I do not see why it is so hard to recursively compute hash of a directory in a deterministic way and this is it. It is up to the entry in the requirements file and the user to assure that the entry points to something which installs an editable directory with the same hash. If it is a git repository, it is easy, you point to a git commit hash, if it is local directory, it is on you to make sure local directory has not been changed.

[mikenerone]

I do not see why it is so hard to recursively compute hash of a directory in a deterministic way...it is on you to make sure local directory has not been changed.

Aren't we talking about an editable install - i.e. the whole purpose is that you're editing it? It is, by definition, changing constantly, as @pradyunsg alluded to in their first comment.

[earthboundkid]

There are many reasons to use an editable install. One is to edit them. Another is to use a package which isn't on PyPI, such as a fork or private repo. For the former case, a hash is impractical, but for the latter it can be useful.

I often use editable packages just to make it easier to read the source for an important dependency in my text editor by putting it in a top level folder instead of buried in a virtual env somewhere (although I admit this is a niche usecase, so I wouldn't put much weight on it per se).

[uranusjr]

The raison d'etre of editable installs is to make the installed code editable. If all you want is to install something from private locations, you can just install from path without the editable flag. This issue is specifically about editable installs, and your latter reason thus does not apply.

[apljungquist]

I have not seen this idea mentioned anywhere so I thought I'd put it on the radar; Allow hashes to be explicitly ignored using a wildcard e.g. like --hash=*.

For context this would facilitate developing multiple packages out of one repo with internal and external dependencies. If for instance the dependency graph looks like below then it would be convenient to be able to do something like pip install -c constraints.txt intarnal1. At present this works only without hashes.

external<--+
internal0<-+
           +-internal1

[sbidoul]

I have been thinking about this again, as I came across the pip install -e . -c constraints.txt use case, where constraints.txt has requirements with hashes. This cannot be resolved by splitting into two pip install calls while keeping the desired semantics.

So I'm wondering we could accept requirements provided as local directories in --require-hashes mode.
I am under the impression (from skimming this thread again) that, combined with #11968, this could resolve most use cases.

Of course that means that --require-hashes will not detect when a user adds a local directory to requirements anymore, but I'm not sure that was ever a goal of --require-hashes.

[sinoroc]

the pip install -e . -c constraints.txt use case, where constraints.txt has requirements with hashes

I am confronted with the exact same use case currently. This looks like it could be a good substitute for poetry install (constraints.txt is generated beforehand with poetry export --format=constraints.txt), but alas, it fails. I could not find a way to do this with two pip commands either.

P.S.: Soon after writing this, I realized that 1. it does not bring the discussion any further so I could have refrained from writing anything at all; and 2. I had excluded using requirements (instead of constraints), but I could not remember why, but now that I have tried properly with requirements again, it seems to be good enough for my use case poetry export > req.txt pip install --no-deps -r req.txt pip install --no-deps -e ..