pip downloads packages with VCS specifiers when using --no-index and --find-links to a local directory.

[gotgenes]

This is a duplicate of #3185, however, I'm trying to document this as a bug in a clear and actionable manner for pip's maintainers. The bug regards pip downloading packages over the Internet rather than using locally available packages.

As mentioned in multiple places throughout the pip documentation (see here and here), using the options --no-index and --find-links </path/to/local_dir> should cause pip to install packages directly from the user's file system, and should prevent pip from downloading and installing packages from the Internet. This behavior applies correctly to packages designated with requirements specifiers. This behavior does not apply correctly to packages designated with version control specifiers.

In the case of VCS specifiers, pip ignores the combination of --no-index and --find-links, and will instead attempt to clone and checkout the package over the Internet and install from the downloaded code. It does this regardless of whether a version of the package is present in the directory passed to --find-links.

The shell session below demonstrates the unexpected behavior.

$ pip -V
pip 7.1.2 from /Users/clasher/.virtualenvs/test/lib/python2.7/site-packages (python 2.7)
$ pwd
/Users/clasher
$ mkdir python_packages
$ pip install --download python_packages PyMySQL
Collecting PyMySQL
  Downloading PyMySQL-0.7.0-py2.py3-none-any.whl (75kB)
    100% |████████████████████████████████| 77kB 2.9MB/s
  Saved ./python_packages/PyMySQL-0.7.0-py2.py3-none-any.whl
Successfully downloaded PyMySQL
$ ls python_packages/
PyMySQL-0.7.0-py2.py3-none-any.whl
$ pip install --no-index --find-links ~/python_packages/ git+https://github.com/PyMySQL/PyMySQL.git#egg=PyMySQL
Ignoring indexes: https://pypi.python.org/simple
Collecting PyMySQL from git+https://github.com/PyMySQL/PyMySQL.git#egg=PyMySQL
  Cloning https://github.com/PyMySQL/PyMySQL.git to /private/var/folders/hl/n4818q5j4sg61mr1m2hv2yym0000gn/T/pip-build-hOBcly/PyMySQL
Installing collected packages: PyMySQL
  Running setup.py install for PyMySQL
Successfully installed PyMySQL-0.7.0

Note that pip successfully downloaded the PyMySQL wheel to ~/python_packages, but then ignored this wheel when installing PyMySQL, when specified using a VCS specifier. This is not the expected behavior.

The expected behavior is that pip will use the available downloaded wheel ~/python_packages/PyMySQL-0.7.0-py2.py3-none-any.whl. pip correctly exhibits this behavior when we provide only the package name (requirements specifier), itself:

$ pip uninstall PyMySQL
Uninstalling PyMySQL-0.7.0:
  /Users/clasher/.virtualenvs/test/lib/python2.7/site-packages/PyMySQL-0.7.0-py2.7.egg-info
  /Users/clasher/.virtualenvs/test/lib/python2.7/site-packages/pymysql/__init__.py
  […]
  /Users/clasher/.virtualenvs/test/lib/python2.7/site-packages/pymysql/util.py
  /Users/clasher/.virtualenvs/test/lib/python2.7/site-packages/pymysql/util.pyc
Proceed (y/n)? y
  Successfully uninstalled PyMySQL-0.7.0
(test)[clasher@clasher-MBP]─[26197]─[11:53]──[~]
$ pip install --no-index --find-links ~/python_packages/ PyMySQL
Ignoring indexes: https://pypi.python.org/simple
Collecting PyMySQL
Installing collected packages: PyMySQL
Successfully installed PyMySQL-0.7.0

To correct this behavior, VCS specifiers should be parsed for the package name (given in the trailing #egg=<package name>), and search for a wheel/zip/tarball matching that package name in the local directory.

[gotgenes]

@dstufft, @xavfernandez, would you have the time to confirm that this behavior is a bug, and if so, would you please provide guidance as to what part of pip's codebase determines this behavior?

[xavfernandez]

I would not consider this as a bug:

--no-index means that pip won't contact any of its indexes (i.e. PyPI) to find a link.

pip install --no-index http://lxml.de/files/lxml-3.5.0.tgz tells pip to download and install lxml-3.5.0.tgz, the --no-index has no impact on that.

git+https://github.com/PyMySQL/PyMySQL.git#egg=PyMySQL is just an other case: --no-index is completely unrelated.

Concerning --find-links, pip has no way to know if any of the wheels in the directory corresponds to the unknown version currently hosted at git+https://github.com/PyMySQL/PyMySQL.git.

[gotgenes]

@xavfernandez, thanks for your reply. Regarding this point,

Concerning --find-links, pip has no way to know if any of the wheels in the directory corresponds to the unknown version currently hosted at git+https://github.com/PyMySQL/PyMySQL.git.

perhaps this is the real issue.

For VCS specifiers, users encode their desired package in the form of a trailing #egg=<PackageName>. In my example of, git+https://github.com/PyMySQL/PyMySQL.git#egg=PyMySQL, I specified #egg=PyMySQL. Note that I did not specify a particular version, like #egg=PyMySQL==0.7.0. In this case, I agree, pip would have a difficult time establishing a priori that the version at the Git location matches the requested version. Without the version specifier, however, as a user, I expect pip to defer to accepting whatever version is present in the directory specified by --find-links. There is no conflict of me asking pip to check the version specifier for a match.

[qbedard]

I am running into this issue as well.

I am building wheels from VCS-specified dependencies (PEP 508 style: <packagename> @ git+ssh://[email protected]/<user>/<repo>@<branch>#egg=<eggname>), and then installing from those wheels. I would expect pip to check the package name against the wheels in the dir passed to --find-links, but instead it just tries to check out the code from VCS again. Now that we have PEP 508, we do have a specified distribution name and version for the package, so pip should be able to check it against the wheels to determine if it's a best match and use it instead of retrieving it, right?

To xavfernandez's point, perhaps this warrants a separate flag from --no-index as well, since the VCS link isn't really an index? 🤷‍♂️

[benoit-pierre]

There's no version specifier when using a direct URL requirement, so now way to know what version is linked, see #5898.

[cjerdonek]

cc @uranusjr

[uranusjr]

I have to agree with the interpretation of @xavfernandez (and @benoit-pierre). The URL requirement should absolutely trump requirements without a specifier. The URL is (conceptually) a specifier, and would override the specifier-less counterpart. The same applies to PEP 508 style; pip has no way to tell whether the wheel is generated from that exact URL or not from find-links or index information, and needs to respect the URL specification, instead of assuming the URL matches the wheel.

I also believe this is in fact what most people expects (URL-based requirement over wildcard). To overcome this, Python needs a way to use the URL as a version specifier, and put it in the file name, like how the version specifier is put inside the wheel and sdist filename. You’ll need a PEP for that.

[pfmoore]

I also agree with @xavfernandez, and in fact I view the situation more simply - --no-index and --find-links affect where pip searches for a named requirement like PyMySQL. But a URL link is not requesting a search, it's saying "Get this precise URL and install it" [1]. So finder related options don't come into the decision.

[1] The #egg= fragment is to tell pip the project name that the user expects to be installed using that URL - it's for use by pip's dependency resolution (see here), not to let pip look elsewhere for alternatives.

[cjerdonek]

Would it make sense, then, to raise an error if either of those options is used with a URL? That could help to head off confusion from users as to why the options aren't behaving like they might expect.

[pfmoore]

Probably not. If the URL requirement had dependencies, the finder options would affect those, and I've no idea how you'd catch that case.

[uranusjr]

By the way, is there an issue for the documentation linked by @pfmoore? This seems outdated (see #5271)

The “project name” component of the url suffix egg=<project name>-<version> is used by pip in its dependency logic to identify the project prior to pip downloading and analyzing the metadata. The optional “version” component of the egg name is not functionally important.

[cjerdonek]

Yes, it was merged today in fact: #5933

[BastienFaure]

Hello, is there any update planned to fix that behavior ? Any work or ongoing chances to look at in the meantime at least as a temporary replacement ?

[pfmoore]

No. As per the comments here, this is not a bug.

[uranusjr]

Should we close this issue? What is the action item here.

[pfmoore]

I think we should, yes.