pip freeze fails to properly return hg URL if virtualenv is under Git controlled project

[ruiwen]

• Pip version: 8.1.2
• Python version: 2.7.12
• Operating System: Ubuntu 16.04.1

Description:

I've got my virtulenv directory (venv) under my project folder

acme-server/
├── bin
├── acme
├── manage.py
├── Procfile
├── requirements
├── requirements.txt
├── venv
└── web

One of my requirements is installed via -e hg+https://, which ends up getting cloned to venv/src.

I'd like to automatically update my requirements.txt via pip freeze, but pip keeps returning my hg repo as a git URL instead.

I'd expect

-e hg+https://bitbucket.org/somedev/some-project@<some_projects_hash>#egg=some_project

but I keep getting this instead

-e [email protected]:Acme/acme-server.git@<my_projects_hash>#egg=some_project&subdirectory=venv/src/some_project

which isn't very helpful for my requirements.txt.

What I've run:

I dug into pip's source, and narrowed it down to controls_location() in pip/vcs/git.py.

   @classmethod
   def controls_location(cls, location):
       if super(Git, cls).controls_location(location):
           return True
       try:
           r = cls().run_command(['rev-parse'],
                                 cwd=location,
                                 show_stdout=False,
                                 on_returncode='ignore')
           return not r
       except BadCommand:
           logger.debug("could not determine if %s is under git control "
                        "because git is not available", location)
           return False

Turns out, running git rev-parse in the hg-cloned copy of some_project does not raise an error, because git traverses the tree upwards, eventually reaching the top level of my own project and finding the .git there instead. That would seem why <my_projects_hash> appears in the pip freeze output.

In pip/vcs::VcsSupport.get_backend_name(), it would appear the the git module is running first, and because git rev-parse is not raising an error from within the hg-cloned directory under venv/src, is claiming the hg-cloned project as a git repo instead (even before the hg module has a chance to run).

[ruiwen]

Using the --show-cdup option to git rev-parse in controls_location() seems to avoid my particular problem.

   @classmethod
   def controls_location(cls, location):
       if super(Git, cls).controls_location(location):
           return True
       try:
-          r = cls().run_command(['rev-parse'],
+          r = cls().run_command(['rev-parse', '--show-cdup'],
                                 cwd=location,
                                 show_stdout=False,
                                 on_returncode='ignore')
           return not r
       except BadCommand:
           logger.debug("could not determine if %s is under git control "
                        "because git is not available", location)
           return False

In the hg-cloned project, with git identifying the top-level project .git/,

$ pwd
/home/user/acme-server/venv/src/some-project
$ git rev-parse --show-cdup
../../..

In a git-cloned project under venv/src/

$ pwd
/home/user/acme-server/venv/src/some-git-project
$ git rev-parse --show-cdup

$

[xavfernandez]

Unfortunately, I don't see why --show-cdup would be of any help: git rev-parse --show-cdup still returns a valid path so pip will believe it found a git repo.

Maybe pip should try all its supported VCS and if it found several repository, select the one with the smallest subdirectory path ?
But even this heuristic would not always provide the right answer so...
We are clearly on an edge case, I'm not certain we want to complicate pip freeze code to deal with this...
cc @dstufft @pfmoore

[pfmoore]

Agreed, this seems like it would be pretty complex code for a rare situation. I'm inclined to just say "don't put your virtualenv directory under your project folder", tbh.

[ruiwen]

Unfortunately, I don't see why --show-cdup would be of any help: git rev-parse --show-cdup still returns a valid path so pip will believe it found a git repo.

Maybe pip should try all its supported VCS and if it found several repository, select the one with the smallest subdirectory path ?

This is what worked for me though. rev-parse --show-cdup returned a non zero-length path in the hg-cloned repo (vs a zero-length path in another git-cloned repo) under venv/src).

The return not r line after calling rev-parse --show-cdup then returned False for the git module's controls_location() in that scenario, which was correct.

Since the command is being run in the cloned project's directory, cwd=location, wouldn't a non zero-length return indicate that this is not a git repo (while potentially being in a git repo)?

We are clearly on an edge case, I'm not certain we want to complicate pip freeze code to deal with this...

Given that pip officially supports doing a pip install -e hg+https://..., shouldn't sanely returning the repo during pip freeze be a non-edge case?

Agreed, this seems like it would be pretty complex code for a rare situation. I'm inclined to just say "don't put your virtualenv directory under your project folder", tbh.

It might be that the "edge case" here is the location of the venv. There are other sites that do show examples of having the virtuelenv directory under the project folder though

• http://docs.python-guide.org/en/latest/dev/virtualenvs/#virtualenv
• http://flask.pocoo.org/docs/0.11/installation/#virtualenv
• https://tutorial.djangogirls.org/en/django_installation/#virtual-environment

[xavfernandez]

This is what worked for me though. rev-parse --show-cdup returned a non zero-length path in the hg-cloned repo (vs a zero-length path in another git-cloned repo) under venv/src).

The return not r line after calling rev-parse --show-cdup then returned False for the git module's controls_location() in that scenario, which was correct.

Since the command is being run in the cloned project's directory, cwd=location, wouldn't a non zero-length return indicate that this is not a git repo (while potentially being in a git repo)?

My bad, I scanned the code too quickly.
git rev-parse should return nothing in a git repo, not the path to root dir.
Adding --show-cdup would break pip freeze for project installed via subdirectory of a git repo (as explained in https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support).

It might be that the "edge case" here is the location of the venv. There are other sites that do show examples of having the virtuelenv directory under the project folder though

I agree that having a venv in your project folder (and hence in a vcs repository) is a common usecase.

So maybe changing get_backend_name to test all VCS and return the one with the smallest subdirectory would be appropriate ? PR welcome :)

[ruiwen]

So maybe changing get_backend_name to test all VCS and return the one with the smallest subdirectory would be appropriate ?

I could take a stab at it when I have some free time from my primary project =) Would this be the preferred way about it?

[xavfernandez]

Would this be the preferred way about it?

I don't think of any other way (as --show-cdup would break the subdirectory use case), so yes.