Incorrect dependency version installed when specifying a git commit. #5973

tobygomersall · 2023-10-11T17:07:53Z

Issue description

When I try to install a specific git commit of a repository, pipenv ignores the commit hash and installs the head.

Expected result

I expect to see the specified commit hash replicated in Pipfile.lock and the specified commit of the package installed in the virtual environment.

Actual result

The Pipfile.lock contains "ref": null in the package entry and the head of the package repository is installed in the virtual environment.

Steps to replicate

mkdir test
cd test
pipenv install git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595

Inspect the myhdl entry in Pipfile.lock. On my system the ref is set to null.

To confirm, perform an additional check:

pipenv shell
pip freeze

Which confirms that the installed myhdl is the head.

My debugging efforts

Release 2023.7.23 works as expected.

Releases 2023.8.19 and 2023.8.20 fail with an error.

Subsequent releases look to complete successfully but install the wrong version and set null in the lock file.

The problematic code looks to be on lines 115 to 118 of pipenv/pipenv/utils/locking.py

The text was updated successfully, but these errors were encountered:

chrisroat · 2023-10-16T21:38:29Z

See same issue (ref is null using git source w/commit hash) on MacOS w/python 3.11. Going back to 2023.7.23 resolves issue.

matteius · 2023-10-18T06:36:13Z

I tried reproducing this without luck on Windows and ubuntu without luck, here is the output from Windows:

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5973
$ pipenv install git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
Creating a virtualenv for this project...
Pipfile: C:\Users\matte\Projects\pipenv-triage\issue-5973\Pipfile
Using default python from C:\Users\matte\AppData\Local\Programs\Python\Python311
\python.exe (3.11.2) to create virtualenv...
[ ===] Creating virtual environment...created virtual environment CPython3.11.2.
final.0-64 in 4083ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\issue-5973-2dbfxMHD
, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: C:\c\Users\matte\.virtualenvs\issue-5973-2dbfxMHD
Creating a Pipfile for this project...
Installing
git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
...
Resolving
git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
...
[    ] Installing...INFO:pipenv.patched.pip._internal.vcs.git:Cloning https://github.com/myhdl/myhdl.git (to revision ae25af4d593d20a26c85fbb17c0cd98a026e8595) to c:\users
\matte\appdata\local\temp\tmp9l5ajje7
[=   ] Installing None...INFO:pip.subprocessor:Running command git clone --filter=blob:none https://github.com/myhdl/myhdl.git 'C:\Users\matte\AppData\Local\Temp\tmp9l5ajj
e7'
INFO:pip.subprocessor:Cloning into 'C:\Users\matte\AppData\Local\Temp\tmp9l5ajje7'...
[=== ] Installing None...INFO:pip.subprocessor:Updating files:  54% (235/430)
INFO:pip.subprocessor:Updating files:  55% (237/430)
INFO:pip.subprocessor:Updating files:  56% (241/430)
INFO:pip.subprocessor:Updating files:  57% (246/430)
INFO:pip.subprocessor:Updating files:  58% (250/430)
INFO:pip.subprocessor:Updating files:  59% (254/430)
[==  ] Installing None...INFO:pip.subprocessor:Updating files:  60% (258/430)
INFO:pip.subprocessor:Updating files:  61% (263/430)
INFO:pip.subprocessor:Updating files:  62% (267/430)
INFO:pip.subprocessor:Updating files:  63% (271/430)
INFO:pip.subprocessor:Updating files:  64% (276/430)
INFO:pip.subprocessor:Updating files:  65% (280/430)
INFO:pip.subprocessor:Updating files:  66% (284/430)
INFO:pip.subprocessor:Updating files:  67% (289/430)
INFO:pip.subprocessor:Updating files:  68% (293/430)
INFO:pip.subprocessor:Updating files:  69% (297/430)
INFO:pip.subprocessor:Updating files:  70% (301/430)
INFO:pip.subprocessor:Updating files:  71% (306/430)
INFO:pip.subprocessor:Updating files:  72% (310/430)
INFO:pip.subprocessor:Updating files:  73% (314/430)
INFO:pip.subprocessor:Updating files:  74% (319/430)
INFO:pip.subprocessor:Updating files:  75% (323/430)
INFO:pip.subprocessor:Updating files:  76% (327/430)
INFO:pip.subprocessor:Updating files:  77% (332/430)
INFO:pip.subprocessor:Updating files:  78% (336/430)
INFO:pip.subprocessor:Updating files:  79% (340/430)
INFO:pip.subprocessor:Updating files:  80% (344/430)
INFO:pip.subprocessor:Updating files:  81% (349/430)
INFO:pip.subprocessor:Updating files:  82% (353/430)
INFO:pip.subprocessor:Updating files:  83% (357/430)
INFO:pip.subprocessor:Updating files:  84% (362/430)
INFO:pip.subprocessor:Updating files:  85% (366/430)
INFO:pip.subprocessor:Updating files:  86% (370/430)
INFO:pip.subprocessor:Updating files:  87% (375/430)
INFO:pip.subprocessor:Updating files:  88% (379/430)
INFO:pip.subprocessor:Updating files:  89% (383/430)
INFO:pip.subprocessor:Updating files:  90% (387/430)
INFO:pip.subprocessor:Updating files:  91% (392/430)
INFO:pip.subprocessor:Updating files:  92% (396/430)
INFO:pip.subprocessor:Updating files:  93% (400/430)
INFO:pip.subprocessor:Updating files:  94% (405/430)
[=   ] Installing None...INFO:pip.subprocessor:Updating files:  95% (409/430)
INFO:pip.subprocessor:Updating files:  96% (413/430)
INFO:pip.subprocessor:Updating files:  97% (418/430)
INFO:pip.subprocessor:Updating files:  98% (422/430)
INFO:pip.subprocessor:Updating files:  99% (426/430)
INFO:pip.subprocessor:Updating files: 100% (430/430)
INFO:pip.subprocessor:Updating files: 100% (430/430), done.
INFO:pip.subprocessor:Running command git rev-parse -q --verify 'sha^ae25af4d593d20a26c85fbb17c0cd98a026e8595'
INFO:pip.subprocessor:Running command git fetch -q https://github.com/myhdl/myhdl.git ae25af4d593d20a26c85fbb17c0cd98a026e8595
[   =] Installing None...INFO:pip.subprocessor:Running command git checkout -q ae25af4d593d20a26c85fbb17c0cd98a026e8595
[=   ] Installing None...INFO:pipenv.patched.pip._internal.vcs.git:Resolved https://github.com/myhdl/myhdl.git to commit ae25af4d593d20a26c85fbb17c0cd98a026e8595
Added myhdl to Pipfile's [packages] ...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
[==  ] Locking...Warning: INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet https://github.com/myhdl/myhdl.git 'C:\Users\matte\AppData\Local\Temp\
pip-temp-k48u2euc\myhdl_bcdd26e1a5b54e039fed9a83be4579ad'
INFO:pip.subprocessor:Running command git rev-parse -q --verify 'sha^ae25af4d593d20a26c85fbb17c0cd98a026e8595'
INFO:pip.subprocessor:Running command git fetch -q https://github.com/myhdl/myhdl.git ae25af4d593d20a26c85fbb17c0cd98a026e8595
INFO:pip.subprocessor:Running command git checkout -q ae25af4d593d20a26c85fbb17c0cd98a026e8595
INFO:pip.subprocessor:Running command git clone --filter=blob:none https://github.com/myhdl/myhdl.git 'C:\Users\matte\AppData\Local\Temp\tmp1bqge362'
INFO:pip.subprocessor:Cloning into 'C:\Users\matte\AppData\Local\Temp\tmp1bqge362'...
INFO:pip.subprocessor:Updating files:  69% (298/430)
INFO:pip.subprocessor:Updating files:  70% (301/430)
INFO:pip.subprocessor:Updating files:  71% (306/430)
INFO:pip.subprocessor:Updating files:  72% (310/430)
INFO:pip.subprocessor:Updating files:  73% (314/430)
INFO:pip.subprocessor:Updating files:  74% (319/430)
INFO:pip.subprocessor:Updating files:  75% (323/430)
INFO:pip.subprocessor:Updating files:  76% (327/430)
INFO:pip.subprocessor:Updating files:  77% (332/430)
INFO:pip.subprocessor:Updating files:  78% (336/430)
INFO:pip.subprocessor:Updating files:  79% (340/430)
INFO:pip.subprocessor:Updating files:  80% (344/430)
INFO:pip.subprocessor:Updating files:  81% (349/430)
INFO:pip.subprocessor:Updating files:  82% (353/430)
INFO:pip.subprocessor:Updating files:  83% (357/430)
INFO:pip.subprocessor:Updating files:  84% (362/430)
INFO:pip.subprocessor:Updating files:  85% (366/430)
INFO:pip.subprocessor:Updating files:  86% (370/430)
INFO:pip.subprocessor:Updating files:  87% (375/430)
INFO:pip.subprocessor:Updating files:  88% (379/430)
INFO:pip.subprocessor:Updating files:  89% (383/430)
INFO:pip.subprocessor:Updating files:  90% (387/430)
INFO:pip.subprocessor:Updating files:  91% (392/430)
INFO:pip.subprocessor:Updating files:  92% (396/430)
INFO:pip.subprocessor:Updating files:  93% (400/430)
INFO:pip.subprocessor:Updating files:  94% (405/430)
INFO:pip.subprocessor:Updating files:  95% (409/430)
INFO:pip.subprocessor:Updating files:  96% (413/430)
INFO:pip.subprocessor:Updating files:  97% (418/430)
INFO:pip.subprocessor:Updating files:  98% (422/430)
INFO:pip.subprocessor:Updating files:  99% (426/430)
INFO:pip.subprocessor:Updating files: 100% (430/430)
INFO:pip.subprocessor:Updating files: 100% (430/430), done.
INFO:pip.subprocessor:Running command git rev-parse -q --verify 'sha^ae25af4d593d20a26c85fbb17c0cd98a026e8595'
INFO:pip.subprocessor:Running command git fetch -q https://github.com/myhdl/myhdl.git ae25af4d593d20a26c85fbb17c0cd98a026e8595
INFO:pip.subprocessor:Running command git checkout -q ae25af4d593d20a26c85fbb17c0cd98a026e8595
Locking [dev-packages] dependencies...
Updated Pipfile.lock (15e8a5eda9949e87102c00fa1884a3c3f27288c8934043801db642adb16969ca)!
Installing dependencies from Pipfile.lock (6969ca)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5973
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
myhdl = {ref = "ae25af4d593d20a26c85fbb17c0cd98a026e8595", git = "git+https://github.com/myhdl/myhdl.git"}

[dev-packages]

[requires]
python_version = "3.9"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5973
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "15e8a5eda9949e87102c00fa1884a3c3f27288c8934043801db642adb16969ca"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "myhdl": {
            "git": "git+https://github.com/myhdl/myhdl.git",
            "ref": "ae25af4d593d20a26c85fbb17c0cd98a026e8595"
        }
    },
    "develop": {}
}

matteius · 2023-10-18T06:41:15Z

Also just verified this example worked on mac os using latest pipenv -- I am at a loss as to what you are saying.

tobygomersall · 2023-10-18T08:47:28Z

Could this caused by different python versions?

Diff between your Pipfile and mine:

12c12
< python_version = "3.10"
---
> python_version = "3.9"

Diff between your Pipfile.lock and mine:

4c4
<             "sha256": "e967d6ee54dd082c5c5035ebd37e2a7f61e5d627203f1775300d3181df5a91a2"
---
>             "sha256": "15e8a5eda9949e87102c00fa1884a3c3f27288c8934043801db642adb16969ca"
8c8
<             "python_version": "3.10"
---
>             "python_version": "3.9"
21c21
<             "ref": null
---
>             "ref": "ae25af4d593d20a26c85fbb17c0cd98a026e8595"

matteius · 2023-10-18T08:54:56Z

The windows test was actually using:

Using default python from C:\Users\matte\AppData\Local\Programs\Python\Python311
Not sure why it had 3.9 in Pipfile (possibly separate issue)

I used a variety of pythons across the three OS tests I ran and could not reproduce it. Doesn't mean its not an issue for you, but I am not sure yet what could be causing it.

chrisroat · 2023-10-18T13:27:11Z

I ran the same test as @matteius on a fresh install, and got the same results. So I scratched my head a moment. I realized this wasn't equivalent to what I was doing.

I find that when you add to an existing Pipfile, you run into the problem here. Simply doing an empty pipenv install first will repro the issue.

$ pipenv install
...
$ pipenv install git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
...

Now you will find that the Pipfile.lock file as a null ref. (Pipfile has correct ref)

chrisroat · 2023-10-31T02:55:37Z

It is likely worth re-labelling this as a regression (in addition to a legitimate bug), unless it is unable to be reproduced.

Aiky30 · 2024-05-01T10:36:43Z

I can replicate this issue on Pipenv 2023.12.1.

Machine: x86 i7, Ubuntu 22.04.4 64bit

Pipfile.lock example:

somepackage = { ref = "Commit hash here", git = "ssh://[email protected]/somerepo/somepackage.git" }

PipLock example:

        "somepackage": {
            "git": "ssh://[email protected]/somerepo/somepackage.git",
            "ref": null
        },

Rolling back to 2023.7.23 as others have said does indeed work. Whatever changed after this version has broken this functionality.

ohshazbot · 2024-10-21T17:57:07Z

I'm experiencing this with 2024.1.0

matteius · 2024-10-21T18:07:47Z

@ohshazbot could you check if you experience it on this branch: #6276

ohshazbot · 2024-10-21T19:03:19Z

Sorry matteius, I think I got to a resolution before I saw your reply.

But in my case, I'm not sure if it's unsupported behavior, as I was trying to install a specific commit whereas all the documentation only seems to support installing a git branch. And lookikng at the log difference between specifying branch and sha, the sha approach just pulls the latest commit from the default branch. So once I flipped my request over to using a branch identifier instead it worked as intended.

So-

2023.7.23 using git branch - works
2024.1.0 using git branch - works
2023.7.23 using git sha - does not work
2024.1.0 using git sha - does not work

I will say that this was also causing other random failures that manifested as

  File "/home/john/.local/share/virtualenvs/portal-HU6noBrU/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 723, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

, which were pretty useless until I realized there was a verbose flag that brought my attention to errors like

pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    cornice-swagger==

This was a package I recently migrated from a requirements.txt file and worked fine with the shas.

matteius · 2024-10-21T21:11:39Z

Thank you so much for this analysis @ohshazbot -- it is exactly as you describe and the cause of why I opened #6266

Your findings will help me narrow this regression down further in the future.

ohshazbot · 2024-10-21T22:29:26Z

Coolio, this is a company setup to sanitizing is a bit of a pain, but if there's any other details I can share LMK

matteius added Type: Regression This issue is a regression of a previous behavior. Priority: High This item is high priority and should be resolved quickly. triage labels Oct 16, 2023

matteius removed Priority: High This item is high priority and should be resolved quickly. Type: Regression This issue is a regression of a previous behavior. labels Oct 18, 2023

matteius added the Type: Possible Bug This issue describes a possible bug in pipenv. label Oct 31, 2023

matteius mentioned this issue Oct 22, 2024

install should not do full lock resolution, as that is the purpose of pipenv lock #6276

Merged

2 tasks

matteius closed this as completed in #6276 Oct 22, 2024

retornam mentioned this issue Nov 20, 2024

Issue with Pipfile Dependency Resolution for Git Branch and SHA Combinations #6317

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect dependency version installed when specifying a git commit. #5973

Incorrect dependency version installed when specifying a git commit. #5973

tobygomersall commented Oct 11, 2023

chrisroat commented Oct 16, 2023 •

edited

Loading

matteius commented Oct 18, 2023

matteius commented Oct 18, 2023

tobygomersall commented Oct 18, 2023

matteius commented Oct 18, 2023

chrisroat commented Oct 18, 2023

chrisroat commented Oct 31, 2023

Aiky30 commented May 1, 2024 •

edited

Loading

ohshazbot commented Oct 21, 2024

matteius commented Oct 21, 2024

ohshazbot commented Oct 21, 2024

matteius commented Oct 21, 2024

ohshazbot commented Oct 21, 2024

Incorrect dependency version installed when specifying a git commit. #5973

Incorrect dependency version installed when specifying a git commit. #5973

Comments

tobygomersall commented Oct 11, 2023

Issue description

Expected result

Actual result

Steps to replicate

My debugging efforts

chrisroat commented Oct 16, 2023 • edited Loading

matteius commented Oct 18, 2023

matteius commented Oct 18, 2023

tobygomersall commented Oct 18, 2023

matteius commented Oct 18, 2023

chrisroat commented Oct 18, 2023

chrisroat commented Oct 31, 2023

Aiky30 commented May 1, 2024 • edited Loading

ohshazbot commented Oct 21, 2024

matteius commented Oct 21, 2024

ohshazbot commented Oct 21, 2024

matteius commented Oct 21, 2024

ohshazbot commented Oct 21, 2024

chrisroat commented Oct 16, 2023 •

edited

Loading

Aiky30 commented May 1, 2024 •

edited

Loading