install should not do full lock resolution, as that is the purpose of pipenv lock #6276

matteius · 2024-10-18T22:41:16Z

The issue and fixes

refactor the install routines further to provide clarity and meet the goal of not doing full lock resolution, which was intended as we moved to 2024.x release series. See: pipenv 2024 install (behavioral refactor) #6098
Fixes pipenv install still doing full lock resolution #6267
Fixes Incorrect dependency version installed when specifying a git commit. #5973
Fixes Issue with VCS installs in 2024.1.0 -- resolver trying to resolve vcs as regular dependency #6266
Fixes Fedora 40 has dropped pipenv #6167

Probably fixes #6233

Possibly fixes #6231
Possibly fixes #6187
Possibly fixes #6068
~~Possibly fixes #6280~~

The checklist

Associated issue
A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

matteius · 2024-10-19T22:42:41Z

tests/integration/test_upgrade.py

            "colorama",
            "atomicwrites",
+            "six",


This should be acceptable because its the default non-sorting case; it is technically a behavioral change that re-installing something moves it to the bottom of the Pipfile, but I feel we get closed to reducing some complexity around the Pipfile creation, and writing in this PR -- it doesn't seem to negatively impact performance either.

matteius · 2024-10-19T23:54:08Z

tests/integration/test_install_basic.py

@@ -379,9 +379,7 @@ def test_system_and_deploy_work(pipenv_instance_private_pypi):
 @pytest.mark.basic
 @pytest.mark.install
 def test_install_creates_pipfile(pipenv_instance_pypi):
-    with pipenv_instance_pypi() as p:


I had to debug this test, and I figure may as well use this simplification that existed in the fixture to not create the pipfile in the first place.

matteius · 2024-10-19T23:54:29Z

tests/integration/test_install_twists.py

-@pytest.mark.skipif(
-    sys.version_info >= (3, 12), reason="Package does not work with Python 3.12"
-)
+@pytest.mark.install


Upgrading this package meant the test could pass on python 3.13 even

matteius · 2024-10-19T23:54:45Z

tests/integration/test_install_twists.py

 def test_install_remote_wheel_file_with_extras(pipenv_instance_pypi):
    with pipenv_instance_pypi() as p:
        c = p.pipenv(
-            "install fastapi[dev]@https://files.pythonhosted.org/packages/4e/1a/04887c641b67e6649bde845b9a631f73a7abfbe3afda83957e09b95d88eb/fastapi-0.95.2-py3-none-any.whl"
+            "install -v fastapi[standard]@https://files.pythonhosted.org/packages/c9/14/bbe7776356ef01f830f8085ca3ac2aea59c73727b6ffaa757abeb7d2900b/fastapi-0.115.2-py3-none-any.whl"


dev is no longer an extra in newer fastapi versions.

matteius · 2024-10-19T23:55:09Z

tests/integration/test_install_uri.py

@@ -143,9 +143,6 @@ def test_install_named_index_alias(pipenv_instance_private_pypi):
 @pytest.mark.index
 @pytest.mark.install
 @pytest.mark.needs_internet
-@pytest.mark.skipif(
-    sys.version_info >= (3, 12), reason="Package does not work with Python 3.12"
-)


This test was failing at one point, and I discovered that it could pass on 3.13 -- I think we had a different bug at one point that lead to this.

matteius · 2024-10-19T23:55:57Z

pipenv/utils/resolver.py

@@ -5,7 +5,7 @@
 import sys
 import tempfile
 import warnings
-from functools import cached_property, lru_cache
+from functools import lru_cache


Too many side-effects encountered in various issue reports relate to our current structure and usage of caching. I am in favor of correcting core functionality, and revisiting small gaps in performance down the road in a more thoughtful way.

matteius · 2024-10-19T23:56:15Z

pipenv/utils/dependencies.py

@@ -1132,6 +1132,11 @@ def install_req_from_pipfile(name, pipfile):
            version = ""
        req_str = f"{name}{extras_str}{version}"

+        # Handle markers before constructing InstallRequirement


AI suggested this change -- awaiting confirmation it fixes relevant issues.

matteius · 2024-10-19T23:56:44Z

pipenv/routines/update.py

@@ -164,7 +160,7 @@ def upgrade(
            which=project._which,
            project=project,
            lockfile={},
-            category="default",


This was definitely a bug in the update/upgrade path, it couldn't have worked too great for non-default categories.

matteius · 2024-10-19T23:57:00Z

pipenv/routines/sync.py

@@ -45,13 +45,11 @@ def do_sync(
    do_init(
        project,
        allow_global=system,
-        requirements_dir=requirements_dir,


Simplifying interface

tests/integration/test_install_uri.py

… goal of not doing full lock resolution, which was intended as we moved to 2024.x release series.

…prevent lock resolution failure. Remove caching properties that can become stale.

… category packages

… test passed locally without this change for some reason)

…ble vs not file installs.

…entry.

…ut the quiet flag yielding too much output

oz123

Looks good to me. Squashed some overlapping commits, before merging.

Bumps [pipenv](https://github.com/pypa/pipenv) from 2024.1.0 to 2024.2.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pypa/pipenv/releases">pipenv's releases</a>.</em></p> <blockquote> <h2>Release v2024.2.0</h2> <h2>What's Changed</h2> <ul> <li>Bump black from 23.3 to 24.3.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6112">pypa/pipenv#6112</a></li> <li>vendor in pip 24.2 by <a href="https://github.com/matteius"><code>@matteius</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6263">pypa/pipenv#6263</a></li> <li>Fix strings in virtualenv.py by <a href="https://github.com/ericbrandwein"><code>@ericbrandwein</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6268">pypa/pipenv#6268</a></li> <li>Update pipdeptree by <a href="https://github.com/oz123"><code>@oz123</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6275">pypa/pipenv#6275</a></li> <li>Fix the assumption that all distributions are named by <a href="https://github.com/TurnrDev"><code>@TurnrDev</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6272">pypa/pipenv#6272</a></li> <li>Updated logic for determining available python version string. by <a href="https://github.com/matteius"><code>@matteius</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6278">pypa/pipenv#6278</a></li> <li>Refactor do_check routine to be more modular and address concerns about quiet flag by <a href="https://github.com/matteius"><code>@matteius</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6277">pypa/pipenv#6277</a></li> <li>install should not do full lock resolution, as that is the purpose of pipenv lock by <a href="https://github.com/matteius"><code>@matteius</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6276">pypa/pipenv#6276</a></li> <li>Some improvemets in requirementslib for clarity by <a href="https://github.com/vshulcz"><code>@vshulcz</code></a> in <a href="https://redirect.github.com/pypa/pipenv/pull/6279">pypa/pipenv#6279</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/ericbrandwein"><code>@ericbrandwein</code></a> made their first contribution in <a href="https://redirect.github.com/pypa/pipenv/pull/6268">pypa/pipenv#6268</a></li> <li><a href="https://github.com/TurnrDev"><code>@TurnrDev</code></a> made their first contribution in <a href="https://redirect.github.com/pypa/pipenv/pull/6272">pypa/pipenv#6272</a></li> <li><a href="https://github.com/vshulcz"><code>@vshulcz</code></a> made their first contribution in <a href="https://redirect.github.com/pypa/pipenv/pull/6279">pypa/pipenv#6279</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/pypa/pipenv/compare/v2024.1.0...v2024.2.0">https://github.com/pypa/pipenv/compare/v2024.1.0...v2024.2.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pipenv/blob/main/CHANGELOG.md">pipenv's changelog</a>.</em></p> <blockquote> <h1>2024.2.0 (2024-10-22)</h1> <h1>Pipenv 2024.2.0 (2024-10-22)</h1> <h2>Bug Fixes</h2> <ul> <li> <h2>Features & Bug Fixes</h2> <ul> <li> <p>Refactored and simplified install routines, improving maintainability and reliability (<a href="https://redirect.github.com/pypa/pipenv/issues/6276">#6276</a>)</p> <ul> <li>Split install logic into smaller, focused functions.</li> <li>Eliminated Pipfile caching for now to prevent bugs and reduce complexity.</li> <li>Fixed edge cases with package category selection.</li> <li>Improved handling of VCS dependencies during updates, fixing when ref is a revision and not a branch.</li> </ul> </li> <li> <p>Enhanced VCS URL handling with better environment variable support (<a href="https://redirect.github.com/pypa/pipenv/issues/6276">#6276</a>)</p> <ul> <li>More reliable expansion of environment variables in Git URLs.</li> <li>Better handling of authentication components in VCS URLs.</li> <li>Improved error messaging for missing environment variables.</li> <li>Fixed issue where Git reference could be dropped during relock. <code>[#6276](pypa/pipenv#6276) <https://github.com/pypa/pipenv/issues/6276></code>_</li> </ul> </li> </ul> </li> </ul> <h2>Vendored Libraries</h2> <ul> <li>Update pipdeptree to version 2.23.4 <code>[#6275](pypa/pipenv#6275) <https://github.com/pypa/pipenv/issues/6275></code>_</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pipenv/commit/0d7160e3a0f5bb088d995be7dac4cd63b86c7c8d"><code>0d7160e</code></a> Release v2024.2.0</li> <li><a href="https://github.com/pypa/pipenv/commit/dd2f21fac27d628f6bcdd76e158bbd4cb8038a51"><code>dd2f21f</code></a> Bumped version to 2024.2.0.</li> <li><a href="https://github.com/pypa/pipenv/commit/0df373899d65dcf4ec14cfcf656c6b4dc4bbd9f8"><code>0df3738</code></a> refactor: use VCS_SCHEMES constant in unpack_url</li> <li><a href="https://github.com/pypa/pipenv/commit/30930e0e6959bc37fa4d513cf6ea932ace7aa7c9"><code>30930e0</code></a> chore: some improvements</li> <li><a href="https://github.com/pypa/pipenv/commit/b872d0b3fd23ba688debe3f7bcbcb61f673749b7"><code>b872d0b</code></a> Merge pull request <a href="https://redirect.github.com/pypa/pipenv/issues/6276">#6276</a> from pypa/issue-6267</li> <li><a href="https://github.com/pypa/pipenv/commit/526ed94a63db2f00d87715d6dce39f55c777e8f1"><code>526ed94</code></a> Refactor do_check routine to be more modular and address concerns about the q...</li> <li><a href="https://github.com/pypa/pipenv/commit/5268bdecbab488ce489a08f5c86e664db1e60994"><code>5268bde</code></a> Updated logic for determining available python version string.</li> <li><a href="https://github.com/pypa/pipenv/commit/8977f13c19f0361f2f92a46e42218efe27a39f82"><code>8977f13</code></a> fix import</li> <li><a href="https://github.com/pypa/pipenv/commit/c74f9a60e10fb6861d61c1147980f21edc5fe88c"><code>c74f9a6</code></a> Clean up lint, add news fragment and address issue 6167</li> <li><a href="https://github.com/pypa/pipenv/commit/b2c094b1c48cdd77ddb01fcf52a5ee1520718f8e"><code>b2c094b</code></a> fix VCS environment variable expansion</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pipenv/compare/v2024.1.0...v2024.2.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pipenv&package-manager=pip&previous-version=2024.1.0&new-version=2024.2.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

matteius force-pushed the issue-6267 branch 3 times, most recently from dab8dc0 to 359df03 Compare October 19, 2024 22:05

matteius mentioned this pull request Oct 19, 2024

pipenv fails to parse platform-specific requirement in Pipfile when using a versioning scheme #6231

Closed

matteius force-pushed the issue-6267 branch from d6299f4 to 5c1af20 Compare October 19, 2024 22:40

matteius commented Oct 19, 2024

View reviewed changes

This was referenced Oct 19, 2024

Pipfile with private index AND optional dependency doesn't work #6187

Closed

pipenv (patched pip) corrupts path for editable requirement on Windows #6068

Closed

matteius requested a review from oz123 October 19, 2024 22:54

This was referenced Oct 19, 2024

Dependency resolution error upgrading a package by a patch version with no changes to dependencies #6154

Closed

Change where we cache the package finder to alleviate issue collecting hashes with multiple indexes. #6252

Closed

matteius commented Oct 19, 2024

View reviewed changes

This was referenced Oct 20, 2024

Private packages not updated unless environment is removed #5791

Open

Incorrect dependency version installed when specifying a git commit. #5973

Closed

matteius commented Oct 22, 2024

View reviewed changes

tests/integration/test_install_uri.py Outdated Show resolved Hide resolved

matteius added 4 commits October 22, 2024 19:49

refactor the install routines further to provide clairty and meet the…

f79cd18

… goal of not doing full lock resolution, which was intended as we moved to 2024.x release series.

Handle the removal from pipfile

15d07f7

Eliminate bugs/complexity of pipfile caching

f292a19

add more careful logic to remove conflicting entries from Pipfile to …

c2385ab

…prevent lock resolution failure. Remove caching properties that can become stale.

matteius and others added 19 commits October 22, 2024 19:49

fix test assumption about not-sorting.

190e629

refine var name and fix issue with update command installing to wrong…

d63e77e

… category packages

fix upgrade url/file dependencies

fe5c4d1

Fix issue with upgrade path hard coding the category default

8515236

Fix edge case of test in CI by ensuring it raises DeployFailure (this…

ef1f4e8

… test passed locally without this change for some reason)

Actually that didn't fix that test

46c5db7

PR cleanup

287d848

eliminate egg fragment as of pip 22.2; also correct behavior of edita…

95e6c87

…ble vs not file installs.

More cleanup

a10cf7e

Try solving for issue where vcs ref is getting dropped from resolver …

da53b47

…entry.

solving for issue where vcs ref is getting dropped from resolver entry.

cf842b0

Fix lint

089bc2d

fix vcs updates commit hash when relocking

1663b3b

fix test that was never run because of name collision

95ab685

fix VCS environment variable expansion

b2c094b

Clean up lint, add news fragment and address issue 6167

c74f9a6

fix import

8977f13

Updated logic for determining available python version string.

5268bde

Refactor do_check routine to be more modular and address concerns abo…

526ed94

…ut the quiet flag yielding too much output

oz123 approved these changes Oct 22, 2024

View reviewed changes

oz123 force-pushed the issue-6267 branch from acbf199 to 526ed94 Compare October 22, 2024 17:57

matteius merged commit b872d0b into main Oct 22, 2024
22 checks passed

matteius deleted the issue-6267 branch October 22, 2024 18:27

matteius added the hacktoberfest-accepted label Oct 26, 2024

retornam mentioned this pull request Nov 20, 2024

Issue with Pipfile Dependency Resolution for Git Branch and SHA Combinations #6317

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

install should not do full lock resolution, as that is the purpose of pipenv lock #6276

install should not do full lock resolution, as that is the purpose of pipenv lock #6276

matteius commented Oct 18, 2024 •

edited

Loading

matteius Oct 19, 2024

matteius Oct 19, 2024

matteius Oct 19, 2024

matteius Oct 19, 2024

matteius Oct 19, 2024

matteius Oct 19, 2024

matteius Oct 19, 2024

matteius Oct 19, 2024

matteius Oct 19, 2024

oz123 left a comment

install should not do full lock resolution, as that is the purpose of pipenv lock #6276

install should not do full lock resolution, as that is the purpose of pipenv lock #6276

Conversation

matteius commented Oct 18, 2024 • edited Loading

The issue and fixes

The checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oz123 left a comment

Choose a reason for hiding this comment

matteius commented Oct 18, 2024 •

edited

Loading