fix a case of duplicate torch packages when using conflicting extras
#11323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The snapshot is too big to meaningfully read, but the problem is
in the dependencies of `torchmetrics`:
[[package]]
name = "torchmetrics"
version = "1.6.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "lightning-utilities" },
{ name = "numpy" },
{ name = "packaging" },
{ name = "torch", version = "2.2.1", source = { registry = "https://pypi.org/simple" } },
{ name = "torch", version = "2.5.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-4-test-chgnet' or extra != 'extra-4-test-m3gnet'" },
]
The conflict markers here are overlapping, which means
both can be included in the same environment.
The underlying cause here, I believe, was that we weren't accounting for the case where an edge could be visited *without* any extras enabled. Because of that, we got into situations where we thought there was only one path to an edge when there were actually more paths. This in turn lead to us erroneously doing simplification where it actually isn't justified. And in turn lead to duplicate versions of the same package being installed in the same environment. The fix for this ends up being really simple: in the case where we don't add any conflict items for a package during graph traversal, we materialize an empty set of conflicts to mark the case of no extras being enabled when visiting the child edges. This is enough to propagate the knowledge of multiple paths to the same edge and causes us to avoid doing improper simplifications. This does fix the problem in the snapshot, but it does also I think lead to other cases where simplifications are no longer possible (hence the changes to the airflow snapshot). But this seems expected, since we are doing strictly less simplification than we were before. It's unclear if all of those cases were actual bugs or not though.
charliermarsh
approved these changes
Feb 8, 2025
| { name = "typing-extensions" }, | ||
| { name = "unicodecsv" }, | ||
| { name = "werkzeug" }, | ||
| { name = "alembic", marker = "extra == 'extra-3-pkg-x1'" }, |
There was a problem hiding this comment.
I'm having some trouble figuring out why the 2.6.0 version of Airflow doesn't have these on it... I guess it's because that's the "default" version that gets installed if no extras are enabled?
There was a problem hiding this comment.
Yeah I think so. There is a non-optional dependency on quickpath-airflow-operator, and the lock entry for that package is this:
[[package]]
name = "quickpath-airflow-operator"
version = "1.0.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "apache-airflow", version = "2.5.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-3-pkg-x1'" },
{ name = "apache-airflow", version = "2.6.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-3-pkg-x2' or extra != 'extra-3-pkg-x1'" },
]I think that conflict marker on 2.6.0 could be further simplified to extra != 'extra-3-pkg-x1'. (A good example of how conflict marker simplification is not perfect.)
tmeijn
pushed a commit
to tmeijn/dotfiles
that referenced
this pull request
Feb 11, 2025
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.29` -> `0.5.30` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.5.30`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0530) [Compare Source](astral-sh/uv@0.5.29...0.5.30) ##### Python The managed PyPy distributions have been updated for PyPy v7.3.18, which includes: - PyPy3.10, which updates the standard library from Python 3.10.14 to 3.10.19 - PyPy3.11, which adds beta support for Python 3.11.11 See the [PyPy release](https://pypy.org/posts/2025/02/pypy-v7318-release.html) for more details. ##### Enhancements - Add `uv sync --dry-run` ([#​11299](astral-sh/uv#11299)) - Ignore `#egg` fragment in HTML Simple API response ([#​11340](astral-sh/uv#11340)) ##### Configuration - Add `NO_BINARY` and `NO_BINARY_PACKAGE` environment variables ([#​11399](astral-sh/uv#11399)) ##### Performance - Avoid re-cloning name when populating ambiguous set ([#​11401](astral-sh/uv#11401)) - Optimize flattening in large workspaces ([#​11313](astral-sh/uv#11313)) ##### Bug fixes - Allow dynamic packages to be overloaded ([#​11400](astral-sh/uv#11400)) - Fix credential caching for index roots when URL ends in `simple/` ([#​11336](astral-sh/uv#11336)) - Fix marker merging for requirements.txt for psycopg ([#​11298](astral-sh/uv#11298)) - Set 777 permissions on locked files ([#​11328](astral-sh/uv#11328)) - Support extras in `@` requests for tools ([#​11335](astral-sh/uv#11335)) - Upgrade `astral-tokio-tar` to v0.5.1 ([#​11359](astral-sh/uv#11359)) - Avoid missing logging for no-op upgrade events ([#​11301](astral-sh/uv#11301)) - Use refined specifiers when logging narrowed Python range ([#​11334](astral-sh/uv#11334)) - Don't use popup-generating `eprintln` in trampoline warnings ([#​11295](astral-sh/uv#11295)) - Patch pkg-config files to be relocatable ([#​11291](astral-sh/uv#11291)) - Fix a case of duplicate `torch` packages when using conflicting extras ([#​11323](astral-sh/uv#11323)) ##### Documentation - Add docs for `uv tool install --editable` ([#​11280](astral-sh/uv#11280)) - Fix broken anchors in README and docs index ([#​11338](astral-sh/uv#11338)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xNjQuMSIsInVwZGF0ZWRJblZlciI6IjM5LjE2NC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
2 tasks
BurntSushi
added
bug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The underlying cause here, I believe, was that we weren't accounting
for the case where an edge could be visited without any extras
enabled. Because of that, we got into situations where we thought there
was only one path to an edge when there were actually more paths. This
in turn lead to us erroneously doing simplification where it actually
isn't justified. And in turn lead to duplicate versions of the same
package being installed in the same environment.
The fix for this ends up being really simple: in the case where we
don't add any conflict items for a package during graph traversal,
we materialize an empty set of conflicts to mark the case of no
extras being enabled when visiting the child edges. This is enough to
propagate the knowledge of multiple paths to the same edge and causes
us to avoid doing improper simplifications.
This does fix the problem in the snapshot, but it does also I think
lead to other cases where simplifications are no longer possible (hence
the changes to the airflow snapshot). But this seems expected, since
we are doing strictly less simplification than we were before. It's
unclear if all of those cases were actual bugs or not though.
The first commit in this PR adds a regression test so that reviewers
can more easily see the diff to the lock file as a result of the fix.
Fixes #11133