Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alternative logic for solver creation tasks #270

Merged
merged 52 commits into from
Sep 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
6235ec9
start branch
jaimergp Aug 24, 2023
5114824
trigger ci
jaimergp Aug 24, 2023
9fe2743
port logic from conda/conda#9614
jaimergp Aug 25, 2023
19c24fe
deselect a couple tests (accelerate and libmamba do not get along)
jaimergp Aug 25, 2023
cb3dc80
add news
jaimergp Aug 25, 2023
1c385eb
Apply suggestions from code review
jaimergp Aug 28, 2023
14061ed
Alternative logic for solver creation tasks
jaimergp Aug 28, 2023
5bb867e
Merge branch 'main' into investigate-test-prune-errors
jezdez Aug 29, 2023
ea64379
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 29, 2023
fc1530c
fix tests/core/test_solve.py::test_auto_update_conda
jaimergp Aug 29, 2023
15486b0
some progress with pins (but not quite there yet)
jaimergp Aug 29, 2023
889fe15
Increase timeouts in CI env setup
jaimergp Aug 29, 2023
e3e0b9a
Merge branch 'main' of github.com:conda/conda-libmamba-solver into re…
jaimergp Aug 29, 2023
74ff234
Merge branch 'investigate-test-prune-errors' of github.com:conda/cond…
jaimergp Aug 29, 2023
e40f0da
allow uninstall by default only on removals
jaimergp Aug 29, 2023
3073a78
handle prune
jaimergp Aug 29, 2023
49f201a
avoid crash
jaimergp Aug 29, 2023
c5cdb04
cleanup
jaimergp Aug 29, 2023
e5b3e75
only lock the installed spec
jaimergp Aug 29, 2023
2395860
pre-commit
jaimergp Aug 29, 2023
ebc2885
add SolverInputState.always_update helper mapping
jaimergp Aug 29, 2023
ba202e1
allow uninstall of conflicts
jaimergp Aug 29, 2023
10a67f8
use always_update list for simplicity
jaimergp Aug 29, 2023
122f835
not update-all if pinned
jaimergp Aug 29, 2023
92f3314
explain pins if involved in errors
jaimergp Aug 29, 2023
81d350d
re-enable original test_neutering_of_historic_specs
jaimergp Aug 29, 2023
629f381
bump minimum required conda and libmamba versions
jaimergp Aug 29, 2023
c41a930
fix tests/core/test_solve.py::test_force_remove_1 and tests/core/test…
jaimergp Aug 30, 2023
07dc6d1
Re-enable tests that are failing
costrouc Sep 8, 2023
a39d1d1
Require libmambapy >= 1.5.1
jaimergp Sep 12, 2023
2c7fa21
pre-commit
jaimergp Sep 12, 2023
03d0e56
amend news
jaimergp Sep 12, 2023
32c4435
Update __init__.py
jaimergp Sep 12, 2023
4c2bdc3
fix test_neutering_of_historic_specs
jaimergp Sep 12, 2023
307c0a0
consider 1.5.1 in warnings too
jaimergp Sep 12, 2023
466182d
pre-commit
jaimergp Sep 12, 2023
5af3deb
adjust test
jaimergp Sep 12, 2023
0f49202
Merge remote-tracking branch 'origin/feat-reenable-tests' into revamp…
jaimergp Sep 12, 2023
7ee5f9f
remove comment
jaimergp Sep 12, 2023
86375e2
fix json serialization
jaimergp Sep 13, 2023
2fab449
xfail in 1.5.1 too
jaimergp Sep 13, 2023
2cccbc6
catch RuntimeError for the whole 1.5.x series
jaimergp Sep 13, 2023
24edbaf
xfail test_explicit
jaimergp Sep 13, 2023
225f5c0
fix test_pinned_1
jaimergp Sep 13, 2023
40d5917
lock installed name-only pinned packages
jaimergp Sep 13, 2023
2533748
do not add installed channels to list if --override-channels in use
jaimergp Sep 13, 2023
efffa95
add adjusted variant for test_priority
jaimergp Sep 13, 2023
67417ce
always define lock
jaimergp Sep 13, 2023
47843a1
explain some deselected tests
jaimergp Sep 13, 2023
9e17e10
adjust test_install_features
jaimergp Sep 13, 2023
8e300d8
xfail test_export
jaimergp Sep 13, 2023
1aeaf0a
pre-commit
jaimergp Sep 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/upstream_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ jobs:
- name: Setup environment
working-directory: conda # CONDA-LIBMAMBA-SOLVER CHANGE
shell: bash -el {0}
timeout-minutes: 10
timeout-minutes: 15
run: |
# CONDA-LIBMAMBA-SOLVER CHANGE
cat ../conda-libmamba-solver/dev/requirements.txt >> tests/requirements.txt
Expand Down
2 changes: 2 additions & 0 deletions conda_libmamba_solver/mamba_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
# 2022.11.14: only keeping channel prioritization and context initialization logic now

import logging
from functools import lru_cache
from importlib.metadata import version
from typing import Dict

Expand All @@ -19,6 +20,7 @@
log = logging.getLogger(f"conda.{__name__}")


@lru_cache(maxsize=1)
def mamba_version():
return version("libmambapy")

Expand Down
4 changes: 3 additions & 1 deletion conda_libmamba_solver/repoquery.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@ def configure_parser(parser: argparse.ArgumentParser):
view_grp.add_argument(
"-t", "--tree", action="store_true", help="Show dependencies in a tree-like format."
)
view_grp.add_argument("--recursive", action="store_true", help="Show dependencies recursively.")
view_grp.add_argument(
"--recursive", action="store_true", help="Show dependencies recursively."
)

subparser = parser.add_subparsers(dest="subcmd")

Expand Down
295 changes: 179 additions & 116 deletions conda_libmamba_solver/solver.py

Large diffs are not rendered by default.

51 changes: 46 additions & 5 deletions conda_libmamba_solver/state.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,8 @@ def __init__(
self._update_modifier = self._default_to_context_if_null(
"update_modifier", update_modifier
)
if prune and self._update_modifier == UpdateModifier.FREEZE_INSTALLED:
self._update_modifier = UpdateModifier.UPDATE_SPECS # revert to default
self._deps_modifier = self._default_to_context_if_null("deps_modifier", deps_modifier)
self._ignore_pinned = self._default_to_context_if_null("ignore_pinned", ignore_pinned)
self._force_remove = self._default_to_context_if_null("force_remove", force_remove)
Expand Down Expand Up @@ -282,6 +284,24 @@ def aggressive_updates(self) -> Mapping[str, MatchSpec]:
"""
return MappingProxyType(self._aggressive_updates)

@property
def always_update(self) -> Mapping[str, MatchSpec]:
"""
Merged lists of packages that should always be updated, depending on the flags, including:
- aggressive_updates
- conda if auto_update_conda is true and we are on the base env
- almost all packages if update_all is true
- etc
"""
pkgs = {pkg: MatchSpec(pkg) for pkg in self.aggressive_updates if pkg in self.installed}
if context.auto_update_conda and paths_equal(self.prefix, context.root_prefix):
pkgs.setdefault("conda", MatchSpec("conda"))
if self.update_modifier.UPDATE_ALL:
for pkg in self.installed:
if pkg != "python" and pkg not in self.pinned:
pkgs.setdefault(pkg, MatchSpec(pkg))
return MappingProxyType(pkgs)

@property
def do_not_remove(self) -> Mapping[str, MatchSpec]:
"""
Expand Down Expand Up @@ -443,6 +463,8 @@ class SolverOutputState:
If a solve attempt is not successful, conflicting specs are kept here for further
relaxation of the version and build constrains. If not provided, their default value is a
blank mapping.
pins
Packages that ended up being pinned. Mostly used for reporting and debugging.

Notes
-----
Expand Down Expand Up @@ -476,6 +498,7 @@ def __init__(
for_history: Optional[Mapping[str, MatchSpec]] = None,
neutered: Optional[Mapping[str, MatchSpec]] = None,
conflicts: Optional[Mapping[str, MatchSpec]] = None,
pins: Optional[Mapping[str, MatchSpec]] = None,
):
self.solver_input_state: SolverInputState = solver_input_state

Expand Down Expand Up @@ -514,15 +537,24 @@ def __init__(
"conflicts", data=(conflicts or {}), reason="From arguments"
)

self.pins: Mapping[str, MatchSpec] = TrackedMap(
"pins", data=(pins or {}), reason="From arguments"
)

def _initialize_specs_from_input_state(self):
"""
Provide the initial value for the ``.specs`` mapping. This depends on whether
there's a history available (existing prefix) or not (new prefix).
"""
# Initialize specs following conda.core.solve._collect_all_metadata()

# First initialization depends on whether we have a history to work with or not
if self.solver_input_state.history:
if self.solver_input_state.prune:
pass # we do not initialize specs with history OR installed pkgs if we are pruning
# Otherwise, initialization depends on whether we have a history to work with or not
elif (
self.solver_input_state.history
and not self.solver_input_state.update_modifier.UPDATE_ALL
):
# add in historically-requested specs
self.specs.update(self.solver_input_state.history, reason="As in history")
for name, record in self.solver_input_state.installed.items():
Expand Down Expand Up @@ -555,7 +587,7 @@ def _initialize_specs_from_input_state(self):
# add everything in prefix if we have no history to work with (e.g. with --update-all)
self.specs.update(
{name: MatchSpec(name) for name in self.solver_input_state.installed},
reason="Installed and no history available",
reason="Installed and no history available (prune=false)",
)

# Add virtual packages so they are taken into account by the solver
Expand Down Expand Up @@ -664,7 +696,7 @@ def _prepare_for_add(self, index: IndexHelper):
)
else:
# every other spec that matches something installed will be configured with
# only a target This is the case for conflicts, among others
# only a target. This is the case for conflicts, among others
self.specs.set(
name, MatchSpec(name, target=record.dist_str()), reason="Spec matches record"
)
Expand Down Expand Up @@ -765,6 +797,13 @@ def _prepare_for_add(self, index: IndexHelper):
reason="Update all, with history: treat pip installed "
"stuff as explicitly installed",
)
elif name not in self.specs:
self.specs.set(
name,
MatchSpec(name),
reason="Update all, with history: "
"adding name-only spec from installed",
)
else:
for name in sis.installed:
if name in sis.pinned:
Expand Down Expand Up @@ -1001,7 +1040,9 @@ def post_solve(self, solver: Type["Solver"]):
history_spec = sis.history.get(name)
if history_spec and spec.strictness < history_spec.strictness:
self.neutered.set(
name, spec, reason="Spec needs less strict constrains than history"
name,
MatchSpec(name, version=history_spec.get("version")),
reason="Spec needs less strict constrains than history",
)

# ## Add inconsistent packages back ###
Expand Down
52 changes: 24 additions & 28 deletions dev/collect_upstream_conda_tests/collect_upstream_conda_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,12 @@
# Features / nomkl involved
"test_features_solve_1",
"test_prune_1",
# TODO: These ones need further investigation
"test_channel_priority_churn_minimized",
"test_update_prune_2",
"test_update_prune_3",
# Message expected, but libmamba does not report constraints
"test_update_prune_5",
# classic expects implicit update to channel with higher priority, including downgrades
# libmamba does not do this, it just stays in the same channel; should it change?
"test_priority_1",
# The following are known to fail upstream due to too strict expectations
# We provide the same tests with adjusted checks in tests/test_modified_upstream.py
Expand All @@ -41,15 +45,13 @@
"test_downgrade_python_prevented_with_sane_message",
],
"tests/test_create.py": [
# libmamba does not support features
"test_remove_features",
# Inconsistency analysis not implemented yet
"test_conda_recovery_of_pip_inconsistent_env",
# Known bug in mamba; see https://github.com/mamba-org/mamba/issues/1197
"test_offline_with_empty_index_cache",
"test_neutering_of_historic_specs",
# Adjusted in tests/test_modified_upstream.py
"test_install_features",
"test_pinned_override_with_explicit_spec",
# TODO: Investigate why this fails on Windows now
"test_install_update_deps_only_deps_flags",
# TODO: https://github.com/conda/conda-libmamba-solver/issues/141
"test_conda_pip_interop_conda_editable_package",
],
Expand Down Expand Up @@ -86,27 +88,16 @@
"tests/conda_env/specs/test_requirements.py": [
"TestRequirements::test_environment",
],
# TODO: Known to fail; should be fixed by
# https://github.com/conda/conda-libmamba-solver/pull/242
# Added to test_modified_upstream.py
"tests/test_priority.py": ["test_reorder_channel_priority"],
}

_broken_by_libmamba_1_4_2 = {

_broken_by_libmamba_1_5_x = {
# conda/tests
"tests/core/test_solve.py": [
"test_force_remove_1",
"test_aggressive_update_packages",
"test_update_deps_2",
],
"tests/test_create.py": [
"test_list_with_pip_wheel",
"test_conda_pip_interop_dependency_satisfied_by_pip", # Linux-only
"test_conda_pip_interop_pip_clobbers_conda", # Linux-only
"test_install_tarball_from_local_channel", # Linux-only
],
# conda-libmamba-solver/tests
"tests/test_modified_upstream.py": [
"test_pinned_1",
"tests/test_export.py": [
"test_explicit",
"test_export",
],
}

Expand All @@ -126,10 +117,15 @@ def pytest_collection_modifyitems(session, config, items):
if item_name_no_brackets in _deselected_upstream_tests.get(path_key, []):
deselected.append(item)
continue
if version(
"libmambapy"
) >= "1.4.2" and item_name_no_brackets in _broken_by_libmamba_1_4_2.get(path_key, []):
item.add_marker(pytest.mark.xfail(reason="Broken by libmamba 1.4.2; see #186"))
if version("libmambapy").startswith(
"1.5."
) and item_name_no_brackets in _broken_by_libmamba_1_5_x.get(path_key, []):
item.add_marker(
pytest.mark.xfail(
reason="Broken in libmamba 1.5.x; "
"see https://github.com/mamba-org/mamba/issues/2431."
)
)
selected.append(item)
items[:] = selected
config.hook.pytest_deselected(items=deselected)
6 changes: 3 additions & 3 deletions dev/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
pip
# run-time
boltons>=23.0.0
conda>=23.3.1
libmamba>=1.4.1
libmambapy>=1.4.1
conda>=23.7.3
libmamba>=1.5.1
libmambapy>=1.5.1
# be explicit about sqlite because sometimes it's removed from the env :shrug:
sqlite
22 changes: 22 additions & 0 deletions news/270-tasks-and-prune
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
### Enhancements

* Rewrite how we create tasks for `libsolv`, making use of `libmamba`'s `add_pin` features. (#270)

### Bug fixes

* Port logic from [conda/conda#9614](https://github.com/conda/conda/pull/9614), which fixes
a bug where the `--prune` flag was not working correctly in `conda env update` commands.
(#270)
* Ensure environments are not aggressively updated to higher priority channels under some conditions. (#240 via #270)

### Deprecations

* <news item>

### Docs

* <news item>

### Other

* <news item>
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ classifiers = [
]
requires-python = ">=3.8"
dependencies = [
"conda >=23.5.0",
"libmambapy >=1.4.1",
"conda >=23.7.3",
"libmambapy >=1.5.1",
"boltons >=23.0.0",
]
dynamic = [
Expand Down
4 changes: 2 additions & 2 deletions recipe/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ requirements:
- hatch-vcs
run:
- python >=3.8
- conda >=23.5.0
- libmambapy >=1.4.1
- conda >=23.7.3
- libmambapy >=1.5.1
- boltons >=23.0.0

test:
Expand Down
Loading
Loading