Deselected upstream tests

[jaimergp]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

Summary

We do not run some upstream tests for two main reasons:

• Test targets behaviour not implemented or implementable in conda-libmamba-solver
• Test failure couldn't be explained but it's not critical. Needs further assessment.

This is done in pyproject.toml via --deselect flags that are then used to overwrite conda/conda's configuration.

This is the current list (to be updated)

Linked Issues & PRs

Conflict report / analysis is done differently with libmamba.

• tests/cli/test_cli_install.py::test_find_conflicts_called_once

SolverStateContainer needed

• tests/core/test_solve.py::test_solve_2
• tests/core/test_solve.py::test_virtual_package_solver
• tests/core/test_solve.py::test_broken_install

features / nomkl involved

• tests/core/test_solve.py::test_features_solve_1
• tests/core/test_solve.py::test_prune_1
• tests/test_create.py::IntegrationTests::test_remove_features

Inconsistency analysis not implemented yet

• tests/test_create.py::IntegrationTests::test_conda_recovery_of_pip_inconsistent_env

Known bug in mamba; see mamba-org/mamba#1197

• tests/test_create.py::IntegrationTests::test_offline_with_empty_index_cache

The following are known to fail upstream due to too strict expectations.
We provide the same tests with adjusted checks in tests/test_modified_upstream.py:

• tests/test_create.py::IntegrationTests::test_neutering_of_historic_specs
• tests/test_create.py::IntegrationTests::test_pinned_override_with_explicit_spec
• tests/core/test_solve.py::test_pinned_1
• tests/core/test_solve.py::test_freeze_deps_1
• tests/core/test_solve.py::test_cuda_fail_1
• tests/core/test_solve.py::test_cuda_fail_2
• tests/core/test_solve.py::test_update_all_1
• tests/core/test_solve.py::test_conda_downgrade
• tests/core/test_solve.py::test_python2_update
• tests/core/test_solve.py::test_fast_update_with_update_modifier_not_set
• tests/core/test_solve.py::test_downgrade_python_prevented_with_sane_message

These use libmamba-incompatible MatchSpecs (name[build_number=1] syntax)

• tests/models/test_prefix_graph.py::test_deep_cyclical_dependency

See #133 (comment)
These failed after enabling the whole unit test suite for conda/conda.
Errors are not critical but would require some further assessment in case fixes are obvious:

• tests/cli/test_main_notices.py::test_notices_appear_once_when_running_decorated_commands
• tests/cli/test_main_notices.py::test_notices_does_not_interrupt_command_on_failure
• tests/conda_env/installers/test_pip.py::PipInstallerTest::test_stops_on_exception
• tests/conda_env/installers/test_pip.py::PipInstallerTest::test_straight_install
• tests/conda_env/specs/test_base.py::DetectTestCase::test_build_msg
• tests/conda_env/specs/test_base.py::DetectTestCase::test_dispatches_to_registered_specs
• tests/conda_env/specs/test_base.py::DetectTestCase::test_has_build_msg_function
• tests/conda_env/specs/test_base.py::DetectTestCase::test_passes_kwargs_to_all_specs
• tests/conda_env/specs/test_base.py::DetectTestCase::test_raises_exception_if_no_detection

TODO: Fix upstream; they seem to assume no other solvers will be active via env var

• tests/plugins/test_solvers.py::test_get_solver_backend
• tests/plugins/test_solvers.py::test_get_solver_backend_multiple

TODO: Investigate these, since they are solver related-ish

• tests/conda_env/specs/test_requirements.py::TestRequiremets::test_environment
• tests/models/test_prefix_graph.py::test_windows_sort_orders_1

TODO: These ones need further investigation

• tests/core/test_solve.py::test_channel_priority_churn_minimized
• tests/core/test_solve.py::test_priority_1

TODO: Investigate why this fails on Windows now

• tests/test_create.py::IntegrationTests::test_install_update_deps_only_deps_flags

TODO:

• tests/test_create.py::IntegrationTests::test_conda_pip_interop_conda_editable_package
  • Deselected upstream tests #143 (comment)

These are not deselected but we might need to at some point. Tracked in #186.

• tests/core/test_solve.py::test_force_remove_1 (different package versions)
• tests/core/test_solve.py::test_aggressive_update_packages (different package versions)
• tests/core/test_solve.py::test_update_deps_2 (Unsatisfiable error)

[jaimergp]

Originally posted by @costrouc in #141

This issue came up in #133. There is an upstream conda test tests/test_create.py::IntegrationTests::test_conda_pip_interop_conda_editable_package which fails.

This issue can be reproduced via

docker run --rm -it \
       -v /home/costrouc/p/conda/conda:/opt/conda-src \
       -v /home/costrouc/p/conda/conda-libmamba-solver:/opt/conda-libmamba-solver-src \
       -e TEST_SPLITS=3 \
       -e TEST_GROUP=1 \
       -e CONDA_SOLVER=libmamba \
       ghcr.io/conda/conda-ci:main-linux-python3.10-conda-forge \
       bash

Then running

source /opt/conda-libmamba-solver/dev/bashrc_linux.sh
cd /opt/conda-src
pytest tests/test_create.py::IntegrationTests::test_conda_pip_interop_conda_editable_package

Getting the following error

________ IntegrationTests.test_conda_pip_interop_conda_editable_package ________
Traceback (most recent call last):
  File "/opt/conda-src/tests/test_create.py", line 1688, in test_conda_pip_interop_conda_editable_package
    stdout, stderr, _ = run_command(Commands.INSTALL, prefix, "urllib3=1.20", "--dry-run")
  File "/opt/conda-src/conda/testing/integration.py", line 287, in run_command
    result = do_call(args, p)
  File "/opt/conda-src/conda/cli/conda_argparse.py", line 92, in do_call
    return getattr(module, func_name)(args, parser)
  File "/opt/conda-src/conda/notices/core.py", line 121, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda-src/conda/cli/main_install.py", line 20, in execute
    install(args, parser, 'install')
  File "/opt/conda-src/conda/cli/install.py", line 264, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/opt/conda-src/conda/core/solve.py", line 130, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
  File "/opt/conda-src/conda/core/solve.py", line 182, in solve_for_diff
    raise RuntimeError("Cannot unlink unmanageable packages:%s"
RuntimeError: Cannot unlink unmanageable packages:
  - <develop>/pypi::urllib3-1.19.1-dev_0

Detailed logs here

Diving into the exact test which fails here

                # should raise an error
                with pytest.raises(PackagesNotFoundError):
                    # TODO: This raises PackagesNotFoundError, but the error should really explain
                    #       that we can't install urllib3 because it's already installed and
                    #       unmanageable. The error should suggest trying to use pip to uninstall it.
                    stdout, stderr, _ = run_command(Commands.INSTALL, prefix, "urllib3=1.20", "--dry-run")

The expect conda behaviour here assuming pip interop which is not an officially supported feature is to remove the urllib3=1.20 spec from being requested by the user since it is controlled by pip. Correctly an error is being thrown but it is not the one that conda expects. It looks as though libmamba-solver does attempt to recommend updating the pip version using conda-forge but conda catches this as an error after the final solve. See https://github.com/conda/conda/blob/main/conda/core/solve.py#L175-L184. Leaving this detailed issue so that in the future if pip interop is more explored we have a good starting point.

[jaimergp]

The deselected tests will be handled a pytest plugin, available in dev/.

[jaimergp]

I'll close this issue because the pytest collection plugin has the same info in code and more up-to-date. Otherwise we would periodically need to update this issue and that's prone to errors.