Fixing recent CI failures since 23.3.0 release

[costrouc]

Investigates #186

There are several failures in CI that occurred at the same time this is a PR to resolve those issues.

[costrouc]

@jaimergp and @jezdez is it appropriate to add this line since I checked the git blame history and @jaimergp did make modifications after the QuantStack/Mamba work. I also excluded this file since it was causing precommit to fail.

[jaimergp]

I guess it's ok :)

[costrouc]

Diving into one of the CI tests which is failing

CONDA_SOLVER=libmamba pytest /opt/conda-src/tests/core/test_solve.py::test_force_remove_1

I can verfiy this passes without the conda solver being specified and using classic. This test looks like it is failing for older versions of conda-libmamba-solver ... leading me to think this may have to do with libmamba. Looking into.

The versions of things:

conda origin/main
conda-libmamba-solver origin/main
libmamba                  1.4.2                hcea66bb_0    conda-forge
libmambapy                1.4.2           py310h1428755_0    conda-forge

I tried the test with libmamba 1.4.1 and it passes. Here is my environment now that I can sudo /opt/conda/bin/conda install libmambapy==1.4.1 --solver=libmamba.

$ conda info
     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/test_user/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 0.0.0.dev0+placeholder
    conda-build version : 3.24.0
         python version : 3.10.10.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.31=0
                          __linux=6.1.15=0
                          __unix=0=0
       base environment : /opt/conda  (read only)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /home/test_user/.conda/pkgs
       envs directories : /home/test_user/.conda/envs
                          /opt/conda/envs
               platform : linux-64
             user-agent : conda/0.0.0.dev0+placeholder requests/2.28.2 CPython/3.10.10 Linux/6.1.15 debian/11 glibc/2.31
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False

(base) test_user@2eb23210041c:/opt/conda-src$ conda config --set solver libmamba
(base) test_user@2eb23210041c:/opt/conda-src$ conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/test_user/.condarc
 populated config files : /opt/conda/.condarc
                          /home/test_user/.condarc
          conda version : 0.0.0.dev0+placeholder
    conda-build version : 3.24.0
         python version : 3.10.10.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.31=0
                          __linux=6.1.15=0
                          __unix=0=0
       base environment : /opt/conda  (read only)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /home/test_user/.conda/pkgs
       envs directories : /home/test_user/.conda/envs
                          /opt/conda/envs
               platform : linux-64
             user-agent : conda/0.0.0.dev0+placeholder requests/2.28.2 CPython/3.10.10 Linux/6.1.15 debian/11 glibc/2.31 solver/libmamba conda-libmamba-solver/23.3.1.dev8+g006b977 libmambapy/1.4.1
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False

Going to repeat for testing purposes. I may restrict in CI the libmamba version for testing purposes.

Tested several times and the tests indeed pass in 1.4.1 libmamba but not 1.4.2.

Between commits:

• mamba-org/mamba@30ca2e2 1.4.2
• mamba-org/mamba@6ed35f8 1.4.1

Compare mamba-org/mamba@libmambapy-1.4.1...libmambapy-1.4.2

[costrouc]

I can reproduce

CONDA_SOLVER=libmamba pytest tests/cli/test_main_clean.py::test_clean_all_mock_lstat

Getting the following error

FAILED tests/cli/test_main_clean.py::test_clean_all_mock_lstat[False] - AssertionError: assert 'WARNING:' in ''

These are the lines https://github.com/conda/conda/blob/main/tests/cli/test_main_clean.py#L326-L327

[costrouc]

I cannot reproduce

CONDA_SOLVER=libmamba pytest tests/core/test_subdir_data.py::test_state_is_not_json

or

CONDA_SOLVER=libmamba pytest tests/test_activate.py::ShellWrapperIntegrationTests::test_dash_basic_integration

[costrouc]

So to summarize:

• several tests are failing due to libmambapy 1.4.1 -> 1.4.2 see Fixing recent CI failures since 23.3.0 release #187 (comment) for summary
• tests/cli/test_main_clean.py::test_clean_all_mock_lstat[False] is failing (unsure exactly why at the moment but can reproduce)
• tests/core/test_subdir_data.py::test_state_is_not_json is failing but cannot reproduce
• tests/test_activate.py::ShellWrapperIntegrationTests::test_dash_basic_integration is failing but cannot reproduce

[wolfv]

Any clues what is going wrong / what changed in libmamba?

[jaimergp]

Good job with the research, @costrouc! A new libmamba release behind the errors makes sense, given how it doesn't affect defaults (1.4.2 not released there yet)

Looking at the changelog, these would be the PRs that touched solver code:

• Small libsolv improvments mamba-org/mamba#2399
• Use custom function to properly parse matchspec mamba-org/mamba#2433
• Fix segfault in add_pin/all_problems_structured mamba-org/mamba#2428

Those are significant changes so I can't tell right away what could be the cause.

[jaimergp]

CONDA_SOLVER=libmamba pytest tests/test_activate.py::ShellWrapperIntegrationTests::test_dash_basic_integration

This one is known to be flaky so it fails here and there, no worries.

[jaimergp]

Given that we have other work in the pipeline blocked by this, and the failing tests are not core issues (kind of edge cases), I'd say we annotate them in our meta-issue and mark them as xfail or skipif with a condition for libmamba==1.4.2.

[jaimergp]

More data:

• tests/cli/test_main_clean.py::test_clean_all_mock_lstat[False]. Fixed. We had some strange logic for json/quiet/verbose combos that didn't work for this particular test.
• tests/core/test_subdir_data.py::test_state_is_not_json only fails on Windows? 🤔
• tests/test_activate.py::ShellWrapperIntegrationTests::test_dash_basic_integration is working now.

[jaimergp]

Cannot reproduce tests/core/test_subdir_data.py::test_state_is_not_json on the Windows VM either :/