Testing the new resolver (test suite, CI)

[pfmoore]

Now that we have at least the start of the new resolver in place, we need to consider how we integrate it into the test suite. I've modified one of the very basic install tests to run twice, once with the legacy resolver and once with --unstable-feature=resolver, but this obviously isn't a scalable approach.

Some issues we therefore need to consider:

1. How do we have a more general mechanism that allows us to run tests with both old and new resolvers?
2. How do we handle tests that use feaures the new resolver doesn't have implemented yet (effectively xfail for the new resolver, but not for the current one)?
3. How do we cater for tests where we expect different results? (Tests that rely on quirks of the legacy resolver, explicit tests of new resolver functionality, ...)

Ultimately, we want the full test suite to be run for both resolvers, so that we have a good regression test ensuring that we have feature parity, but we're likely a good way from that position yet. But the only way to know for sure how much there still is to do is to (reasonably regularly) run the full suite with the new resolver.

[pradyunsg]

One of the things to make sure of is, that there are YAML tests that are being skipped because they don't work with the legacy resolver. We'd want to unskip them.

[pfmoore]

Is there anything more we need to do here? @pradyunsg @uranusjr

We currently have:

1. A flag --new-resolver that allows us to run the tests against the new resolver.
2. A way of marking tests that are known to still fail with the new resolver (maybe we want to repurpose this issue to track getting the number of tests marked like this down to zero?)
3. A fixture that says "using the new resolver" that allows us to cater for differences in behaviour in the tests.

These test features will have to be modified once the new resolver becomes default, and ripped out when we get rid of the old resolver, but I'd suggest that those activities are better done as independent issues once they are needed.

If we are going to track the number of remaining "known failures" here, we should start linking issues for each of the remaining types of failure from this one, and keeping a running total here, for visibility.

[pfmoore]

For information, the following are the outstanding "known failures":

FAILED tests/functional/test_install.py::test_install_nonlocal_compatible_wheel_path - AssertionError: Script returned code: 1
FAILED tests/functional/test_install.py::test_install_pip_does_not_modify_pip_when_satisfied[True-install_args1-Requirement already up-to-date: pip in] - Assert...
FAILED tests/functional/test_install.py::test_install_pip_does_not_modify_pip_when_satisfied[False-install_args1-Requirement already up-to-date: pip in] - Asser...
FAILED tests/functional/test_install_reqs.py::test_install_distribution_union_with_versions - AssertionError: Script returned code: 1
FAILED tests/functional/test_install.py::test_hashed_install_failure_later_flag - AssertionError: Could not find 'Hashes are required in --require-hashes mode, ...
FAILED tests/functional/test_install_upgrade.py::test_upgrade_with_newest_already_installed - AssertionError: Looking in links: file:///C:/Users/Gustav/AppData/...
FAILED tests/functional/test_install_upgrade.py::test_only_if_needed_does_not_upgrade_deps_when_satisfied - AssertionError: did not print correct message for no...
FAILED tests/functional/test_uninstall_user.py::Tests_UninstallUserSite::test_uninstall_from_usersite_with_dist_in_global_site - AssertionError: Script returned...
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_install_user_conflict_in_globalsite_and_usersite - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_user.py::Tests_UserSite::test_upgrade_user_conflict_in_globalsite - AssertionError: Script returned code: 1
FAILED tests/functional/test_install_upgrade.py::test_upgrade_to_same_version_from_url - AssertionError: INITools 0.3 reinstalled same version

[pradyunsg]

I think the YAML tests need to be updated to respect --new-resolver, but other than that, I think this looks good to go to me! ^.^

[pradyunsg]

There's still a bunch of message-printing failures. :)

[pfmoore]

You mean in that list I posted? I know, I keep planning on fixing them (or at least acknowledging that we print something different and moving on) but getting side-tracked with the harder cases :-)

I'll aim at easy stuff today and see if that helps 😉

[pfmoore]

#8462 sorts out test_install_nonlocal_compatible_wheel_path.

[pradyunsg]

Well, how about you tackle the difficult stuff and I do the easy bits? 🙈

[pfmoore]

@pradyunsg Can you take a look at my comment on #8462? I swear I tried to pick a simple one, and it turned complicated. I think I'm jinxed... 🙄

[pfmoore]

Found some straightforward ones (I hope): #8466 fixes 4 more tests.

[brainwane]

Per conversation in last week's meeting this is a tracking issue for the remaining tests that are still failing, and needs to be finished before we switch pip to defaulting to the new resolver. As of last week, the count of fails_on_new_resolver in master is down to 3 failures, and all are actively being addressed.

[brainwane]

Per conversation today with @pradyunsg I'm closing this as finished. Thanks all!

[uranusjr]

Would it be a good idea after 20.3 to flip the current setup, testing the new resolver by default on all platforms, and the legacy resolver only on Travis?

[pradyunsg]

I'd say before. :)

I'm considering that a part of #8937.