Port sort-research-rs test suite to Rust stdlib tests

[Voultapher]

This PR is a followup to #124032. It replaces the tests that test the various sort functions in the standard library with a test-suite developed as part of https://github.com/Voultapher/sort-research-rs. The current tests suffer a couple of problems:

• They don't cover important real world patterns that the implementations take advantage of and execute special code for.
• The input lengths tested miss out on code paths. For example, important safety property tests never reach the quicksort part of the implementation.
• The miri side is often limited to len <= 20 which means it very thoroughly tests the insertion sort, which accounts for 19 out of 1.5k LoC.
• They are split into to core and alloc, causing code duplication and uneven coverage.
• The randomness is tied to a caller location, wasting the space exploration capabilities of randomized testing. The randomness is not repeatable, as it relies on std::hash::RandomState::new().build_hasher().

Most of these issues existed before #124032, but they are intensified by it. One thing that is new and requires additional testing, is that the new sort implementations specialize based on type properties. For example Freeze and non Freeze execute different code paths.

Effectively there are three dimensions that matter:

• Input type
• Input length
• Input pattern

The ported test-suite tests various properties along all three dimensions, greatly improving test coverage. It side-steps the miri issue by preferring sampled approaches. For example the test that checks if after a panic the set of elements is still the original one, doesn't do so for every single possible panic opportunity but rather it picks one at random, and performs this test across a range of input length, which varies the panic point across them. This allows regular execution to easily test inputs of length 10k, and miri execution up to 100 which covers significantly more code. The randomness used is tied to a fixed - but random per process execution - seed. This allows for fully repeatable tests and fuzzer like exploration across multiple runs.

Structure wise, the tests are previously found in the core integration tests for sort_unstable and alloc unit tests for sort. The new test-suite was developed to be a purely black-box approach, which makes integration testing the better place, because it can't accidentally rely on internal access. Because unwinding support is required the tests can't be in core, even if the implementation is, so they are now part of the alloc integration tests. Are there architectures that can only build and test core and not alloc? If so, do such platforms require sort testing? For what it's worth the current implementation state passes miri --target mips64-unknown-linux-gnuabi64 which is big endian.

The test-suite also contains tests for properties that were and are given by the current and previous implementations, and likely relied upon by users but weren't tested. For example self_cmp tests that the two parameters a and b passed into the comparison function are never references to the same object, which if the user is sorting for example a &mut [Mutex<i32>] could lead to a deadlock.

Instead of using the hashed caller location as rand seed, it uses seconds since unix epoch / 10, which given timestamps in the CI should be reasonably easy to reproduce, but also allows fuzzer like space exploration.

---

Test run-time changes:

Setup:

Linux 6.10
rustc 1.83.0-nightly (f79a912d9 2024-09-18)
AMD Ryzen 9 5900X 12-Core Processor (Zen 3 micro-architecture)
CPU boost enabled.

master: e9df22f

Before core integration tests:

$ LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/ hyperfine build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/coretests-219cbd0308a49e2f
  Time (mean ± σ):     869.6 ms ±  21.1 ms    [User: 1327.6 ms, System: 95.1 ms]
  Range (min … max):   845.4 ms … 917.0 ms    10 runs

# MIRIFLAGS="-Zmiri-disable-isolation" to get real time
$ MIRIFLAGS="-Zmiri-disable-isolation" ./x.py miri library/core
  finished in 738.44s

After core integration tests:

$ LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/ hyperfine build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/coretests-219cbd0308a49e2f
  Time (mean ± σ):     865.1 ms ±  14.7 ms    [User: 1283.5 ms, System: 88.4 ms]
  Range (min … max):   836.2 ms … 885.7 ms    10 runs

$ MIRIFLAGS="-Zmiri-disable-isolation" ./x.py miri library/core
  finished in 752.35s

Before alloc unit tests:

LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/ hyperfine build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/alloc-19c15e6e8565aa54
  Time (mean ± σ):     295.0 ms ±   9.9 ms    [User: 719.6 ms, System: 35.3 ms]
  Range (min … max):   284.9 ms … 319.3 ms    10 runs

$ MIRIFLAGS="-Zmiri-disable-isolation" ./x.py miri library/alloc
  finished in 322.75s

After alloc unit tests:

LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/ hyperfine build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/alloc-19c15e6e8565aa54
  Time (mean ± σ):      97.4 ms ±   4.1 ms    [User: 297.7 ms, System: 28.6 ms]
  Range (min … max):    92.3 ms … 109.2 ms    27 runs

$ MIRIFLAGS="-Zmiri-disable-isolation" ./x.py miri library/alloc
  finished in 309.18s

Before alloc integration tests:

$ LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/ hyperfine build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/alloctests-439e7300c61a8046
  Time (mean ± σ):     103.2 ms ±   1.7 ms    [User: 135.7 ms, System: 39.4 ms]
  Range (min … max):    99.7 ms … 107.3 ms    28 runs

$ MIRIFLAGS="-Zmiri-disable-isolation" ./x.py miri library/alloc
  finished in 231.35s

After alloc integration tests:

$ LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/ hyperfine build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/alloctests-439e7300c61a8046
  Time (mean ± σ):     379.8 ms ±   4.7 ms    [User: 4620.5 ms, System: 1157.2 ms]
  Range (min … max):   373.6 ms … 386.9 ms    10 runs

$ MIRIFLAGS="-Zmiri-disable-isolation" ./x.py miri library/alloc
  finished in 449.24s

In my opinion the results don't change iterative library development or CI execution in meaningful ways. For example currently the library doc-tests take ~66s and incremental compilation takes 10+ seconds. However I only have limited knowledge of the various local development workflows that exist, and might be missing one that is significantly impacted by this change.

[rustbot]

r? @joboet

rustbot has assigned @joboet.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[Voultapher]

r? @thomcc

[thomcc]

Tremendous work, and great to see this followed up on. Since it's test-only and relatively clean I don't really have any changes I want.

[thomcc]

CC @RalfJung IIRC you want to be made aware of cases where we add miri-specific conditionals to the test suite? (Sorry if I'm misremembering). This adds quite a few (most generated by a macro), but is done only for performance.

[RalfJung]

Hard for me to say what this means for test coverage in Miri, but I assume it is still a lot better than before this PR, so -- LGTM.

How much does this add to the test runtime in Miri, compared to before?

[Voultapher]

How much does this add to the test runtime in Miri, compared to before?

Take a look at the PR description.

[RalfJung]

Ah, thanks. So the main hit is on alloc integration tests, which get 3.5 minutes slower. That's quite a bit, and CI machines will probably be slower than this. But these are also valuable tests so it's probably worth it.

Cc @rust-lang/infra FYI

[Voultapher]

There is something funky going on here, though, in my project running the same tests, on a branch not main to avoid confusion, I get a test time of ~135s, so the ~220s in the Rust project are a bit weird. With nextest I get ~40s.

I think there are a couple non-blocking future points in terms of miri CI time:

• Monitor it more closely, I did the same measurements a month ago and now again and the core integration tests went from ~500s to ~700s, so there seems to be an ongoing creep.
• Run the tests with nextest to exploit multi-core machines.
• Go through the tests and figure out which ones are poorly tuned or have a bad run-time to information ratio.

[RalfJung]

There is something funky going on here, though, in my project running the same tests, on a branch not main to avoid confusion, I get a test time of ~135s, so the ~220s in the Rust project are a bit weird.

Maybe std debug assertions are enabled in CI? Not sure.

Run the tests with nextest to exploit multi-core machines.

I think @saethlin looked into using nextest and it didn't work... it's very tricky since advanced hackery is needed to make this work as part of bootstrap at all.

Go through the tests and figure out which ones are poorly tuned or have a bad run-time to information ratio.

I do sometimes look at the times here (where things are set up to show how long each test takes), to identify extreme cases. But more can probably be done.

[saethlin]

I think @saethlin looked into using nextest and it didn't work... it's very tricky since advanced hackery is needed to make this work as part of bootstrap at all.

I looked into using it for miri-test-libstd, and what got me hung up there is that nextest uses a very different test filtering system, and miri-test-libstd does a lot of test filtering.

[Voultapher]

I think this is good to be merged, right?

[thomcc]

@bors r+

[bors]

📌 Commit 71bb0e7 has been approved by thomcc

It is now in the queue for this repository.