Skip to content

Commit

Permalink
Docs about --iterate
Browse files Browse the repository at this point in the history
  • Loading branch information
sourcefrog committed Jun 20, 2024
1 parent 37612c7 commit 308aa43
Show file tree
Hide file tree
Showing 3 changed files with 35 additions and 1 deletion.
1 change: 1 addition & 0 deletions book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
- [Using nextest](nextest.md)
- [Baseline tests](baseline.md)
- [Testing in-place](in-place.md)
- [Iterating on missed mutants](iterate.md)
- [Generating mutants](mutants.md)
- [Error values](error-values.md)
- [Improving performance](performance.md)
Expand Down
31 changes: 31 additions & 0 deletions book/src/iterate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Iterating on missed mutants

When you're working to improve test coverage in a tree, you might use a process like this:

1. Run `cargo-mutants` to find code that's untested, possibly filtering to some selected files.

2. Think about why some mutants are missed, and then write tests that will catch them.

3. Run cargo-mutants again to learn whether your tests caught all the mutants, or if any remain.

4. Repeat until everything is caught.

You can speed up this process by using the `--iterate` option. This tells cargo-mutants to skip mutants that were either caught or unviable in a previous run, and to accumulate the results.

You can run repeatedly with `--iterate`, adding tests each time, until all the missed mutants are caught (or skipped.)

## How it works

When `--iterate` is given, cargo-mutants reads `mutants.out/caught.txt`, `previously_caught.txt`, and `unviable.txt` before renaming that directory to `mutants.out.old`. If those files don't exist, the lists are assumed to be empty.

Mutants are then tested as usual, but excluding all the mutants named in those files. `--list --iterate` also applies this exclusion and shows you which mutants will be tested.

Mutants are matched based on their file name, line, column, and description, just as shown in `--list` and in those files. As a result, if you insert or move text in a source file, some mutants may be re-tested.

After testing, all the previously caught, caught, and unviable are written into `previously_caught.txt` so that they'll be excluded on future runs.

`previously_caught.txt` is only written when `--iterate` is given.

## Caution

`--iterate` is a heuristic, and makes the assumption that any new changes you make won't reduce coverage, which might not be true. After you think you've caught all the mutants, you should run again without `--iterate` to make sure.
4 changes: 3 additions & 1 deletion book/src/mutants-out.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,10 @@ The output directory contains:

* `caught.txt`, `missed.txt`, `timeout.txt`, `unviable.txt`, each listing mutants with the corresponding outcome.

* `previously_caught.txt` accumulates a list of mutants caught in previous runs with [`--iterate`](iterate.md).

The contents of the directory and the format of these files is subject to change in future versions.

These files are incrementally updated while cargo-mutants runs, so other programs can read them to follow progress.

There is generally no reason to include this directory in version control, so it is recommended that you add `/mutants.out*` to your `.gitignore` file. This will exclude both `mutants.out` and `mutants.out.old`.
There is generally no reason to include this directory in version control, so it is recommended that you add `/mutants.out*` to your `.gitignore` file. This will exclude both `mutants.out` and `mutants.out.old`.

0 comments on commit 308aa43

Please sign in to comment.