Generate mutant only for the changed lines in a range of commits

[PauloGoncalvesBH]

Is your feature request related to a problem? Please describe.
The mutation test is very expensive and time-consuming in medium and large projects, and its execution in all Pull Requests is not feasible. That is why we have adopted the following strategy:

1. We perform a complete mutation test once a day.
2. We perform mutation testing only on changed files using the stryker-diff-runner library (https://www.npmjs.com/package/stryker-diff-runner).

The problem is that even selecting which files to run the mutation test ends up being time consuming and it also forces the disabling of the break threshold, as it generates mutants for the entire file, and not for the changed lines.
Something that would help users of the Striker library to use it routinely, in every code change, would be to have an option to generate mutation test only for the changed lines in a given range of commits.

Describe the solution you'd like
Provide an option via CLI to pass the range of commits, like --range.

Example (default branch is main):

"scripts": {
  "test:mutation": "stryker run",
  "test:mutation:diff:last-commit": "stryker run --range 'HEAD'",
  "test:mutation:diff:actual-branch-to-default": "stryker run --range 'origin/main..HEAD'"
}

Describe alternatives you've considered
This solution came to mind when I saw this comment from @nicojs: #2740 (comment)

Additional context
N/A

[rene-leanix]

Would something like this work for you in the meantime?

npx stryker run --mutate $(git diff --name-only HEAD $(git merge-base HEAD origin/main) -- "*.ts" ":(exclude)*.spec.ts" | tr '\n' ',' | sed 's/,$//')

The downside of this approach is that the path globs specified in the Stryker config file would need to be replicated here. Alternatively one could create a short NodeJS script to execute this command based on those path globs.

The best long-term solution seems to be the incremental-mode suggested in #2753.

[PauloGoncalvesBH]

@rene-leanix this solution makes it generate mutant for the entire file, and what I'm looking for is that it generates mutant only for the changed lines in certain commits.
This suggestion you provided is similar to what the stryker-diff-runner library does.
https://github.com/tverhoken/stryker-diff-runner/blob/e2ecb3d15cc5093c981c1d26891f53cf8f7165e1/src/index.ts#L32

[nicojs]

We've recently added support to mutate a specific range only, see https://stryker-mutator.io/docs/stryker-js/configuration#mutate-string

Example: stryker run --mutate foo.js:25-30

This can be combined with some kind of pipeline git diff command probably, or with a small script around our public API: https://github.com/stryker-mutator/stryker-js/tree/master/packages/core#programmatic-use

[lbtoma]

@PauloGoncalvesBH , I'm creating a new checker plugin for this:

https://github.com/lbtoma/stryker-git-checker

I only ran this in a small project, works only for the last commit, has a lot to improve, but the idea is to provide some ways to accomplish what you described.

[nicojs]

@lbtoma thanks for pitching in! I had a quick look and it looks great. I've opened #3228, as you will need the Ignored check status to be supported.

Note: An "ignore plugin" (name pending) might be better suited for your use case. Checker plugins are really designed to "check" mutants (typescript type check, eslint lint check, etc). For example, when you run Stryker with --concurrency 10, 5 checker plugins get created, so the git command would be executed 5 times. Probably better for performance to hook into the mutant generation mechanism instead.

EDIT: I've also opened #3229 I think that plugin would be a better fit for your use case. We have a hackathon planned this weekend, we might be able to pick this one up.

[lbtoma]

@nicojs , thank you! These "Ignore" plugin type really would be great for my use case

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.