Skip to content

Commit

Permalink
Report mutation test coverage of proposed changes (#346)
Browse files Browse the repository at this point in the history
Thanks to a free Arcmutate OSS license, GitHub Actions now runs Pitest against
files changed in the context of a PR. 

While there, update Pitest's configuration to ignore Refaster rule collection
classes, as mutations of those classes will not impact the associated unit
tests.

See:
- https://www.arcmutate.com
- https://pitest.org
  • Loading branch information
Stephan202 authored Nov 30, 2022
1 parent f46859a commit 3303283
Show file tree
Hide file tree
Showing 5 changed files with 100 additions and 1 deletion.
37 changes: 37 additions & 0 deletions .github/workflows/pitest-analyze-pr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Performs mutation testing analysis on the files changed by a pull request and
# uploads the results. The associated PR is subsequently updated by the
# `pitest-update-pr.yml` workflow. See https://blog.pitest.org/oss-pitest-pr/
# for details.
name: "Mutation testing"
on:
pull_request:
permissions:
contents: read
jobs:
analyze-pr:
runs-on: ubuntu-22.04
steps:
- name: Check out code
uses: actions/[email protected]
with:
fetch-depth: 2
- name: Set up JDK
uses: actions/[email protected]
with:
java-version: 17.0.4
distribution: temurin
cache: maven
- name: Run Pitest
# By running with features `+GIT(from[HEAD~1]), +gitci`, Pitest only
# analyzes lines changed in the associated pull request, as GitHub
# exposes the changes unique to the PR as a single commit on top of the
# target branch. See https://blog.pitest.org/pitest-pr-setup for
# details.
run: mvn test pitest:mutationCoverage -DargLine.xmx=2048m -Dverification.skip -Dfeatures="+GIT(from[HEAD~1]), +gitci"
- name: Aggregate Pitest reports
run: mvn pitest-git:aggregate -DkilledEmoji=":tada:" -DmutantEmoji=":zombie:" -DtrailingText="Mutation testing report by [Pitest](https://pitest.org/). Review any surviving mutants by inspecting the line comments under [_Files changed_](${{ github.event.number }}/files)."
- name: Upload Pitest reports as artifact
uses: actions/[email protected]
with:
name: pitest-reports
path: ./target/pit-reports-ci
35 changes: 35 additions & 0 deletions .github/workflows/pitest-update-pr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Updates a pull request based on the corresponding mutation testing analysis
# performed by the `pitest-analyze-pr.yml` workflow. See
# https://blog.pitest.org/oss-pitest-pr/ for details.
name: "Mutation testing: post results"
on:
workflow_run:
workflows: ["Mutation testing"]
types:
- completed
permissions:
actions: read
checks: write
contents: read
pull-requests: write
jobs:
update-pr:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-22.04
steps:
- name: Check out code
uses: actions/[email protected]
- name: Set up JDK
uses: actions/[email protected]
with:
java-version: 17.0.4
distribution: temurin
cache: maven
- name: Download Pitest analysis artifact
uses: dawidd6/[email protected]
with:
workflow: ${{ github.event.workflow_run.workflow_id }}
name: pitest-reports
path: ./target/pit-reports-ci
- name: Update PR
run: mvn -DrepoToken="${{ secrets.GITHUB_TOKEN }}" pitest-github:updatePR
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ Some other commands one may find relevant:

- `mvn fmt:format` formats the code using
[`google-java-format`][google-java-format].
- `./run-mutation-tests.sh` runs mutation tests using [PIT][pitest]. The
- `./run-mutation-tests.sh` runs mutation tests using [Pitest][pitest]. The
results can be reviewed by opening the respective
`target/pit-reports/index.html` files. For more information check the [PIT
Maven plugin][pitest-maven].
Expand Down
7 changes: 7 additions & 0 deletions cdg-pitest-licence.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Arcmutate license for Error Prone Support, requested by sending an email to
# [email protected].
expires=07/11/2023
keyVersion=1
signature=MhZxMbnO6UovNfllM0JuVWkZyvRT3/G5o/uT0Mm36c7200VpZNVu03gTAGivnl9W5RzvZhfpIHccuQ5ctjQkrqhsFSrl4fyqPqu3y5V2fsHIdFXP/G72EGj6Kay9ndLpaEHalqE0bEwxdnHMzEYq5y3O9vUPv8MhUl57xk+rvBo\=
packages=tech.picnic.errorprone.*
type=OSSS
20 changes: 20 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@
<version.mockito>4.9.0</version.mockito>
<version.nopen-checker>1.0.1</version.nopen-checker>
<version.nullaway>0.10.5</version.nullaway>
<version.pitest-git>1.0.1</version.pitest-git>
<version.surefire>2.22.2</version.surefire>
</properties>

Expand Down Expand Up @@ -428,6 +429,16 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.groupcdg</groupId>
<artifactId>pitest-git-maven-plugin</artifactId>
<version>${version.pitest-git}</version>
</plugin>
<plugin>
<groupId>com.groupcdg</groupId>
<artifactId>pitest-github-maven-plugin</artifactId>
<version>${version.pitest-git}</version>
</plugin>
<plugin>
<groupId>com.spotify.fmt</groupId>
<artifactId>fmt-maven-plugin</artifactId>
Expand Down Expand Up @@ -1231,8 +1242,12 @@
<version>1.9.11</version>
<configuration>
<excludedClasses>
<!-- AutoValue generated classes. -->
<excludedClass>*.AutoValue_*</excludedClass>
<!-- Refaster rule collections. -->
<excludedClass>*.refaster*.*Rules*</excludedClass>
</excludedClasses>
<failWhenNoMutations>false</failWhenNoMutations>
<!-- Use multiple threads to speed things up. Extend
timeouts to prevent false positives as a result of
contention. -->
Expand All @@ -1241,6 +1256,11 @@
<timestampedReports>false</timestampedReports>
</configuration>
<dependencies>
<dependency>
<groupId>com.groupcdg</groupId>
<artifactId>pitest-git-plugin</artifactId>
<version>${version.pitest-git}</version>
</dependency>
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-junit5-plugin</artifactId>
Expand Down

0 comments on commit 3303283

Please sign in to comment.