Skip to content

Commit

Permalink
Introduce Checkstyle integration test (#865)
Browse files Browse the repository at this point in the history
The new `integration-tests/checkstyle-10.12.4.sh` script can be executed
locally or by adding a comment containing the string `/integration-test`
to a pull request.

The integration test comprises a shell script that:
1. Checks out a Checkstyle release tag.
2. Applies a small patch to enable Error Prone Support.
3. Runs the build in Error Prone patch mode, until no further changes
   are applied.
4. Validates that:
   - The build (including tests) still passes.
   - The set of changes matches a pre-recorded diff.
   - The set of Error Prone Support-emitted warnings matches a
     pre-recorded list.

The script also has a `--sync` mode using which the expected diff and 
warnings can be updated. Combined, the script makes it easy to assess
and track the impact of changes to Error Prone Support in relation to
the Checkstyle code base.

Over time this setup will be generalized to include integration tests
against other code bases.
  • Loading branch information
Stephan202 authored Oct 30, 2023
1 parent 8886080 commit e8d1499
Show file tree
Hide file tree
Showing 6 changed files with 64,322 additions and 0 deletions.
43 changes: 43 additions & 0 deletions .github/workflows/run-integration-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# If requested by means of a pull request comment, runs integration tests
# against the project, using the code found on the pull request branch.
# XXX: Generalize this to a matrix build of multiple integration tests,
# possibly using multiple JDK or OS versions.
# XXX: Investigate whether the comment can specify which integration tests run
# run. See this example of a dynamic build matrix:
# https://docs.github.com/en/actions/learn-github-actions/expressions#example-returning-a-json-object
name: "Integration tests"
on:
issue_comment:
types: [ created ]
permissions:
contents: read
jobs:
run-integration-tests:
name: On-demand integration test
if: |
github.event.issue.pull_request && contains(github.event.comment.body, '/integration-test')
runs-on: ubuntu-22.04
steps:
- name: Check out code
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
persist-credentials: false
ref: refs/pull/${{ github.event.issue.number }}/head
- name: Set up JDK
uses: actions/setup-java@0ab4596768b603586c0de567f2430c30f5b0d2b0 # v3.13.0
with:
java-version: 17.0.8
distribution: temurin
cache: maven
- name: Install project to local Maven repository
run: mvn -T1C install -DskipTests -Dverification.skip
- name: Run integration test
run: xvfb-run ./integration-tests/checkstyle-10.12.4.sh "${{ runner.temp }}/artifacts"
- name: Upload artifacts on failure
if: ${{ failure() }}
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3
with:
name: integration-test-checkstyle-10.12.4
path: "${{ runner.temp }}/artifacts"
- name: Remove installed project artifacts
run: mvn build-helper:remove-project-artifact
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,6 @@
target
*.iml
*.swp

# The Git repositories checked out by the integration test framework.
integration-tests/.repos
Loading

0 comments on commit e8d1499

Please sign in to comment.