Skip to content

Commit

Permalink
Merge pull request #90 from dorny/mocha-json
Browse files Browse the repository at this point in the history
Add support for mocha-json
  • Loading branch information
dorny authored Apr 19, 2021
2 parents 39f7ac7 + 368fd24 commit e81e375
Show file tree
Hide file tree
Showing 27 changed files with 24,743 additions and 81 deletions.
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ jobs:
- run: npm test

- name: Upload test results
if: success() || failure()
uses: actions/upload-artifact@v2
with:
name: test-results
Expand Down
58 changes: 39 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ This [Github Action](https://github.com/features/actions) displays test results
- .NET / [xUnit](https://xunit.net/) / [NUnit](https://nunit.org/) / [MSTest](https://github.com/Microsoft/testfx-docs)
- Dart / [test](https://pub.dev/packages/test)
- Flutter / [test](https://pub.dev/packages/test)
- JavaScript / [JEST](https://jestjs.io/)
- Java / [JUnit](https://junit.org/)
- JavaScript / [JEST](https://jestjs.io/) / [Mocha](https://mochajs.org/)

For more information see [Supported formats](#supported-formats) section.

Expand Down Expand Up @@ -54,9 +54,9 @@ jobs:
## Recommended setup for public repositories
Workflows triggered by pull requests from forked repositories are executed with read-only token and therefore can't create check runs.
To workaround this security restriction it's required to use two separate workflows:
1. `CI` runs in the context of PR head branch with read-only token. It executes the tests and uploads test results as build artifact
2. `Test Report` runs in the context of repository main branch with read/write token. It will download test results and create reports
To workaround this security restriction, it's required to use two separate workflows:
1. `CI` runs in the context of the PR head branch with the read-only token. It executes the tests and uploads test results as a build artifact
2. `Test Report` runs in the context of the repository main branch with read/write token. It will download test results and create reports

**PR head branch:** *.github/workflows/ci.yml*
```yaml
Expand Down Expand Up @@ -116,7 +116,7 @@ jobs:
# Coma separated list of paths to test results
# Supports wildcards via [fast-glob](https://github.com/mrmlnc/fast-glob)
# All matched result files must be of same format
# All matched result files must be of the same format
path: ''
# Format of test results. Supported options:
Expand All @@ -125,6 +125,7 @@ jobs:
# flutter-json
# java-junit
# jest-junit
# mocha-json
reporter: ''
# Limits which test suites are listed:
Expand All @@ -142,7 +143,7 @@ jobs:
# Must be less or equal to 50.
max-annotations: '10'
# Set action as failed if test report contain any failed test
# Set action as failed if test report contains any failed test
fail-on-error: 'true'
# Relative path under $GITHUB_WORKSPACE where the repository was checked out.
Expand Down Expand Up @@ -224,8 +225,8 @@ Or with (undocumented) CLI argument:


According to documentation `dart_test.yaml` should be at the root of the package, next to the package's pubspec.
On current `stable` and `beta` channels it doesn't work and you have to put `dart_test.yaml` inside your `test` folder.
On `dev` channel it's already fixed.
On current `stable` and `beta` channels it doesn't work, and you have to put `dart_test.yaml` inside your `test` folder.
On `dev` channel, it's already fixed.

For more information see:
- [test package](https://pub.dev/packages/test)
Expand All @@ -239,17 +240,17 @@ For more information see:
<summary>java-junit (Experimental)</summary>

Support for [JUnit](https://Junit.org/) XML is experimental - should work but it was not extensively tested.
To have code annotations working properly it's required your directory structure matches package name.
This is due to the fact Java stacktraces doesn't contains full path to the source file.
Some heuristic was necessary to figure out mapping between line in stack trace and actual source file.
To have code annotations working properly, it's required your directory structure matches the package name.
This is due to the fact Java stack traces don't contain a full path to the source file.
Some heuristic was necessary to figure out the mapping between the line in the stack trace and an actual source file.
</details>

<details>
<summary>jest-Junit</summary>

[JEST](https://jestjs.io/) testing framework support requires usage of [jest-Junit](https://github.com/jest-community/jest-Junit) reporter.
[JEST](https://jestjs.io/) testing framework support requires the usage of [jest-Junit](https://github.com/jest-community/jest-Junit) reporter.
It will create test results in Junit XML format which can be then processed by this action.
You can use following example configuration in `package.json`:
You can use the following example configuration in `package.json`:
```json
"scripts": {
"test": "jest --ci --reporters=default --reporters=jest-Junit"
Expand All @@ -272,19 +273,38 @@ You can use following example configuration in `package.json`:
Configuration of `uniqueOutputName`, `suiteNameTemplate`, `classNameTemplate`, `titleTemplate` is important for proper visualization of test results.
</details>

<details>
<summary>mocha-json</summary>

[Mocha](https://mochajs.org/) testing framework support requires:
- Mocha version [v7.2.0](https://github.com/mochajs/mocha/releases/tag/v7.2.0) or higher
- Usage of [json](https://mochajs.org/#json) reporter.

You can use the following example configuration in `package.json`:
```json
"scripts": {
"test": "mocha --reporter json > test-results.json"
}
```

Test processing might fail if any of your tests write anything on standard output.
Mocha, unfortunately, doesn't have the option to store `json` output directly to the file, and we have to rely on redirecting its standard output.
There is a work in progress to fix it: [mocha#4607](https://github.com/mochajs/mocha/pull/4607)
</details>

## GitHub limitations

Unfortunately there are some known issues and limitations caused by GitHub API:
Unfortunately, there are some known issues and limitations caused by GitHub API:

- Test report (i.e. Check Run summary) is markdown text. No custom styling or HTML is possible.
- Maximum report size is 65535 bytes. Input parameters `list-suites` and `list-tests` will be automatically adjusted if max size is exceeded.
- Test report can't reference any additional files (e.g. screenshots). You can use `actions/upload-artifact@v2` to upload them and inspect manually.
- Check Runs are created for specific commit SHA. it's not possible to specify under which workflow test report should belong if there are more
workflows running for same SHA. Thanks to this GitHub "feature" it's possible your test report will appear in unexpected place in GitHub UI.
For more information see [#67](https://github.com/dorny/test-reporter/issues/67).
- Test report can't reference any additional files (e.g. screenshots). You can use `actions/upload-artifact@v2` to upload them and inspect them manually.
- Check Runs are created for specific commit SHA. It's not possible to specify under which workflow test report should belong if more
workflows are running for the same SHA. Thanks to this GitHub "feature" it's possible your test report will appear in an unexpected place in GitHub UI.
For more information, see [#67](https://github.com/dorny/test-reporter/issues/67).

## See also
- [paths-filter](https://github.com/dorny/paths-filter) - Conditionally run actions based on files modified by PR, feature branch or pushed commits
- [paths-filter](https://github.com/dorny/paths-filter) - Conditionally run actions based on files modified by PR, feature branch, or pushed commits

## License

Expand Down
29 changes: 29 additions & 0 deletions __tests__/__outputs__/mocha-json.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
![Tests failed](https://img.shields.io/badge/tests-1%20passed%2C%204%20failed%2C%201%20skipped-critical)
## ❌ <a id="user-content-r0" href="#r0">fixtures/mocha-json.json</a>
**6** tests were completed in **12ms** with **1** passed, **4** failed and **1** skipped.
|Test suite|Passed|Failed|Skipped|Time|
|:---|---:|---:|---:|---:|
|[test/main.test.js](#r0s0)|1✔️|3❌||1ms|
|[test/second.test.js](#r0s1)||1❌|1✖️|8ms|
### ❌ <a id="user-content-r0s0" href="#r0s0">test/main.test.js</a>
```
Test 1
✔️ Passing test
Test 1 Test 1.1
❌ Exception in target unit
Some error
❌ Failing test
Expected values to be strictly equal:
false !== true
Test 2
❌ Exception in test
Some error
```
### ❌ <a id="user-content-r0s1" href="#r0s1">test/second.test.js</a>
```
✖️ Skipped test
❌ Timeout test
Timeout of 1ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (C:\Users\Michal\Workspace\dorny\test-reporter\reports\mocha\test\second.test.js)
```
41 changes: 41 additions & 0 deletions __tests__/__outputs__/mocha-test-results.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
![Tests passed successfully](https://img.shields.io/badge/tests-833%20passed%2C%206%20skipped-success)
## ✔️ <a id="user-content-r0" href="#r0">fixtures/external/mocha/mocha-test-results.json</a>
**839** tests were completed in **6s** with **833** passed, **0** failed and **6** skipped.
|Test suite|Passed|Failed|Skipped|Time|
|:---|---:|---:|---:|---:|
|test/node-unit/buffered-worker-pool.spec.js|14✔️|||8ms|
|test/node-unit/cli/config.spec.js|10✔️|||8ms|
|test/node-unit/cli/node-flags.spec.js|105✔️|||9ms|
|test/node-unit/cli/options.spec.js|36✔️|||250ms|
|test/node-unit/cli/run-helpers.spec.js|9✔️|||8ms|
|test/node-unit/cli/run.spec.js|40✔️|||4ms|
|test/node-unit/mocha.spec.js|24✔️|||33ms|
|test/node-unit/parallel-buffered-runner.spec.js|19✔️|||23ms|
|test/node-unit/reporters/parallel-buffered.spec.js|6✔️|||16ms|
|test/node-unit/serializer.spec.js|40✔️|||31ms|
|test/node-unit/stack-trace-filter.spec.js|2✔️||4✖️|1ms|
|test/node-unit/utils.spec.js|5✔️|||1ms|
|test/node-unit/worker.spec.js|15✔️|||92ms|
|test/unit/context.spec.js|8✔️|||5ms|
|test/unit/duration.spec.js|3✔️|||166ms|
|test/unit/errors.spec.js|13✔️|||5ms|
|test/unit/globals.spec.js|4✔️|||0ms|
|test/unit/grep.spec.js|8✔️|||2ms|
|test/unit/hook-async.spec.js|3✔️|||1ms|
|test/unit/hook-sync-nested.spec.js|4✔️|||1ms|
|test/unit/hook-sync.spec.js|3✔️|||0ms|
|test/unit/hook-timeout.spec.js|1✔️|||0ms|
|test/unit/hook.spec.js|4✔️|||0ms|
|test/unit/mocha.spec.js|115✔️||1✖️|128ms|
|test/unit/overspecified-async.spec.js|1✔️|||3ms|
|test/unit/parse-query.spec.js|2✔️|||1ms|
|test/unit/plugin-loader.spec.js|41✔️||1✖️|16ms|
|test/unit/required-tokens.spec.js|1✔️|||0ms|
|test/unit/root.spec.js|1✔️|||0ms|
|test/unit/runnable.spec.js|55✔️|||122ms|
|test/unit/runner.spec.js|77✔️|||43ms|
|test/unit/suite.spec.js|57✔️|||14ms|
|test/unit/test.spec.js|15✔️|||0ms|
|test/unit/throw.spec.js|9✔️|||9ms|
|test/unit/timeout.spec.js|8✔️|||109ms|
|test/unit/utils.spec.js|75✔️|||24ms|
Loading

0 comments on commit e81e375

Please sign in to comment.