Produce JSON reports

[tymfear]

Please provide details about:

• What you're trying to do
  I want tova ability to have reports in JSON format to process it later to any other format I need.
• Why you can't use AVA for this
  Only TAP report is available at the time

Discussed with @novemberborn in #2583

[novemberborn]

Internally AVA has quite a lot of data about the test execution that is not easily represented in TAP form. I think it'd be more powerful to convert that into some other machine-readable format.

[codetheweb]

Is ZAP (by @jamiebuilds) still the approach we want to take? Might start looking at building this out.

[novemberborn]

I wasn't aware of ZAP. It's got the right motivation. But it's also ~4 years old, is there any ecosystem or tooling around it?

[codetheweb]

There was a little discussion around it in #1878. ZAP has some basic tooling but I don't think it's actually being used by anyone at the moment. On the other hand, I'm not aware of any standard test output formats besides JUnit & TAP.

I don't really care what format is used, but having something that's machine readable (besides TAP) would be really helpful for CI tasks. 😃

[novemberborn]

Could you elaborate on your CI use cases?

Our reporting pipeline is in quite a state, and I'd want to make sure your contributions won't be in vain.

[codetheweb]

Sure, a few use cases we might want to explore:

• Does this PR significantly increase the duration of any tests?
• Splitting tests into separate jobs based on historical runtime (ex. 1 job for tests that usually finish in under 2 seconds, and 1 for everything else)

In general, the test output can be hard to read--especially if there's a lot of concurrent tests. Would be nice if we could, for example, comment in PRs with a table of failing tests with a link to their logs. Or even just add a GitHub annotation to failing files.

[novemberborn]

I wonder if there is a different approach we can take. The problem with reporter output is wanting it to be both human and machine readable. So even if AVA exposed its internal events, you'd then want to pipe it back to AVA for display. All of that suggests a level of documentation, tooling and stability that we're not yet ready to commit to.

AVA 4 has shared workers: dedicated worker threads that persist across test runs. We could make it so that you can load those from the config and provide an API so that they are notified of test runs. We could then add a subscription to AVA's internal events. And then we could have a completion mechanism that would allow you to write files or use the GitHub API.

This would be more generally useful.

I don't think we could commit to the internal events falling under semantic versioning, but a) if you're interested in these kinds of features you probably use pin dependencies, and b) they're unlikely to change any time soon.

What do you think?

[codetheweb]

That sounds like a great approach to me! It would allow people to write and share reporters for JUnit, ZAP, or whatever else they need which seems perfect.

Please let me know if I can help build this out in any way.

[novemberborn]

@codetheweb let's take it to #3004.

[jamiebuilds]

@novemberborn If you are interested in ZAP I can definitely return to it. The issue at the time is that I couldn't find any tools that immediately wanted to adopt it. I actually have a Rust version of it with some changes to its design (simplifying the spec a fair bit) that runs super fast.

[novemberborn]

@jamiebuilds thank you for offering. I think at this stage, I'd rather we invest in the human readable reporter and making AVA more extensible through shared workers. Down the line we can return to our machine readable reporters.