Standard reporter interface

[jzaefferer]

We want to provide a reporter interface that we can share with other testing tools like Jasmine and Mocha, to make it easier for integration tools - like Karma, browserstack-runner, grunt plugins - to hook into QUnit and other tools. Anyone writing a new JavaScript testing tool should be encouraged to provide the same interface, making it more reasonable to support these tools.

This requires a bunch of research. So far we identified two components that need to be standardised:

• needs a data format ala JUnit XML (here's an unofficial xsd to validate the output) or TAP, that is flexible enough to support QUnit and others. Specifically, Jasmine and other BDD styles support (more or less) infinite nesting, which QUnit doesn't
• need a list of events that provide this data. Something like done should provide the full datastructure, other events should help to output progress

Related tickets:

• (testDone should provide aggregated assertion results #351) testDone should provide aggregated assertion results
• (Implement reporter interface #405) Implement reporter interface
• (Implement QUnit callbacks event listener style #422) Implement QUnit callbacks event listener style
• (QUnit.done should provide aggregated results #472) QUnit.done should provide aggregated results

[JamesMGreene]

I reviewed Mocha's reporter interface first.

Important details:

• All reporters seem to inherit from a base reporter, although it doesn't appear that this is an actual requirement. The base reporter is responsible for providing many base utilities, such diff-ing displays, etc.
• The reporters are setup as an event listener that listens for the following events to be emitted:
  • runner.on('start', fn) → QUnit.begin(fn);
  • runner.on('suite', fn) → Mostly equivalent to QUnit.moduleStart(fn) except that Mocha suites can be infinitely nested. The nesting is simply achieved by treating suites as a stack: if a new suite event is emitted without a suite end event for the previous suite, the new suite is a child of the previous suite.
  • runner.on('test', fn) → QUnit.testStart(fn)
  • runner.on('pass', fn) → Equivalent to QUnit.testDone(fn) with a "success" status. This is a convenience event only as the same info could be obtained via the test end event.
  • runner.on('fail', fn) → Equivalent to QUnit.testDone(fn) with a "failure" status. This is a convenience event only as the same info could be obtained via the test end event.
  • runner.on('pending', fn) → Equivalent to QUnit.testDone(fn) with a "pending" status. This is a convenience event only as the same info could be obtained via the test end event.
  • runner.on('test end', fn) → QUnit.testDone(fn)
  • runner.on('suite end', fn) → Mostly equivalent to QUnit.moduleDone(fn) except that Mocha suites can be infinitely nested.
  • runner.on('end', fn) → QUnit.done(fn)

There are also two more events that relate to the before, beforeEach, afterEach, and after functions:

• runner.on('hook', fn)
• runner.on('hook end', fn)

The reporter can be set by calling mocha.reporter(reporter) or via the Mocha constructor options [if you're using Mocha programmatically].

Mocha will only allow a single reporter.

Mocha does not have an event for assertions.

[JamesMGreene]

Jasmine reporters are classes that can implement up to 6 core eventing methods which will be invoked during the appropriate times:

• reporter.jasmineStarted → QUnit.begin
• reporter.suiteStarted → Mostly equivalent to QUnit.moduleStart except that Jasmine suites can be infinitely nested.
• reporter.specStarted → QUnit.testStart
• reporter.specDone → QUnit.testDone
• reporter.suiteDone → Mostly equivalent to QUnit.moduleDone except that Jasmine suites can be infinitely nested.
• reporter.jasmineDone → QUnit.done

Reporters are added via jasmine.getEnv().addReporter(reporter).

Jasmine will allow using multiple reporters simultaneously.

Jasmine does not have an event for assertions.

[jzaefferer]

That is looking promising, since the overall structure in all three projects is the same. QUnit only has one level of nesting for "suites", so the common standard should include the nesting, even if we won't make use of it.

The EventEmmiter style that Mocha implements and we've discussed previously (#422) seems like a good starting point.

Both mocha and jasmine use "suite" for the component we call "module". That reminds me of the discussion in #190. Not that I have any interest in reopening that once more.

[Krinkle]

Are we looking for a standard reporting interface (e.g. the EventEmitter interface, and a standardised set of event names and arguments, which HTML or CLI formatters can use to display results and indicate progress). Or for a standard test suite summary format (e.g. like a JSON variant of jUnit TestSuite XML output) that one would retrieve at once to store and display results (either all, or just the failures).

I suppose both are interesting, but they're different things. I think the latter is more valuable for now (for grunt plugins and browserstack/saucelabs interpreters). The way to get that summary is probably not a big deal and can be slightly different per test framework (it's unlikely they'd factor out that one line of code to be the same between all frameworks). It'd be like QUnit.on( "<eventname>", function( summary ) { callback( summary ); });

[Krinkle]

Regarding the summary format, see also jquery/testswarm#183.

[Krinkle]

After today's QUnit meeting, @JamesMGreene put together a proposal: https://gist.github.com/JamesMGreene/9786246

---

Couple of thoughts on https://gist.github.com/JamesMGreene/9786246/90f16610b02b8267d18f61b7648b6364ca84da39:

• I'm not sure if the universal reporter interface should distinguish between suites and tests. Not every test framework might have these. And the ones that do, implement them quite differently. For example, in Jasmine suites and tests can be nested. Is there a use case of separating these?
• I'd recommend using names that are also valid identifiers in javascript (ideally both es3 and es5, but at least es5) so that we don't need bracket syntax and string literals all over the place.
• Though this is just coding style, knowing examples sometimes get blindly copied and are hard to kill once they get spread, I'd reduce the examples to 1) Not do typeof function check (just plain boolean check), 2) Use dot notation, 3) Not create local variables for each of them.

[leobalter]

@Krinkle: I did some changes based on your suggestions: https://gist.github.com/leobalter/226637d148577ac9b5c3

I did some research about reserved words as well to check for irregularities, everything is ok.

[leobalter]

@garu is a friend who is a Perl developers used to the TAP specification. I've talked to him and he gently agreed to help prototyping the methods for the TAP output.

Btw, now I also saw that we should expand the methods exportation to handle the passed data in each of the callback arguments. Probably Jasmine and Mocha are both different from QUnit too.

[JamesMGreene]

Cross-reference issues:

• Share a standard Reporter interface? mochajs/mocha#1326
• Share a standard Reporter interface? jasmine/jasmine#659
• Share a standard Reporter interface? theintern/intern#257

[JamesMGreene]

Created central org and repo for discussion and prototying: https://github.com/js-reporters/js-reporters

Discussions: https://github.com/js-reporters/js-reporters/issues/

Will be inviting folks from the various unit testing framework teams to join.

[supunasp]

Gsoc 2015 -

I am interested in this project idea. but I cant understand this and how to work. Any help or references would be great.

[jzaefferer]

@asped08 you should check out https://github.com/js-reporters/js-reporters - starting with the readme there, then going into the open issues, there's plenty to read.

[jzaefferer]

Based on a discussion on IRC about "why would a reporter consume TAP?", I think there are multiple scenarios that we should at least consider. For QUnit we currently have a pretty simple model for just one abstraction layer:

Runner -> Reporter

Runner runs tests, reporter registers event handlers to output result, usually via HTML reporter in the browser, but could also be on CLI etc.

When running tests in CI envs, the often an intermediate data format that is picked up by something else:

Runner -> Data -> Reporter

Here the runner could write JUnit XML to disk (that would effectively be another reporter, though), which then is picked up by a Jenkins plugin that visulizes test failures over time. Instead of xml, this could also use TAP. Instead of a Jenkins plugin, this could get visualized using a HTML reporter. In that case, we'd have TAP fed into a reporter.

The jsreporter project is currently mostly about the runner interface that provides the data in the first place. That's an important aspect and worth focusing on. It would also be great to do research on other data formats and interchangable reporters (e.g. around node-tap).

[fcarstens]

Thanks for the clarification!
Looking at @JamesMGreene's work, wouldn't it be something like

Runner -> Standardized Data/Interface -> TAP-Reporter -> consumer (e.g. Jenkins)

My point on IRC was that the TAP-Reporter would be one of the (interchangeable) reporters, as in http://mochajs.org/#tap-reporter and not directly part of the standardized interface.

[jzaefferer]

Yes, I agree, that's a better description.

[leobalter]

We're definitely using js-reporters as our standard reporter interface.