Support for retrying flaky tests

[Dhul-Husni]

Hi, we use qunit and we've recently been dealing with a lot of flaky tests. We loose a lot of productivity as the team has to monitor and rerun the CI build multiple times to get the test to pass. Does QUnit support retrying a failed test from config? or a hook before the test is done to confirm or retry the failed test?

Thank you!

[Krinkle]

@Dhul-Husni

Hi!

When QUnit is run by a developer in a browser during development, and there are failing test, than upon reloading the test runner it automatically remembers which tests failed last time, and re-runs those first. You can also use the module menu, filter field, or "Re-run" buttons to focus on a subset of the tests.

These features are decoupoeld from the HTML page however, and can also be used directly via QUnit.config by setting QUnit.config.module or QUnit.config.filter.

In the CI environment - it'll depend on how you run the tests to know how to pass this information. Is this a Node.js project, browser project, or both? What do you currently use to run the tests? Is it the QUnit CLI, Karma runner, qunit-puppeteer, etc?

[Dhul-Husni]

Thanks @Krinkle for replying so quickly, we use qunit or rather ember-qunit as the test runner, the application is an spa running ember. In dev mode we make use of filter & rerun as you’ve mentioned. In CI (circle CI) we run tests headless and the tests continue to run until the test queue is empty, even when a failure occurs, it then returns an exit code on whether all the tests passed or failed.

TBH here’s what I’m really trying to do, I am working on way to reduce the number of flaky tests in our codebase by a) detecting flaky tests b) automatically creating tickets to resolve the flaky tests.
The flaky tests detection will work by retrying the failed test (in CI) a few times, if the test passes after during a retry, a process is kicked off to create the tickets.

The way i hope to achieve this is by using hooks/callbacks in Qunit like beforeTestDone to achieve the above. The goal is to retry tests so developers don’t have to and to hook into the process and apply custom logic. Is this possible with Qunit hooks? Are there other ways to achieve this with Qunit?

Thank you!

[Krinkle]

@Dhul-Husni

Great, yeah, that should work. I suppose you'll first need to identify which tests failed (in a way that is programmatically understandable and deterministic), and then have a way to run only those.

For identification, a very crude approach might be to look for the module names (test suites) of a failing test. Depending on how large individidual test modules are, that might be granular enough for a speedy re-run. I don't know what formatting your build output currently uses, but module names are likely easiest to extract in some way. Or, if you want to write them to a file during the test run, then you could use a hook that does this. E.g. by adding the following to an init or bootstrap file that is required once before the first test in any run:

https://api.qunitjs.com/callbacks/

QUnit.moduleDone(function (details) {
  if (details.failed) {
    __phone_home('module with errors', details.name);
  }
});

It depends on how you'd like the "flaky test reducer" to operate, e.g. based on a build artefact file that it reads from failed CI run, or e.g. based on some kind of string pattern that it reads from the build output text, or e.g. fully integrated where there may be a late execution step or job within the same CI workflow that inspects a file or environment variable, and then based on that schedules the reducer system in the background.

This could involve e.g. console.log(), or fs.appendFileSync() or process.env.

Then on the other side of it, in the CI run spawned by the "flaky test reducer" for a given failed module, you'd inject it perhaps build parameter or env variable, or some other way you can pass data from CI into the ember-qunit process:

if (injected_to_reduce_flaky_module) {
  QUnit.config.module = injected_to_reduce_flaky_module;
}

And that will make it skip all other test modules and run only this one.

Alternatively, for greater granularity, one could also use testId.

QUnit.testDone(function (details) {
  if (details.failed) {
    __phone_home("failed test", details.testId);
  }
});

if (injected_flaky_testId) {
  QUnit.config.testId = [ injected_flaky_testId ]; // or perhaps: injected_flaky_testId.split( ',' );
}

Hope that helps!

/cc @rwjblue Would love to know if you have ideas on this as well, and possibly insights specific to ember-qunit!

[Dhul-Husni]

That is great help, thanks @Krinkle , should be enough to get me started!

[lifeart]

Wondering, for v3, is it possible to have api like this? https://playwright.dev/docs/test-retries