🚀 Feature: Detect and report on invalid test pass/fail/pending state

[boneskull]

If an task created by a test would throw an exception (or rejection) and this test has already passed or failed, then Mocha's output gets wonky (using spec reporter, for example):

• the test name may repeat multiple times, with different results for each
• we can't just limit it, because suppression thereof means errors can't easily be cross-referenced by number
• the total pass/fail/pending count will be larger than the total number of tests executed, because a test will have been reported to be in multiple states
• multiple errors may be output for a single test

---

Here are two ideas to mitigate this:

1. Decouple any exception a test causes after it has completed from the test itself, for reporting purposes. Such errors would simply not count towards statistics, but would display nevertheless. The test, then, would not necessarily fail (unless it did originally), but the test run would fail.
   This could be significantly disruptive to the reporting system. Simply swapping a reporter out for another should not change the results!

2. Determine, by way of async hooks or otherwise, when a test is truly "complete". Don't record the test result until there are no further tasks in the queue.
   Difficult. May slow down some tests significantly. Would need to cancel the timeout when done() is called (or whatever); might want to record the delta between when the test finishes and when the test's tasks complete. Consuming async hooks / zones / realms etc. could either be great (eliminate many edge and corner cases) or terrible (increased complexity and black magic).

There's no end to problems from these types of errors. They shatter Mocha's assumptions about the current state of things; Mocha is trying to run tests serially, after all. Test B should never seem to fail because of something Test A did; and even so, we should know that it was Test A's fault.

---

Here's a philosophical question:

Can Mocha rely on the user to tell Mocha when a test completes, and how it completed?

[Nokel81]

This also seems quite interesting and does seem to be related to #3256. Re the philosophical question would it be possible to tell what async hooks were sent off within a test? If so could it be used that if the test has "completed" by normal means, ie done() has been called or if the test wasn't told to be async then the function completed, but there are still async hooks present then fail the test with a suitable error message saying that an async function did not complete before the test completed?

[ctaylo21]

I know this is a tricky issue, but there might be some potential insight someone can offer. Is there a way to debug these sort of errors? We have a tricky race condition that will sometimes cause a test to fail after it. In the spec reporter, it will mark the test as both passed and failed.

Is there a way to debug this? If I could at least see the exception, that would help debug which async operation is triggering the failure. Try/catch doesn't seem to work. Any ideas?

[Nokel81]

So, from the research that I have done I don't believe that the required information is available from the node runtime to prevent the double reporting but the error message should be able to be reported if it occurs before the suite has completed

[ctaylo21]

Found my issues. I was using an old version of mocha (>2.3) and when I upgraded to the latest version, I actually saw the errors outputted. On the older version the reporter just marked the test as failed with no error details. I appreciate the reply!