Add ability to run tests in a mocha instance multiple times #4234

nicojs · 2020-04-21T20:31:18Z

Description of the Change

These changes allow mocha to run multiple times with the same test suite programmatically (without reload). It does this with minimal code changes.

It also adds to validations, for when people call mocha.run at the wrong time.

For more details and reasoning, please see #2783 (comment)

Fixes #2783

Alternate Designs

Alternatively, we could have totally removed all state from tests, suites and runnable, but we don't want to change the entire way of keeping state. Please see #2783 for the entire history.

Why should this be in core?

A lot of people seem interested in running a test suite multiple times without reloading files from disk. See #2783. I personally will use it to implement mutation switching in Stryker

Benefits

Huge performance gain for my mutation testing use case. Other people have valid use cases as well.

Possible Drawbacks

It should be backward compatible, but since I had to rewrite parts in the core, I might have introduced undesired side effects 🤷‍♂️. Although the integration test suite is awesome!

Applicable issues

#2783

Todo:

jsf-clabot · 2020-04-21T20:31:21Z

All committers have signed the CLA.

coveralls · 2020-04-22T06:42:29Z

Coverage increased (+0.2%) to 93.268% when pulling f3f82c2 on nicojs:feat/hot-reload into 240cb3d on mochajs:master.

nicojs · 2020-04-22T09:22:54Z

Coverage increased (+0.2%) to 93.213%

You're welcome 💁‍♂️

nicojs · 2020-04-22T10:18:19Z

@boneskull the documentation website build failed because of an http error, not my doing. Please rerun that job if you can, thanks!

afonsojramos · 2020-04-22T15:40:48Z

I will be paying close attention to this PR, for now, it looks good!

boneskull · 2020-04-23T18:54:10Z

Huh, I am not sure how to re-deploy this on netlify. I don't see the option.

boneskull · 2020-04-23T19:06:26Z

browser-entry.js

+ * Implements listenerCount for 'uncaughtException'.
+ */
+
+process.listenerCount = function(name) {


NOTE: this is an observation and not a request for changes.

I don't love how process is shimmed here. we might be able to get away with changing the prototype to the EventEmitter shim (loaded via browserify via events, because process should be an EventEmitter.

Yeah I was surprised that that wasn't yet the case.

boneskull

Thanks, great work here! good tests too.

I have a couple concerns which I've commented on, but overall looks excellent. If you're unable to do further work on it, I think it could land as-is, and I could take care of those myself later. I'd like to see if #4245 can consume it.

lib/errors.js

lib/mocha.js

lib/runner.js

boneskull · 2020-04-23T19:50:53Z

lib/suite.js

+  });
+  this.tests.forEach(function(test) {
+    test.reset();
+  });


what of hooks?

I'm not 100% sure I know what you mean with hooks, but I think it is beforeEach and friends.

I super duper don't want them to be cleaned. They should stay, just like the test fn's themselves. We're going to need them for the next run.

yes, hooks are beforeEach etc. I think I am confused about why suites and tests are getting reset() called, but not hooks.

to be clear: I'm still confused.

I think I now understand your criticism. Hooks have state as well. And they inherit from Runnable. Didn't realize. Will add it when I have some time. Thanks for pointing it out!

test/unit/runnable.spec.js

test/integration/fixtures/multiple-runs/multiple-runs-different-output.fixture.js

nicojs · 2020-04-24T17:09:20Z

I've implemented (most of) the requested changes.

I've also added one last bit. When disposed of, a Mocha instance will now unload files internally. That makes a lot of sense to me. If someone wants to create a new mocha instance in the same process and add the same files again, he should be able to.

boneskull · 2020-04-24T19:19:06Z

ty for the update. looks like there's a conflict.

nicojs · 2020-04-24T21:00:41Z

Conflict has gone 🤸‍♂️

boneskull · 2020-04-24T23:32:41Z

@nicojs there's another conflict 😜

nicojs · 2020-04-25T08:37:39Z

You're a busy guy. Fixed once more 🤸‍♂️

nicojs · 2020-04-26T09:21:02Z

You're quite good at keeping me busy 🙊

I've implemented the reset on hooks. I saw that state and err could also be set on a test/hook during a run, so I've added those fields to Runnable.reset(). I've also solved the merge conflict.

As far as I know, I've fixed all issues/recommendations.

afonsojramos · 2020-04-29T16:00:50Z

lib/errors.js

+    default:
+      throw new Error('unknown pluginType "' + pluginType + '"');
+  }
+


You seem to have missed a closing bracket.

Yes, fixed now. Did a merge in the browser 🤷‍♂️

Fixed now. I'm hoping you guys do squash commit? If not, I could do it as well.

yes, will squash.

afonsojramos · 2020-04-29T19:53:28Z

@boneskull can you take a look if it is good to go? 🚀

boneskull · 2020-04-29T21:57:03Z

Seems to be yet another conflict.

boneskull · 2020-04-29T22:00:03Z

I would like to call this a "bugfix", since it really always should have worked properly. But I could see how others may consider this a feature. Thoughts, @mochajs/core?

EDIT: either way, this shouldn't be breaking.

boneskull · 2020-04-29T22:21:30Z

Something's gone wonky with the changesets here.

I tried to rebase back onto master, but that didn't work, and I had more conflicts. then I did a merge via the GitHub UI to resolve those conflicts, and now I just have...more conflicts.

@nicojs Sorry, I tried to help. You'll need to pull your branch again from your fork, and pull master as well, then retry a rebase back onto master.

boneskull · 2020-04-29T22:23:34Z

It looks as if there are changesets in here that shouldn't be in here;
66d9b29 and later. A rebase will need to ensure those changesets don't end up in this PR.

@nicojs I'll give you a chance, but please let me know if you want me to give it another go. I may have screwed it up myself!

Might even be best to not pull your branch back down, and instead just retry a rebase onto master from your working copy's branch.

Rename since we cannot dispose the entire mocha instance after a test run. We should keep the process.on('uncaughtException') handlers in place in order to not break existing functionality.

boneskull · 2020-05-07T23:24:00Z

Will give @juergba 48h to look at this; if he declines to review I'll merge it

boneskull · 2020-05-07T23:25:01Z

This is one of those "missing features" that is only a bug because it was missing. 😄

nicojs · 2020-05-08T10:03:55Z

True.

To be fair, I think we same problem in Stryker itself. Creating an instance of Stryker and running mutation testing twice is probably not working 🙄 Just don't tell anyone 🤐

nicojs · 2020-05-11T12:57:29Z

So... I guess this will be merged soon 🤗

boneskull · 2020-05-11T15:14:56Z

yes

boneskull · 2020-05-11T23:08:15Z

So it looks like I can't actually use this for the parallel runner. What I had hoped to do is re-use the same Mocha instance... except with different files (specifically, a single file) on every run.

@nicojs What I'm trying to do is:

Initialize a Mocha instance. Set cleanReferencesAfterRun to false.
Repeat for n files:
1. Add a file to Mocha instance
2. call Mocha#run()
3. Upon completion, "reset" Mocha and remove all files from the Mocha#files array.

I attempted to "reset" by calling dispose() upon completion (step 2.iii.). But this fails at the next run, because:

Mocha instance is already disposed, cannot start a new test run. Please create a new mocha instance. Be sure to set disable `cleanReferencesAfterRun` when you want to reuse the same mocha instance for multiple test runs

Of course, if I don't call dispose(), things get really confused due to extra hooks hanging around.

Why can't I reuse the Mocha instance after dispose()?

boneskull · 2020-05-11T23:09:20Z

Should I just not call cleanReferencesAfterRun? Would it work then?

boneskull · 2020-05-11T23:09:30Z

(I suppose I'm rubbber-ducking now...)

boneskull · 2020-05-11T23:19:25Z

I think, maybe, that it wasn't the problem you were trying to solve. but it seems to get most of the way there.

boneskull · 2020-05-11T23:58:18Z

OK, so, I am thinking it's really not worth the bother to try to reuse Mocha instances in the manner I'm trying (in #4245), even with your code now merged. Too many things to worry about. Like dumping files out of the module cache--but we're still running root-level hooks from those files, root-level hooks being applied nondeterministically, etc. I mean, it's possible, just difficult to do correctly (for some definition of correctly).

anyway, thanks for the PR 😄

nicojs · 2020-05-12T00:08:16Z

Yeah, it wasn't the problem I was trying to solve. The reason I added the behavior of throwing an error on run after dispose was just because thought that made the most sense. Feel free to change that behavior, no harm done since it wasn't released yet, right? As long as I can run the exact same suite without reloading files, I'm happy ;)

…loses #2783 * Add ability to run tests in a mocha instance multiple times * Rename `autoDispsoe` to `cleanReferencesAfterRun`, Rename since we cannot dispose the entire mocha instance after a test run. We should keep the process.on('uncaughtException') handlers in place in order to not break existing functionality. * Allow `unloadFiles` to reset `referencesCleaned`. * Complete rename of _cleanReferencesAfterRun * Add integration test for running a suite multiple times * improve api docs * Docs: fix dead link * Make sure tests run on older node versions * Remove `.only` 😅 * Implement `process.listenerCount` in the browser * Implement mocha states in a finite state machine * Fix some small remarks * Make integration tests more damp * Keep `Runner` api backward compatible * Unload files when disposed * Runnable.reset should also reset `err` and `state` * Also reset hooks Co-authored-by: Christopher Hiller <[email protected]>

birtles · 2020-05-26T00:02:58Z

This appears to cause breakage for me:
https://github.com/birtles/rikaichamp/runs/707671695#step:5:71
I am getting createMochaInstanceAlreadyRunningError when trying to update mocha.

I haven't had a chance to debug why yet. Any tips would be appreciated.

Fix issue with ability run tests in multiple times mochajs/mocha#4234

craigtaub · 2020-05-29T13:30:08Z

@birtles Please could you submit this bug as a new issue? Please include details of versions as well as how you are calling mocha. Thanks

birtles · 2020-06-02T02:10:42Z

@birtles Please could you submit this bug as a new issue? Please include details of versions as well as how you are calling mocha. Thanks

Hi. I went to submit a bug but the bug form was requiring a little bit more time than I can currently afford, unfortunately. I discovered that dropping a call to mocha.run() resolved the issue for me. I have no idea why the call to mocha.run() was needed in the first place.

For what it's worth, I'm running with karma and karma-firefox-launcher.

audioscavenger · 2020-10-26T22:54:24Z

i updated the thread #2314 and I found out that there must be a bug in the options parser.
by forcing _cleanReferencesAfterRun = false after you instantiated a new mocha, it actually let you re-run your tests! lost 2 hours on this one. You're welcome.

var mocha = new Mocha(mochaOptions)
mocha._cleanReferencesAfterRun = false

nicojs force-pushed the feat/hot-reload branch from 26ec939 to 6e8d72c Compare April 22, 2020 08:46

nicojs marked this pull request as ready for review April 22, 2020 09:06

nicojs mentioned this pull request Apr 23, 2020

Implement mutation switching 👽🔀 stryker-mutator/stryker-js#1514

Closed

79 tasks

boneskull mentioned this pull request Apr 23, 2020

add Root Hook Plugins #4244

Merged

boneskull reviewed Apr 23, 2020

View reviewed changes

afonsojramos suggested changes Apr 29, 2020

View reviewed changes

boneskull added type: bug a defect, confirmed by a maintainer semver-patch implementation requires increase of "patch" version number; "bug fixes" labels Apr 29, 2020

boneskull force-pushed the feat/hot-reload branch from 1e28dea to 70f4554 Compare April 29, 2020 22:17

nicojs added 3 commits April 30, 2020 13:29

Add ability to run tests in a mocha instance multiple times

3a58213

Rename autoDispsoe to cleanReferencesAfterRun,

89279b9

Rename since we cannot dispose the entire mocha instance after a test run. We should keep the process.on('uncaughtException') handlers in place in order to not break existing functionality.

Allow unloadFiles to reset referencesCleaned.

077560b

Merge branch 'master' into feat/hot-reload

3eb8ac1

boneskull added type: feature enhancement proposal semver-minor implementation requires increase of "minor" version number; "features" and removed type: bug a defect, confirmed by a maintainer semver-patch implementation requires increase of "patch" version number; "bug fixes" labels May 7, 2020

boneskull merged commit fbe3ce4 into mochajs:master May 11, 2020

craigtaub added this to the next milestone May 21, 2020

wKich added a commit to creevey/creevey that referenced this pull request May 27, 2020

fix: 🐛 soft-freeze mocha version on 7.1

5aa3f57

Fix issue with ability run tests in multiple times mochajs/mocha#4234

wKich added a commit to creevey/creevey that referenced this pull request May 27, 2020

fix: 🐛 soft-freeze mocha version on 7.1

06b9574

Fix issue with ability run tests in multiple times mochajs/mocha#4234

craigtaub mentioned this pull request Jun 16, 2020

Running the test suite twice skips tests that were already required #736

Closed

nicojs mentioned this pull request Aug 21, 2020

Implement hot reload for mocha test runner stryker-mutator/stryker-js#2413

Closed

kanej mentioned this pull request Dec 9, 2021

Run tests twice NomicFoundation/hardhat#2140

Merged

3 tasks

Omrisnyk mentioned this pull request Dec 7, 2023

[Snyk] Fix for 1 vulnerabilities Omrisnyk/npm-lockfiles#131

Open

tpicard586 mentioned this pull request Jul 21, 2024

[Snyk] Upgrade mocha from 7.1.1 to 7.2.0 tpicard586/damn-vulnerable-defi#3

Open

Add ability to run tests in a mocha instance multiple times #4234

Add ability to run tests in a mocha instance multiple times #4234

Conversation

nicojs commented Apr 21, 2020 • edited Loading

Description of the Change

Alternate Designs

Why should this be in core?

Benefits

Possible Drawbacks

Applicable issues

Todo:

jsf-clabot commented Apr 21, 2020 • edited Loading

coveralls commented Apr 22, 2020 • edited Loading

nicojs commented Apr 22, 2020

nicojs commented Apr 22, 2020

afonsojramos commented Apr 22, 2020

boneskull commented Apr 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boneskull left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicojs commented Apr 24, 2020 • edited Loading

boneskull commented Apr 24, 2020

nicojs commented Apr 24, 2020 • edited Loading

boneskull commented Apr 24, 2020

nicojs commented Apr 25, 2020

nicojs commented Apr 26, 2020

Choose a reason for hiding this comment

nicojs Apr 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afonsojramos commented Apr 29, 2020

boneskull commented Apr 29, 2020

boneskull commented Apr 29, 2020 • edited Loading

boneskull commented Apr 29, 2020

boneskull commented Apr 29, 2020

boneskull commented May 7, 2020

boneskull commented May 7, 2020

nicojs commented May 8, 2020

nicojs commented May 11, 2020

boneskull commented May 11, 2020

boneskull commented May 11, 2020 • edited Loading

boneskull commented May 11, 2020

boneskull commented May 11, 2020

boneskull commented May 11, 2020

boneskull commented May 11, 2020

nicojs commented May 12, 2020

birtles commented May 26, 2020

craigtaub commented May 29, 2020

birtles commented Jun 2, 2020

audioscavenger commented Oct 26, 2020

nicojs commented Apr 21, 2020 •

edited

Loading

jsf-clabot commented Apr 21, 2020 •

edited

Loading

coveralls commented Apr 22, 2020 •

edited

Loading

nicojs commented Apr 24, 2020 •

edited

Loading

nicojs commented Apr 24, 2020 •

edited

Loading

nicojs Apr 29, 2020 •

edited

Loading

boneskull commented Apr 29, 2020 •

edited

Loading

boneskull commented May 11, 2020 •

edited

Loading