Recipe for handling concurrency and atomic tests

[sindresorhus]

Concurrency complicates some things like creating fixtures on disk, database, services, etc. Would be nice to document best-practises for handling this.

For example, for disk fixtures the recommended best-practise is using temp files.

What kind of challenges did you have converting/creating tests to be atomic? Let's discuss.

From the gitter chat:

I have a few questions about Ava.
Firstly good idea, my tests take a while to run, and they could all be
run concurrently in a fraction of the time.
Currently I use mocha. The Ava project would run concurrent tests. But a great
number of tests that currently exist are set up to populate DBs etc with data then
perhaps an api is hit and the expected data that is received is tested against
that stuffed into the DB.
Some tests hit remote APIs. Some hit local ones. Although running things concurrently
does make things faster it also makes testing trickier. Have thoughts been given
to this?
I am quite happy to start making contributions by explaining the concurrency in the
readme.

[sindresorhus]

This should also include handling dependency mocking.

From Gitter:

any best practice or suggestion on how to handle external dependencies while keeping tests running in parallel? Currently I'm using babel-plugin-rewire and on each test I call a function that overwrite the previous rewire. However the imported module is of course global and I may end up overwriting a rewire while a test is running in parallel that is not good... What exactly happens when ava spawns a new process for each test? If the imports are going to be resolved each time, maybe this is not a problem, otherwise I'd need a solution (maybe using require inside a function that will return the mocked dependency...) any experience on similar situations?

[novemberborn]

Ideally each test file controls its own fixtures (e.g. different database name from other test files). If that's not possible you'd have to run a setup process before invoking AVA and a teardown process after. I wonder if we could support global --setup/--teardown options.

[markelog]

It seems you would need to create a database and user with correct permissions (or use the root one) for every test file if run with serial mode otherwise you would need to create it for every test. And your app have to have the ability to change connection settings dynamically, like with arguments or something.

There is also question about selenium like e2e testing. Not sure how ava would fit.

With databases it is getting interesting, initially i didn't think it would be applicable... but selenium thing seems like a problem.

I think though those all should be different recipes - database, mocking, e2e, etc

[nfcampos]

for tests that needed do stuff with a mongodb database (with mongoose) I used mockgoose to mock the database connection (so that the tests aren't potentially changing the same db models) and created a file that populates the db with sample models common to all tests. This way all the tests run concurrently happily without conflicting with each other. I can simplify it and write up a recipe if you guys want

[markelog]

I can simplify it and write up a recipe if you guys want

I would love that

[nfcampos]

I'll see if I can find some time :)

[nfcampos]

one inconvenience with mockgoose is that it currently has a bug that prevents it from working with ava. There's an open PR to fix it but it hasn't been merged in yet. For now I've had manually patch the installed mockgoose to be able to run my tests. (this: mccormicka/Mockgoose#204)

[sibelius]

@nfcampos could u post an example of how to use ava with mockgoose ?

[nfcampos]

@sibelius

very generally there's a couple things to be aware of

• mock mongoose before calling mongoose.connect
• after mocking (mockgoose(mongoose)) call mongoose.connect
• after that and before the tests run populate mongoose with some models you'll use in the tests if you need to
• if some of your tests change the contents of the db you'll need to run all of the tests that use the db serially, otherwise you'll get very confused with some test changing the db while another test is running.

[sibelius]

can't I have a different "database" for each test?

[nfcampos]

@sibelius answered you on the other thread

[osdiab]

reviving an old discussion, but i've hacked together something for my company with a jest test run setup script that relies on its JEST_WORKER_ID environment variable + the maxWorkers config variable in setup scripts, to spin up a single db instance per worker thread, taking advantage of the fact that within a single worker (including across files) tests are not run concurrently.

I'd rather use ava though, so if i switch it I'd need to benchmark how much setting up a separate db per file would harm db test run perf (besides dev time to change to this paradigm). Does ava have similar environment variables/setup test functionality that I could potentially use to reuse the existing "1 db per worker" logic?

EDIT: seems that the require config array would be a reasonable place to put worker setup, though printing out the environment looks like ava includes nothing in there for knowing the number of workers / current worker. though maybe i can make a test db tagged by that random value, save that to a global variable, and then access it from within tests, that could work potentially?

hard to find docs for stuff like this that plays around with the internals of the library. or alternatively, does the recent shared workers functionality provide a potential alternative? though sounds like that'd be more like making one shared db that all the parallel workers would have to share, like taking turns grabbing a mutex to access it; sounds not great so probably not that lol #2605

[novemberborn]

@osdiab would you mind starting a discussion for this?