Proposal: Adding a built-in test runner

[jasnell]

At the risk of opening a whole can of worms given that literally everyone gets super opinionated about test runners... I'd like to propose that we add a built-in test runner to Node.js.

Specifically allowing for something like node --test foo/* to run all tests found in the foo directory ... or node --test foo.js to run all tests found in the foo.js file, etc.

Obviously, this begs the question: Which test runner do we go with. There are options.

1. Vendor in an existing test runner (in which case which should we use?) ... Note: this is not an invitation to start advocating for your specific favorite test runner in this thread. At this point we just need to decide if vendoring in an existing runner is the right choice. We can bike shed on exactly which one that should be later.

2. Implement our own minimalistic test runner with the specific goal of it being extremely small and intentionally lite on features.

[Jamesernator]

I think the most important thing is the ability to scaffold on whatever Node provides that would allow test runners with more features to easily build on top of so that more powerful tests could be enabled.

As one example, suppose someone is wanting to do UI testing with playwright or similar, then they probably want to have certain things like pages made available to each test. Some way of adding things to tests would be neccessary for such things to be ergonomic.

I feel like what would be healthy for the community is for Node to specify a test format that can be scaffolded on top of and provide a basic default implementation on top of that. By having a community standard test format different testing tools could accept the same test files while providing their own features such as optimizations, extra assertions, improved debugging, browser integration, etc etc.

[Trott]

@nodejs/testing

[thebergamo]

I totally agree with @Jamesernator in the regard of creating some base tooling for those test runners be built on top and provide other more advanced features in the user land.

If we look in some of the "modern" programming languages like Rust and Go they already have something built in their std library that can provide out of the box such functionality (limited, but still powerful)

For long time we have had many important and dominant libraries being created as NPM modules so users would be able to choose their own flavors, but if Node.js itself could provide some interfaces/standards itself for those runners be built on top the interoperability between them would be improved significantly and the configuration hassle would be decreased also when thinking in simple or smaller cases for experimentation.

From my user perspective shipping an existing test runner would end up creating other issues for later on in case this runner goes away or a newer and modern one comes out and some requests to "adopt" it in the core could be avoided.

Having a clear state and for sure using those existing as a base for create this abstration would be wider accepted and even adopted for the existing test runners.

[vdeturckheim]

@jasnell does it have to be a cli option or could it be a core lib

// test/foo.js
import { it, describe } from '@node/test';

and call node test/foo.js ? I guess there must be a flag to allow running multiple files in a directory now I write it 🤔

[targos]

@vdeturckheim It probably needs to be both (a flag to enable certain behaviors like watching for changes or coverage output and a core lib for test utilities).

[jasnell]

I would say combination of CLI flags with API, yes.

Specifically:

• CLI for indicating that we want to specifically run tests in a file. It doesn't really need to run all tests in a directory. That's something userland can add. Just something like node --test foo.js == run all tests found in foo.js
• CLI for configuring how to output test results.
• API for declaring tests with support for:
  • Individual tests e.g. test(() => { ... }, 'it works!')
  • Groups of related tests
  • Setup and teardown per test and per group

We already have the built in assert module to use with it. And third parties can extend from there.

[bengl]

There's certainly room for improving support for testing. A limited subset of the functionality provided by all/most test frameworks makes sense to have in Node.js core.

Some further thoughts, some echoing what others have already said in this thread:

• Vendoring an existing test library is a favourite-picking exercise that's going to be fraught with pain, due to rather diverse state of Node.js testing today.
• As previously described here, a minimal runner command (e.g. node --run-tests test/**/*.js) could be used to run test files and determine their pass/fail status based on exit code.
  • This could look very similar to the Node.js core test suite.
  • TAP output is mostly ubiquitous, and so should be the native output here. Userland tools exist for making this more palatable to end-users. (e.g. node --run-tests test/**/*.js | npx tap-colorize)
  • It could also detect TAP output from individual tests and output them as subtests.
  • Options could be added for parallelism, etc.
  • The test files glob could have a default.
• An assertion API is already included. It would be preferable to extend it rather than replace it.
• Agreeing on an API to organize tests might not be easy, but it might be easier to provide some helper/building-block functions to build a test framework out of. For example:
  • A "single test runner" that runs a function, creating a pass/fail status based on whether it throws, returns a promise that rejects, or calls a callback with an error.
  • Helpers for producing TAP output. Note that I mean strictly for producing the output, and not for running tests or organizing test code.

Here's a sketch of what a super-minimalist test library built on such tools could look like:

import { createTest, TAP } from 'assert'

const tests = {}

export function test(name, fn) {
  tests[name] = createTest(fn)
}

export async function run() {
  const tap = new TAP(process.stdout)
  tap.begin(Object.keys(tests).length)
  process.exitCode = 0
  for (const testName in tests) {
    try {
      await tests[testName].run()
      tap.pass(testName)
    } catch (err) {
      tap.fail(testName, err)
      process.exitCode += 1
    }
  }
  tap.end()
}

[cjihrig]

import { createTest, TAP } from 'assert'

I think it would be best if the testing APIs and assertion APIs were kept separately.

I think it would also be preferable if each test ran in a separate process and globals were not altered (or new globals introduced).

[devsnek]

I think at the core here we should definitely not get complex. rust's test runner really speaks to me. you define some functions that pass on return and fail on panic (throw in js). the test name is just the function name. then we just need a standard tap output and a super minimal human readable output. if you want to get fancy, just compose the test function api with your own api. then you just glob with node --test and node defines the test register function and you're done.

import { test } from 'assert or test or smth';

test(function foo() {
  // ...
});

test(async function bar() {
  // ...
});

this also doesn't force tests to be run in the same process/context/etc. I can see us setting up a new node main context to run each function, especially now that we have snapshots. we don't have to do this though.

[jasnell]

Ok, so from the feedback so far, I think we can answer two specific (and important) questions:

1. Yes, we should add a test runner to core.
2. No, we should not vendor one in but instead create as minimal of a bespoke runner as we can.

Big +1 on keeping it simple.

@cjihrig:

I think it would be best if the testing APIs and assertion APIs were kept separately.

I agree. However, something like import { test } from 'assert/test' would accomplish that. I don't think we should add a new top-level module.

@cjihrig:

I think it would also be preferable if each test ran in a separate process and globals were not altered (or new globals introduced).

I don't think we need every test to be in a separate process. If we stick the the idea that node -test foo.js just runs all the tests that are found in foo.js, then all of those will be run in a single process. If I have a different set of tests that I'd like to run on their own, I just put those in a different file. It's essentially the same as what we do in Node.js own test/parallel/*.

@devsnek :

I think at the core here we should definitely not get complex. rust's test runner really speaks to me. you define some functions that pass on return and fail on panic (throw in js).

Big +1 but I think we do need to have a separate test label. The function name itself is not expressive enough.

import { test } from 'assert/test';

test(() => { /** ... *// }, 'The thing and the other thing do a thing unlike the other other thing');

test(() => { /** ... *//}, 'The thing when modified by this thing, does something else unlike the original thing');

@devsnek:

this also doesn't force tests to be run in the same process/context/etc. I can see us setting up a new node main context to run each function, especially now that we have snapshots. we don't have to do this though.

We also have the option of running each test within a file in its own worker_thread. This can be controlled by the API:

test('/* some test code */', 'a test that runs in a worker', { isolation: 'worker' }); // other values for `isolation` could be 'context', 'process', etc)

@bengl:

TAP output is mostly ubiquitous

Big +1 on just adopting TAP as the output format.

[thebergamo]

Quick note, that for me it feels a bit strange having a "test" lib inside an assertion one, usually we have it in the opposite or at least it's how we're used to.

[cjihrig]

I've been thinking about this some more and started some implementation work (help is welcome - it's currently in my fork, but would be happy to put this in a fork under the nodejs org). Here are some more thoughts I've come up with, including what has already been stated earlier in this thread:

How to run a test file?

• A test file should run without using a test runner. In other words, if I run node test.js, Node should execute the file as it would any other file, but if you are using the testing API in the file, it should still report test results.
• A test file should run via a simple test runner built into Node. The test runner would be executed by running node --test pattern. More details on the test runner itself in the next section.

How should the test runner work?

Node should execute zero or more test files when started with the --test flag. This exposes one current limitation in Node - the lack of glob support. This is something that needs to be addressed one way or another - vendor a module, build our own, do not support globs, use regular expressions instead, etc.

Each test file should be run in isolation. This can be done in a variety of ways - separate context, worker thread, child process. I propose using a separate child process for each file for maximum isolation (for example, a bad test file cannot crash the test runner process).

How are test results reported?

TAP seems to be the agreed upon way here.

What does the programmatic API look like?

I propose a minimal test(name[, options], fn), where name is the test's name in the output report, options can be used to configure the test if necessary, and fn is the test function. fn should take a context object as an argument. I haven't thought a ton about this object yet, but I think we should include one for test utilities like skipping a test or creating child tests.

When test() executes, it registers the test, but does not actually run it. The tests are actually run later either an auto start mechanism or programmatically. This seems to be a fairly common approach based on the test runners I looked into.

Another open question here is - how much effort should we put into isolating tests in the same file? We want them to be able to share some state such as a database connection (think before(), beforeEach(), etc.). However, what do we want to do if a test creates a long running timer that escapes the test? Do we really need to care that much? My opinion is that we don't need to care that much, at least for an initial version.

How to expose the API?

The general consensus earlier in this thread appears to be that the assert module should not export the testing API. In my opinion, Node should export a new top level module, but only via the node: prefix, eliminating any issues relating to existing modules in the ecosystem.

What should we strive to do or not do?

We should strive to avoid surprising behavior such as modified/injected globals. We should also aim to keep this simple, but extensible by userland (for example, custom reporters or adding custom utilities to the test context object).

Other things worth considering

Here are some features I think are worth at least discussing for an initial or future version.

• Built in code coverage.
• Snapshot testing support.
• Some story around transpilation?
• Additional execution modes (dry run, fail fast, etc.)
• Running individual tests based on the test name, regular expression, etc. This is different from file globbing.
• Skip and Only support.
• Test retry support.
• Test timeout support.
• Global leak detection.
• Handle leak detection.

[bcomnes]

What is the goal here and what would this enable that isn’t already available in userland?

Is ‘standardizing’ on a blessed test runner for use across the node ecosystem a goal? Seems like that would be a painful path and generally promote makework where things are rewritten for very little benefit to anyone. People seem uniformly reluctant to pick favorites so far but the process of adding this would essentially be that in some form or another.

Is the goal to establish a series of conventions around testing and test files? If so, why settle for custom glob rules that people have to maintain and cart around from project to project and vary from project to project. Languages (go being mentioned) that have have blessed test runners provide a series of conventions for test naming and location rules that are very rigid (generally a net positive). eg foo.go can have tests in foo_test.go as well as a few other well defined and conventional locations like example comments.

node --test foo.js

Is collocating tests inside of module code a goal? Is this not achievable already as is? Or is foo.js only containing tests in this example?

Or is the goal to increase / standardize test primitives? To what end? People desire to swap higher level runners on the same set of tests? The only area that pops out to me here is really low level APIs like coverage reporting, though solutions already exist here. For example tap ships reporting built in, but doesn’t currently work for type=module so you end up having to find an out of band solution like adding c8 to the command when you run. Having a built in api that existing user land runners could rely on to reliably delegate the repetitive task to the runtime instead of another userland tool seems like it could improve things here. Hypothetically here, tap could just get coverage from the runtime and not concern itself with finding it’s own solution. Maybe coverage and profiling primitives at the run time level could help here. Anything else?

So to summarize, I’m confused on the goals along the lines of the 3 following directions:

• is the goal to ship a built in test runner into the node runtime with its own set of test and organization conventions? (This seems unavoidably a favorite picking process, even if you redesign something totally new. I would personally favor just picking an existing runner and test convention over making something new. At least some population at that point wouldn’t have to rewrite their tests.)
• Is the goal to ship lower level runtime features to expand and unify testing capabilities (like coverage or profiling built into the runtime etc) and make it simpler to implement userland test runners?
• is the goal to establish test and test file organization conventions that userland runners would consume interchangeably? Not really sure what this even looks like but it seems people are thinking about something here.

[tony-go]

Hi there 👋

Regarding the comment of @cjihrig:

This exposes one current limitation in Node - the lack of glob support.

Just to mention that including glob/minimatch in the core was already asked here. Maybe It's worth it to add suggestions/concerns there?

[Trott]

@cjihrig If you haven't yet, you might want to look at what deno does.

[cjihrig]

Thanks @Trott. Yea, I've been looking at Deno, tap, tape, lab, mocha, etc. trying to see what features we can use in Node while trying to keep things fairly minimal.

[bcoe]

My two ¢s, I would love if we could extract a minimal runner based on the tests currently in Node.js' own codebase. We could then:

• cleanup and standardize Node.js' own tests based on this built-in test runner (this would be good for test health), and move away from test.py.
• expose the functionality to the world, so that someone small Node.js applications need not pull in a dependency for testing.

Node's built in assert is great at this point, a test runner seems like a lovely next step.

Edit: what if you do something like we've been doing for parseArgs, and develop a shim for the test-runner in the pkgjs org?

[targos]

I would love if we could extract a minimal runner based on the tests currently in Node.js' own codebase.

I agree. This would also allow to iterate on the test runner internally before exposing it to users

[cjihrig]

This would also allow to iterate on the test runner internally before exposing it to users

I'm hoping to have something in a state we could use internally in the next week or two.

[isaacs]

@cjihrig Your list of requirements above is remarkably similar to https://node-tap.org/#why-tap

As far as picking a convention, there are two main approaches in the JS community: tap-style and mocha-style.

// tap style: tap, tape, ava, etc.
const t = require('node:test') // <-- one thing returned
t.test('test that ends with .end()', t => { // <-- which looks like the args to children
  doSomething()
  t.end()
})
t.test('test that ends with Promise return', async t => {
  await doSomething()
})
t.test('test that ends with fulfilled plan', t => {
  t.plan(2) // expect 2 things
  t.doSomething()
  t.doSomething()
})
t.test('nesting', t => {
  t.test('nested', async t => { ... })
  t.end()
})

// mocha style: mocha, lab, jest, etc.
describe('a suite of tests', function () { // <-- globals injected
  describe('another layer of suite', function () {
    it('is the function that throws or doesnt throw', function () {
      // throw or don't, if you don't throw, you pass
    })
  })
})

Personally, I prefer the first option obviously, but the second is slightly more popular, likely owning to the popularity of mocha and jest within the frontend community, each historically getting a boost from jquery and react, respectively. Among node-only (or at least, node-first) modules, tap-style is very popular.

Built in code coverage.

Yes, please. Note that this is complicated! If you can't exclude known-impossible paths, you can't get to 100% coverage (which means, you don't really have full coverage support). It's super convenient to be able to handle the kind of reporting that nyc and tap do out of the box, and avoids needing to rely on a third party service like codecov or coveralls, just run the tests locally, generate a html page, open it up.

Snapshot testing support.

Check out how jest, ava, and tap do this (they're all different).

Some story around transpilation?

Thar be dragons. I keep thinking I got this working, and then something changes in the rube goldberg house of cards, and it all falls apart. The latest headache: typescript + "type": "module" in package.json, apparently it makes something blow up, idk why.

Additional execution modes (dry run, fail fast, etc.)

Not sure what a "dry run" would be, just figure out what test files will be run? It's not really safe to run a file and not run its tests, since the test setup might do something to alter the environment.

"Fail fast" is called --bail or -b or --bailout in tap parlance.

Other "modes" to consider:

• only: run tests marked as {only: true}, and no other tests
• watch: keep a process open, and re-run tests when a change is detected in any file they cover (and open a little repl to trigger tests manually, add new test files as they're created, etc.)

Running individual tests based on the test name, regular expression, etc. This is different from file globbing.

How do you imagine this working?

Test retry support.

If this means "run until pass, max X times", then strong objection here. This is an antipattern, and should not be supported.

If this means "run X times, and fail if it fails ever", then ok, this is useful 😅

If your test fails sometimes, it fails. Flaky tests should be fixed or removed.

Global leak detection.

I'm probably going to add this to node-tap soon. I have a bunch of modules that do it manually, and that's just silly.

Handle leak detection.

Leaking memory, or leaking fd/handles? Tap does this, but it'd be really nice if there was a better way than what it does now. It's gross.

For very large test suites, parallel test execution is also really important. That is, running multiple test files in parallel, up to os.cpus().length. Within a test file, it's very rare that you want to parallelize test execution. This is supported in tap, and the output is still consistent regardless of what runs in parallel, but within a single test file, the cases are almost always CPU-bound anyway (or, if you're running 8 files on 8 cpu's, you can't go any faster by parallelizing within one of them, anyway).

So, wishlist:

• handle/fd leak detection in process.on('beforeExit') (or process.exit, that'd be helpful too)
• built in coverage (if possible, with output that nyc can interpret), on by default, fail by default if coverage is not complete
• better stories around transpiled modules, sourcemaps, esm, and "type": "module"

[Raynos]

This builtin test runner looks very similar to a package I wrote in user space ( https://github.com/socketsupply/tapzero ). I approve of adding a lightweight test runner to core, this improves the out of the box behavior.

Thanks for doing this work.

[AlenDavid]

Hey guys, I just read this article about this feature and I enjoy it!

My concern is about skipping/making a test todo. In this next example (the one showed in the website), this is going to be the sintax to skip a test:

test('skip option with message', { skip: 'this is skipped' }, (t) => { // This code is never executed. });

I'm tempting to contribute so the test framework could, also, expose a skip method, so the sintax would look similar to:

test.skip('skip without option with message', (t) => { // This code is never executed. });

Let me know how I can contribute for this feature! Thank you!

[iansu]

@AlenDavid it's probably best to open a new issue suggesting this change.

This feature (the test runner) has been implemented so I'm going to close this issue.