Skip to content

Commit

Permalink
test_runner: add initial code coverage support
Browse files Browse the repository at this point in the history
This commit adds code coverage functionality to the node:test
module. When node:test is used in conjunction with the new
--test-coverage CLI flag, a coverage report is created when
the test runner finishes. The coverage summary is forwarded to
any test runner reporters so that the display can be customized
as desired. This new functionality is compatible with the
existing NODE_V8_COVERAGE environment variable as well.

There are still several limitations, which will be addressed in
subsequent pull requests:

- Coverage is only reported for a single process. It is possible
  to merge coverage reports together. Once this is done, the
  --test flag will be supported as well.
- Source maps are not currently supported.
- Excluding specific files or directories from the coverage
  report is not currently supported. Node core modules and
  node_modules/ are excluded though.

PR-URL: #46017
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Geoffrey Booth <[email protected]>
cjihrig authored and MylesBorins committed Feb 18, 2023
1 parent c9d5dfd commit 6119289
Showing 16 changed files with 796 additions and 33 deletions.
12 changes: 12 additions & 0 deletions doc/api/cli.md
Original file line number Diff line number Diff line change
@@ -1228,6 +1228,17 @@ Starts the Node.js command line test runner. This flag cannot be combined with
See the documentation on [running tests from the command line][]
for more details.

### `--test-coverage`

<!-- YAML
added: REPLACEME
-->

When used in conjunction with the `node:test` module, a code coverage report is
generated as part of the test runner output. If no tests are run, a coverage
report is not generated. See the documentation on
[collecting code coverage from tests][] for more details.

### `--test-name-pattern`

<!-- YAML
@@ -2346,6 +2357,7 @@ done
[`unhandledRejection`]: process.md#event-unhandledrejection
[`v8.startupSnapshot` API]: v8.md#startup-snapshot-api
[`worker_threads.threadId`]: worker_threads.md#workerthreadid
[collecting code coverage from tests]: test.md#collecting-code-coverage
[conditional exports]: packages.md#conditional-exports
[context-aware]: addons.md#context-aware-addons
[debugger]: debugger.md
87 changes: 87 additions & 0 deletions doc/api/test.md
Original file line number Diff line number Diff line change
@@ -368,6 +368,54 @@ Otherwise, the test is considered to be a failure. Test files must be
executable by Node.js, but are not required to use the `node:test` module
internally.

## Collecting code coverage

When Node.js is started with the [`--test-coverage`][] command-line flag, code
coverage is collected and statistics are reported once all tests have completed.
If the [`NODE_V8_COVERAGE`][] environment variable is used to specify a
code coverage directory, the generated V8 coverage files are written to that
directory. Node.js core modules and files within `node_modules/` directories
are not included in the coverage report. If coverage is enabled, the coverage
report is sent to any [test reporters][] via the `'test:coverage'` event.

Coverage can be disabled on a series of lines using the following
comment syntax:

```js
/* node:coverage disable */
if (anAlwaysFalseCondition) {
// Code in this branch will never be executed, but the lines are ignored for
// coverage purposes. All lines following the 'disable' comment are ignored
// until a corresponding 'enable' comment is encountered.
console.log('this is never executed');
}
/* node:coverage enable */
```

Coverage can also be disabled for a specified number of lines. After the
specified number of lines, coverage will be automatically reenabled. If the
number of lines is not explicitly provided, a single line is ignored.

```js
/* node:coverage ignore next */
if (anAlwaysFalseCondition) { console.log('this is never executed'); }

/* node:coverage ignore next 3 */
if (anAlwaysFalseCondition) {
console.log('this is never executed');
}
```

The test runner's code coverage functionality has the following limitations,
which will be addressed in a future Node.js release:

* Although coverage data is collected for child processes, this information is
not included in the coverage report. Because the command line test runner uses
child processes to execute test files, it cannot be used with `--test-coverage`.
* Source maps are not supported.
* Excluding specific files or directories from the coverage report is not
supported.

## Mocking

The `node:test` module supports mocking during testing via a top-level `mock`
@@ -1215,6 +1263,42 @@ A successful call to [`run()`][] method will return a new {TestsStream}
object, streaming a series of events representing the execution of the tests.
`TestsStream` will emit events, in the order of the tests definition

### Event: `'test:coverage'`

* `data` {Object}
* `summary` {Object} An object containing the coverage report.
* `files` {Array} An array of coverage reports for individual files. Each
report is an object with the following schema:
* `path` {string} The absolute path of the file.
* `totalLineCount` {number} The total number of lines.
* `totalBranchCount` {number} The total number of branches.
* `totalFunctionCount` {number} The total number of functions.
* `coveredLineCount` {number} The number of covered lines.
* `coveredBranchCount` {number} The number of covered branches.
* `coveredFunctionCount` {number} The number of covered functions.
* `coveredLinePercent` {number} The percentage of lines covered.
* `coveredBranchPercent` {number} The percentage of branches covered.
* `coveredFunctionPercent` {number} The percentage of functions covered.
* `uncoveredLineNumbers` {Array} An array of integers representing line
numbers that are uncovered.
* `totals` {Object} An object containing a summary of coverage for all
files.
* `totalLineCount` {number} The total number of lines.
* `totalBranchCount` {number} The total number of branches.
* `totalFunctionCount` {number} The total number of functions.
* `coveredLineCount` {number} The number of covered lines.
* `coveredBranchCount` {number} The number of covered branches.
* `coveredFunctionCount` {number} The number of covered functions.
* `coveredLinePercent` {number} The percentage of lines covered.
* `coveredBranchPercent` {number} The percentage of branches covered.
* `coveredFunctionPercent` {number} The percentage of functions covered.
* `workingDirectory` {string} The working directory when code coverage
began. This is useful for displaying relative path names in case the tests
changed the working directory of the Node.js process.
* `nesting` {number} The nesting level of the test.

Emitted when code coverage is enabled and all tests have completed.

### Event: `'test:diagnostic'`

* `data` {Object}
@@ -1595,6 +1679,7 @@ added:

[TAP]: https://testanything.org/
[`--import`]: cli.md#--importmodule
[`--test-coverage`]: cli.md#--test-coverage
[`--test-name-pattern`]: cli.md#--test-name-pattern
[`--test-only`]: cli.md#--test-only
[`--test-reporter-destination`]: cli.md#--test-reporter-destination
@@ -1603,6 +1688,7 @@ added:
[`MockFunctionContext`]: #class-mockfunctioncontext
[`MockTracker.method`]: #mockmethodobject-methodname-implementation-options
[`MockTracker`]: #class-mocktracker
[`NODE_V8_COVERAGE`]: cli.md#node_v8_coveragedir
[`SuiteContext`]: #class-suitecontext
[`TestContext`]: #class-testcontext
[`context.diagnostic`]: #contextdiagnosticmessage
@@ -1613,4 +1699,5 @@ added:
[describe options]: #describename-options-fn
[it options]: #testname-options-fn
[stream.compose]: stream.md#streamcomposestreams
[test reporters]: #test-reporters
[test runner execution model]: #test-runner-execution-model
3 changes: 3 additions & 0 deletions doc/node.1
Original file line number Diff line number Diff line change
@@ -391,6 +391,9 @@ Specify the minimum allocation from the OpenSSL secure heap. The default is 2. T
.It Fl -test
Starts the Node.js command line test runner.
.
.It Fl -test-coverage
Enable code coverage in the test runner.
.
.It Fl -test-name-pattern
A regular expression that configures the test runner to only execute tests
whose name matches the provided pattern.
43 changes: 13 additions & 30 deletions lib/internal/process/pre_execution.js
Original file line number Diff line number Diff line change
@@ -20,6 +20,7 @@ const {
exposeInterface,
exposeLazyInterfaces,
defineReplaceableLazyAttribute,
setupCoverageHooks,
} = require('internal/util');

const {
@@ -66,15 +67,7 @@ function prepareExecution(options) {
setupFetch();
setupWebCrypto();
setupCustomEvent();

// Resolve the coverage directory to an absolute path, and
// overwrite process.env so that the original path gets passed
// to child processes even when they switch cwd.
if (process.env.NODE_V8_COVERAGE) {
process.env.NODE_V8_COVERAGE =
setupCoverageHooks(process.env.NODE_V8_COVERAGE);
}

setupCodeCoverage();
setupDebugEnv();
// Process initial diagnostic reporting configuration, if present.
initializeReport();
@@ -304,6 +297,17 @@ function setupWebCrypto() {
}
}

function setupCodeCoverage() {
// Resolve the coverage directory to an absolute path, and
// overwrite process.env so that the original path gets passed
// to child processes even when they switch cwd. Don't do anything if the
// --test-coverage flag is present, as the test runner will handle coverage.
if (process.env.NODE_V8_COVERAGE && !getOptionValue('--test-coverage')) {
process.env.NODE_V8_COVERAGE =
setupCoverageHooks(process.env.NODE_V8_COVERAGE);
}
}

// TODO(daeyeon): move this to internal/bootstrap/browser when the CLI flag is
// removed.
function setupCustomEvent() {
@@ -315,27 +319,6 @@ function setupCustomEvent() {
exposeInterface(globalThis, 'CustomEvent', CustomEvent);
}

// Setup User-facing NODE_V8_COVERAGE environment variable that writes
// ScriptCoverage to a specified file.
function setupCoverageHooks(dir) {
const cwd = require('internal/process/execution').tryGetCwd();
const { resolve } = require('path');
const coverageDirectory = resolve(cwd, dir);
const { sourceMapCacheToObject } =
require('internal/source_map/source_map_cache');

if (process.features.inspector) {
internalBinding('profiler').setCoverageDirectory(coverageDirectory);
internalBinding('profiler').setSourceMapCacheGetter(sourceMapCacheToObject);
} else {
process.emitWarning('The inspector is disabled, ' +
'coverage could not be collected',
'Warning');
return '';
}
return coverageDirectory;
}

function setupStacktracePrinterOnSigint() {
if (!getOptionValue('--trace-sigint')) {
return;
371 changes: 371 additions & 0 deletions lib/internal/test_runner/coverage.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,371 @@
'use strict';
const {
ArrayPrototypeMap,
ArrayPrototypePush,
JSONParse,
MathFloor,
NumberParseInt,
RegExp,
RegExpPrototypeExec,
RegExpPrototypeSymbolSplit,
StringPrototypeIncludes,
StringPrototypeLocaleCompare,
StringPrototypeStartsWith,
} = primordials;
const {
copyFileSync,
mkdirSync,
mkdtempSync,
opendirSync,
readFileSync,
} = require('fs');
const { setupCoverageHooks } = require('internal/util');
const { tmpdir } = require('os');
const { join, resolve } = require('path');
const { fileURLToPath } = require('url');
const kCoveragePattern =
`^coverage\\-${process.pid}\\-(\\d{13})\\-(\\d+)\\.json$`;
const kCoverageFileRegex = new RegExp(kCoveragePattern);
const kIgnoreRegex = /\/\* node:coverage ignore next (?<count>\d+ )?\*\//;
const kLineEndingRegex = /\r?\n$/u;
const kLineSplitRegex = /(?<=\r?\n)/u;
const kStatusRegex = /\/\* node:coverage (?<status>enable|disable) \*\//;

class CoverageLine {
#covered;

constructor(line, src, startOffset) {
const newlineLength =
RegExpPrototypeExec(kLineEndingRegex, src)?.[0].length ?? 0;

this.line = line;
this.src = src;
this.startOffset = startOffset;
this.endOffset = startOffset + src.length - newlineLength;
this.ignore = false;
this.#covered = true;
}

get covered() {
return this.#covered;
}

set covered(isCovered) {
// V8 can generate multiple ranges that span the same line.
if (!this.#covered) {
return;
}

this.#covered = isCovered;
}
}

class TestCoverage {
constructor(coverageDirectory, originalCoverageDirectory, workingDirectory) {
this.coverageDirectory = coverageDirectory;
this.originalCoverageDirectory = originalCoverageDirectory;
this.workingDirectory = workingDirectory;
}

summary() {
internalBinding('profiler').takeCoverage();
const coverage = getCoverageFromDirectory(this.coverageDirectory);
const coverageSummary = {
__proto__: null,
workingDirectory: this.workingDirectory,
files: [],
totals: {
__proto__: null,
totalLineCount: 0,
totalBranchCount: 0,
totalFunctionCount: 0,
coveredLineCount: 0,
coveredBranchCount: 0,
coveredFunctionCount: 0,
coveredLinePercent: 0,
coveredBranchPercent: 0,
coveredFunctionPercent: 0,
},
};

if (!coverage) {
return coverageSummary;
}

for (let i = 0; i < coverage.length; ++i) {
const { functions, url } = coverage[i];

if (StringPrototypeStartsWith(url, 'node:') ||
StringPrototypeIncludes(url, '/node_modules/') ||
// On Windows some generated coverages are invalid.
!StringPrototypeStartsWith(url, 'file:')) {
continue;
}

// Split the file source into lines. Make sure the lines maintain their
// original line endings because those characters are necessary for
// determining offsets in the file.
const filePath = fileURLToPath(url);
const source = readFileSync(filePath, 'utf8');
const linesWithBreaks =
RegExpPrototypeSymbolSplit(kLineSplitRegex, source);
let ignoreCount = 0;
let enabled = true;
let offset = 0;
let totalBranches = 0;
let totalFunctions = 0;
let branchesCovered = 0;
let functionsCovered = 0;

const lines = ArrayPrototypeMap(linesWithBreaks, (line, i) => {
const startOffset = offset;
const coverageLine = new CoverageLine(i + 1, line, startOffset);

offset += line.length;

// Determine if this line is being ignored.
if (ignoreCount > 0) {
ignoreCount--;
coverageLine.ignore = true;
} else if (!enabled) {
coverageLine.ignore = true;
}

if (!coverageLine.ignore) {
// If this line is not already being ignored, check for ignore
// comments.
const match = RegExpPrototypeExec(kIgnoreRegex, line);

if (match !== null) {
ignoreCount = NumberParseInt(match.groups?.count ?? 1, 10);
}
}

// Check for comments to enable/disable coverage no matter what. These
// take precedence over ignore comments.
const match = RegExpPrototypeExec(kStatusRegex, line);
const status = match?.groups?.status;

if (status) {
ignoreCount = 0;
enabled = status === 'enable';
}

return coverageLine;
});

for (let j = 0; j < functions.length; ++j) {
const { functionName, isBlockCoverage, ranges } = functions[j];

for (let k = 0; k < ranges.length; ++k) {
const range = ranges[k];

mapRangeToLines(range, lines);

if (isBlockCoverage) {
if (range.count !== 0 ||
range.ignoredLines === range.lines.length) {
branchesCovered++;
}

totalBranches++;
}
}

if (functionName.length > 0 && ranges.length > 0) {
const range = ranges[0];

if (range.count !== 0 || range.ignoredLines === range.lines.length) {
functionsCovered++;
}

totalFunctions++;
}
}

let coveredCnt = 0;
const uncoveredLineNums = [];

for (let j = 0; j < lines.length; ++j) {
const line = lines[j];

if (line.covered || line.ignore) {
coveredCnt++;
} else {
ArrayPrototypePush(uncoveredLineNums, line.line);
}
}

ArrayPrototypePush(coverageSummary.files, {
__proto__: null,
path: filePath,
totalLineCount: lines.length,
totalBranchCount: totalBranches,
totalFunctionCount: totalFunctions,
coveredLineCount: coveredCnt,
coveredBranchCount: branchesCovered,
coveredFunctionCount: functionsCovered,
coveredLinePercent: toPercentage(coveredCnt, lines.length),
coveredBranchPercent: toPercentage(branchesCovered, totalBranches),
coveredFunctionPercent: toPercentage(functionsCovered, totalFunctions),
uncoveredLineNumbers: uncoveredLineNums,
});

coverageSummary.totals.totalLineCount += lines.length;
coverageSummary.totals.totalBranchCount += totalBranches;
coverageSummary.totals.totalFunctionCount += totalFunctions;
coverageSummary.totals.coveredLineCount += coveredCnt;
coverageSummary.totals.coveredBranchCount += branchesCovered;
coverageSummary.totals.coveredFunctionCount += functionsCovered;
}

coverageSummary.totals.coveredLinePercent = toPercentage(
coverageSummary.totals.coveredLineCount,
coverageSummary.totals.totalLineCount
);
coverageSummary.totals.coveredBranchPercent = toPercentage(
coverageSummary.totals.coveredBranchCount,
coverageSummary.totals.totalBranchCount
);
coverageSummary.totals.coveredFunctionPercent = toPercentage(
coverageSummary.totals.coveredFunctionCount,
coverageSummary.totals.totalFunctionCount
);
coverageSummary.files.sort(sortCoverageFiles);

return coverageSummary;
}

cleanup() {
// Restore the original value of process.env.NODE_V8_COVERAGE. Then, copy
// all of the created coverage files to the original coverage directory.
if (this.originalCoverageDirectory === undefined) {
delete process.env.NODE_V8_COVERAGE;
return;
}

process.env.NODE_V8_COVERAGE = this.originalCoverageDirectory;
let dir;

try {
mkdirSync(this.originalCoverageDirectory, { recursive: true });
dir = opendirSync(this.coverageDirectory);

for (let entry; (entry = dir.readSync()) !== null;) {
const src = join(this.coverageDirectory, entry.name);
const dst = join(this.originalCoverageDirectory, entry.name);
copyFileSync(src, dst);
}
} finally {
if (dir) {
dir.closeSync();
}
}
}
}

function toPercentage(covered, total) {
return total === 0 ? 100 : (covered / total) * 100;
}

function sortCoverageFiles(a, b) {
return StringPrototypeLocaleCompare(a.path, b.path);
}

function setupCoverage() {
let originalCoverageDirectory = process.env.NODE_V8_COVERAGE;
const cwd = process.cwd();

if (originalCoverageDirectory) {
// NODE_V8_COVERAGE was already specified. Convert it to an absolute path
// and store it for later. The test runner will use a temporary directory
// so that no preexisting coverage files interfere with the results of the
// coverage report. Then, once the coverage is computed, move the coverage
// files back to the original NODE_V8_COVERAGE directory.
originalCoverageDirectory = resolve(cwd, originalCoverageDirectory);
}

const coverageDirectory = mkdtempSync(join(tmpdir(), 'node-coverage-'));
const enabled = setupCoverageHooks(coverageDirectory);

if (!enabled) {
return null;
}

// Ensure that NODE_V8_COVERAGE is set so that coverage can propagate to
// child processes.
process.env.NODE_V8_COVERAGE = coverageDirectory;

return new TestCoverage(coverageDirectory, originalCoverageDirectory, cwd);
}

function mapRangeToLines(range, lines) {
const { startOffset, endOffset, count } = range;
const mappedLines = [];
let ignoredLines = 0;
let start = 0;
let end = lines.length;
let mid;

while (start <= end) {
mid = MathFloor((start + end) / 2);
let line = lines[mid];

if (startOffset >= line.startOffset && startOffset <= line.endOffset) {
while (endOffset > line?.startOffset) {
// If the range is not covered, and the range covers the entire line,
// then mark that line as not covered.
if (count === 0 && startOffset <= line.startOffset &&
endOffset >= line.endOffset) {
line.covered = false;
}

ArrayPrototypePush(mappedLines, line);

if (line.ignore) {
ignoredLines++;
}

mid++;
line = lines[mid];
}

break;
} else if (startOffset >= line.endOffset) {
start = mid + 1;
} else {
end = mid - 1;
}
}

// Add some useful data to the range. The test runner has read these ranges
// from a file, so we own the data structures and can do what we want.
range.lines = mappedLines;
range.ignoredLines = ignoredLines;
}

function getCoverageFromDirectory(coverageDirectory) {
// TODO(cjihrig): Instead of only reading the coverage file for this process,
// combine all coverage files in the directory into a single data structure.
let dir;

try {
dir = opendirSync(coverageDirectory);

for (let entry; (entry = dir.readSync()) !== null;) {
if (RegExpPrototypeExec(kCoverageFileRegex, entry.name) === null) {
continue;
}

const coverageFile = join(coverageDirectory, entry.name);
const coverage = JSONParse(readFileSync(coverageFile, 'utf8'));
return coverage.result;
}
} finally {
if (dir) {
dir.closeSync();
}
}
}

module.exports = { setupCoverage };
41 changes: 40 additions & 1 deletion lib/internal/test_runner/harness.js
Original file line number Diff line number Diff line change
@@ -56,6 +56,44 @@ function createProcessEventHandler(eventName, rootTest) {
};
}

function configureCoverage(rootTest) {
if (!getOptionValue('--test-coverage')) {
return null;
}

const { setupCoverage } = require('internal/test_runner/coverage');

try {
return setupCoverage();
} catch (err) {
const msg = `Warning: Code coverage could not be enabled. ${err}`;

rootTest.diagnostic(msg);
process.exitCode = kGenericUserError;
}
}

function collectCoverage(rootTest, coverage) {
if (!coverage) {
return null;
}

let summary = null;

try {
summary = coverage.summary();
coverage.cleanup();
} catch (err) {
const op = summary ? 'clean up' : 'report';
const msg = `Warning: Could not ${op} code coverage. ${err}`;

rootTest.diagnostic(msg);
process.exitCode = kGenericUserError;
}

return summary;
}

function setup(root) {
if (wasRootSetup.has(root)) {
return root;
@@ -84,8 +122,9 @@ function setup(root) {
createProcessEventHandler('uncaughtException', root);
const rejectionHandler =
createProcessEventHandler('unhandledRejection', root);

const coverage = configureCoverage(root);
const exitHandler = () => {
root.coverage = collectCoverage(root, coverage);
root.postRun(new ERR_TEST_FAILURE(
'Promise resolution is still pending but the event loop has already resolved',
kCancelledByParent));
38 changes: 38 additions & 0 deletions lib/internal/test_runner/reporter/tap.js
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@ const {
ArrayPrototypeForEach,
ArrayPrototypeJoin,
ArrayPrototypePush,
NumberPrototypeToFixed,
ObjectEntries,
RegExpPrototypeSymbolReplace,
SafeMap,
@@ -12,6 +13,7 @@ const {
} = primordials;
const { inspectWithNoCustomRetry } = require('internal/errors');
const { isError, kEmptyObject } = require('internal/util');
const { relative } = require('path');
const kDefaultIndent = ' '; // 4 spaces
const kFrameStartRegExp = /^ {4}at /;
const kLineBreakRegExp = /\n|\r\n/;
@@ -46,6 +48,9 @@ async function * tapReporter(source) {
case 'test:diagnostic':
yield `${indent(data.nesting)}# ${tapEscape(data.message)}\n`;
break;
case 'test:coverage':
yield reportCoverage(data.nesting, data.summary);
break;
}
}
}
@@ -68,6 +73,39 @@ function reportTest(nesting, testNumber, status, name, skip, todo) {
return line;
}

function reportCoverage(nesting, summary) {
const pad = indent(nesting);
let report = `${pad}# start of coverage report\n`;

report += `${pad}# file | line % | branch % | funcs % | uncovered lines\n`;

for (let i = 0; i < summary.files.length; ++i) {
const {
path,
coveredLinePercent,
coveredBranchPercent,
coveredFunctionPercent,
uncoveredLineNumbers,
} = summary.files[i];
const relativePath = relative(summary.workingDirectory, path);
const lines = NumberPrototypeToFixed(coveredLinePercent, 2);
const branches = NumberPrototypeToFixed(coveredBranchPercent, 2);
const functions = NumberPrototypeToFixed(coveredFunctionPercent, 2);
const uncovered = ArrayPrototypeJoin(uncoveredLineNumbers, ', ');

report += `${pad}# ${relativePath} | ${lines} | ${branches} | ` +
`${functions} | ${uncovered}\n`;
}

const { totals } = summary;
report += `${pad}# all files | ` +
`${NumberPrototypeToFixed(totals.coveredLinePercent, 2)} | ` +
`${NumberPrototypeToFixed(totals.coveredBranchPercent, 2)} | ` +
`${NumberPrototypeToFixed(totals.coveredFunctionPercent, 2)} |\n`;

report += `${pad}# end of coverage report\n`;
return report;
}

function reportDetails(nesting, data = kEmptyObject) {
const { error, duration_ms } = data;
2 changes: 1 addition & 1 deletion lib/internal/test_runner/runner.js
Original file line number Diff line number Diff line change
@@ -49,7 +49,7 @@ const {
exitCodes: { kGenericUserError },
} = internalBinding('errors');

const kFilterArgs = ['--test', '--watch'];
const kFilterArgs = ['--test', '--test-coverage', '--watch'];
const kFilterArgValues = ['--test-reporter', '--test-reporter-destination'];

// TODO(cjihrig): Replace this with recursive readdir once it lands.
6 changes: 6 additions & 0 deletions lib/internal/test_runner/test.js
Original file line number Diff line number Diff line change
@@ -262,6 +262,7 @@ class Test extends AsyncResource {
this.#outerSignal?.addEventListener('abort', this.#abortHandler);

this.fn = fn;
this.coverage = null; // Configured on the root test by the test harness.
this.mock = null;
this.name = name;
this.parent = parent;
@@ -646,6 +647,11 @@ class Test extends AsyncResource {
this.reporter.diagnostic(this.nesting, kFilename, `skipped ${counters.skipped}`);
this.reporter.diagnostic(this.nesting, kFilename, `todo ${counters.todo}`);
this.reporter.diagnostic(this.nesting, kFilename, `duration_ms ${this.#duration()}`);

if (this.coverage) {
this.reporter.coverage(this.nesting, kFilename, this.coverage);
}

this.reporter.push(null);
}
}
4 changes: 4 additions & 0 deletions lib/internal/test_runner/tests_stream.js
Original file line number Diff line number Diff line change
@@ -55,6 +55,10 @@ class TestsStream extends Readable {
this.#emit('test:diagnostic', { __proto__: null, nesting, file, message });
}

coverage(nesting, file, summary) {
this.#emit('test:coverage', { __proto__: null, nesting, file, summary });
}

#emit(type, data) {
this.emit(type, data);
this.#tryPush({ type, data });
22 changes: 22 additions & 0 deletions lib/internal/util.js
Original file line number Diff line number Diff line change
@@ -714,6 +714,27 @@ function getLazy(initializer) {
};
}

// Setup user-facing NODE_V8_COVERAGE environment variable that writes
// ScriptCoverage objects to a specified directory.
function setupCoverageHooks(dir) {
const cwd = require('internal/process/execution').tryGetCwd();
const { resolve } = require('path');
const coverageDirectory = resolve(cwd, dir);
const { sourceMapCacheToObject } =
require('internal/source_map/source_map_cache');

if (process.features.inspector) {
internalBinding('profiler').setCoverageDirectory(coverageDirectory);
internalBinding('profiler').setSourceMapCacheGetter(sourceMapCacheToObject);
} else {
process.emitWarning('The inspector is disabled, ' +
'coverage could not be collected',
'Warning');
return '';
}
return coverageDirectory;
}

module.exports = {
getLazy,
assertCrypto,
@@ -749,6 +770,7 @@ module.exports = {
SideEffectFreeRegExpPrototypeSymbolSplit,
sleep,
spliceOne,
setupCoverageHooks,
toUSVString,
removeColors,

3 changes: 2 additions & 1 deletion src/inspector_profiler.cc
Original file line number Diff line number Diff line change
@@ -422,7 +422,8 @@ void StartProfilers(Environment* env) {
Local<String> coverage_str = env->env_vars()->Get(
isolate, FIXED_ONE_BYTE_STRING(isolate, "NODE_V8_COVERAGE"))
.FromMaybe(Local<String>());
if (!coverage_str.IsEmpty() && coverage_str->Length() > 0) {
if ((!coverage_str.IsEmpty() && coverage_str->Length() > 0) ||
env->options()->test_runner_coverage) {
CHECK_NULL(env->coverage_connection());
env->set_coverage_connection(std::make_unique<V8CoverageConnection>(env));
env->coverage_connection()->Start();
9 changes: 9 additions & 0 deletions src/node_options.cc
Original file line number Diff line number Diff line change
@@ -140,6 +140,12 @@ void EnvironmentOptions::CheckOptions(std::vector<std::string>* errors,
}

if (test_runner) {
if (test_runner_coverage) {
// TODO(cjihrig): This restriction can be removed once multi-process
// code coverage is supported.
errors->push_back("--test-coverage cannot be used with --test");
}

if (syntax_check_only) {
errors->push_back("either --test or --check can be used, not both");
}
@@ -549,6 +555,9 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
AddOption("--test",
"launch test runner on startup",
&EnvironmentOptions::test_runner);
AddOption("--test-coverage",
"enable code coverage in the test runner",
&EnvironmentOptions::test_runner_coverage);
AddOption("--test-name-pattern",
"run tests whose name matches this regular expression",
&EnvironmentOptions::test_name_pattern);
1 change: 1 addition & 0 deletions src/node_options.h
Original file line number Diff line number Diff line change
@@ -154,6 +154,7 @@ class EnvironmentOptions : public Options {
std::string redirect_warnings;
std::string diagnostic_dir;
bool test_runner = false;
bool test_runner_coverage = false;
std::vector<std::string> test_name_pattern;
std::vector<std::string> test_reporter;
std::vector<std::string> test_reporter_destination;
89 changes: 89 additions & 0 deletions test/fixtures/test-runner/coverage.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
'use strict';
const http = require('node:http'); // node:* should be filtered out.

try {
// This block does not throw.
} catch { /* node:coverage ignore next 3 */
// So this block is uncovered.

/* node:coverage disable */

/* node:coverage enable */

/* node:coverage ignore next */
}

function uncalledTopLevelFunction() {
if (true) {
return 5;
}

return 9;
}

const test = require('node:test');

if (false) {
console.log('this does not execute');
} else {
require('./invalid-tap.js');
}

test('a test', function testHandler() {
const uncalled = () => {};

function fnWithControlFlow(val) {
if (val < 0) {
return -1;
} else if (val === 0) {
return 0;
} else if (val < 100) {
return 1;
} else {
return Infinity;
}
}

fnWithControlFlow(-10);
fnWithControlFlow(99);
});

try {
require('test-nm'); // node_modules should be filtered out.
} catch {

}

async function main() {
if (false) { console.log('boo'); } else { /* console.log('yay'); */ }

if (false) {
console.log('not happening');
}

if (true) {
if (false) {
console.log('not printed');
}

// Nothing to see here.
} else {
console.log('also not printed');
}

try {
const foo = {};

foo.x = 1;
require('../v8-coverage/throw.js');
foo.y = 2;
return foo;
} catch (err) {
let bar = [];
bar.push(5);
} finally {
const baz = 1;
}
}

main();
98 changes: 98 additions & 0 deletions test/parallel/test-runner-coverage.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
'use strict';
const common = require('../common');
const assert = require('node:assert');
const { spawnSync } = require('node:child_process');
const { readdirSync } = require('node:fs');
const { test } = require('node:test');
const fixtures = require('../common/fixtures');
const tmpdir = require('../common/tmpdir');

tmpdir.refresh();

function findCoverageFileForPid(pid) {
const pattern = `^coverage\\-${pid}\\-(\\d{13})\\-(\\d+)\\.json$`;
const regex = new RegExp(pattern);

return readdirSync(tmpdir.path).find((file) => {
return regex.test(file);
});
}

function getCoverageFixtureReport() {
const report = [
'# start of coverage report',
'# file | line % | branch % | funcs % | uncovered lines',
'# test/fixtures/test-runner/coverage.js | 78.65 | 38.46 | 60.00 | 12, ' +
'13, 16, 17, 18, 19, 20, 21, 22, 27, 39, 43, 44, 61, 62, 66, 67, 71, 72',
'# test/fixtures/test-runner/invalid-tap.js | 100.00 | 100.00 | 100.00 | ',
'# test/fixtures/v8-coverage/throw.js | 71.43 | 50.00 | 100.00 | 5, 6',
'# all files | 78.35 | 43.75 | 60.00 |',
'# end of coverage report',
].join('\n');

if (common.isWindows) {
return report.replaceAll('/', '\\');
}

return report;
}

test('--test-coverage and --test cannot be combined', () => {
// TODO(cjihrig): This test can be removed once multi-process code coverage
// is supported.
const result = spawnSync(process.execPath, ['--test', '--test-coverage']);

// 9 is the documented exit code for an invalid CLI argument.
assert.strictEqual(result.status, 9);
assert.match(
result.stderr.toString(), /--test-coverage cannot be used with --test/
);
});

test('handles the inspector not being available', (t) => {
if (process.features.inspector) {
return;
}

const fixture = fixtures.path('test-runner', 'coverage.js');
const args = ['--test-coverage', fixture];
const result = spawnSync(process.execPath, args);

assert(!result.stdout.toString().includes('# start of coverage report'));
assert(result.stderr.toString().includes('coverage could not be collected'));
assert.strictEqual(result.status, 0);
assert(!findCoverageFileForPid(result.pid));
});

test('coverage is reported and dumped to NODE_V8_COVERAGE if present', (t) => {
if (!process.features.inspector) {
return;
}

const fixture = fixtures.path('test-runner', 'coverage.js');
const args = ['--test-coverage', fixture];
const options = { env: { ...process.env, NODE_V8_COVERAGE: tmpdir.path } };
const result = spawnSync(process.execPath, args, options);
const report = getCoverageFixtureReport();

assert(result.stdout.toString().includes(report));
assert.strictEqual(result.stderr.toString(), '');
assert.strictEqual(result.status, 0);
assert(findCoverageFileForPid(result.pid));
});

test('coverage is reported without NODE_V8_COVERAGE present', (t) => {
if (!process.features.inspector) {
return;
}

const fixture = fixtures.path('test-runner', 'coverage.js');
const args = ['--test-coverage', fixture];
const result = spawnSync(process.execPath, args);
const report = getCoverageFixtureReport();

assert(result.stdout.toString().includes(report));
assert.strictEqual(result.stderr.toString(), '');
assert.strictEqual(result.status, 0);
assert(!findCoverageFileForPid(result.pid));
});

0 comments on commit 6119289

Please sign in to comment.