test_runner: add level-based diagnostic handling for reporter

[hpatel292-seneca]

This fixes #55922

Change summary

Updated the reporter.diagnostic to accept level parameter like this

  diagnostic(nesting, loc, message, level = 'info') {
    this[kEmitMessage]('test:diagnostic', {
      __proto__: null,
      nesting,
      message,
      level,
      ...loc,
    });
  }

Then I updated #handleEvent like this

 #handleEvent({ type, data }) {
    switch (type) {
      case 'test:fail':
        if (data.details?.error?.failureType !== kSubtestsFailed) {
          ArrayPrototypePush(this.#failedTests, data);
        }
        return this.#handleTestReportEvent(type, data);
      case 'test:pass':
        return this.#handleTestReportEvent(type, data);
      case 'test:start':
        ArrayPrototypeUnshift(this.#stack, { __proto__: null, data, type });
        break;
      case 'test:stderr':
      case 'test:stdout':
        return data.message;
      case 'test:diagnostic':  // Here I added logic
        const diagnosticColor =
          reporterColorMap[data.level] || reporterColorMap['test:diagnostic'];
        return `${diagnosticColor}${indent(data.nesting)}${
          reporterUnicodeSymbolMap[type]
        }${data.message}${colors.white}\n`;
      case 'test:coverage':
        return getCoverageReport(
          indent(data.nesting),
          data.summary,
          reporterUnicodeSymbolMap['test:coverage'],
          colors.blue,
          true
        );
    }
  }

And I am Updated reporterColorMap like this

const reporterColorMap = {
  __proto__: null,
  get 'test:fail'() {
    return colors.red;
  },
  get 'test:pass'() {
    return colors.green;
  },
  get 'test:diagnostic'() {
    return colors.blue;
  },
  get info() { // Here I added logic
    return colors.blue;
  },
  get debug() { 
    return colors.gray;
  },
  get warn() { 
    return colors.yellow;
  },
  get error() { 
    return colors.red;
  },
};

and color already contain logic for this colors

I also set the reporter.diagnostic call from test.js like this (level="Error")

if (actual < threshold) {
            harness.success = false;
            process.exitCode = kGenericUserError;
            reporter.diagnostic(
              nesting,
              loc,
              `Error: ${NumberPrototypeToFixed(
                actual,
                2
              )}% ${name} coverage does not meet threshold of ${threshold}%.`,
              'error'  // Level is set to error for red color
            );
          }

Here is Demo output:

[nodejs-github-bot]

Review requested:

• @nodejs/test_runner

[hpatel292-seneca]

Hi @pmarchini, Please review and let me know if you want me to change anything.

[pmarchini]

Hey @hpatel292-seneca, I won't be able to review until Monday.
In the meantime, I suggest you take a look at the test runner output tests; we definitely need to add tests there.

I've also requested other reviews in the meantime.

[codecov]

Codecov Report

Attention: Patch coverage is 81.25000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 87.96%. Comparing base (e92499c) to head (4048621).
Report is 33 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/test_runner/reporter/utils.js	77.77%	2 Missing ⚠️
lib/internal/test_runner/test.js	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #55964      +/-   ##
==========================================
- Coverage   87.99%   87.96%   -0.04%     
==========================================
  Files         653      656       +3     
  Lines      188091   188384     +293     
  Branches    35941    35982      +41     
==========================================
+ Hits       165516   165708     +192     
- Misses      15751    15845      +94     
- Partials     6824     6831       +7     

Files with missing lines	Coverage Δ
lib/internal/test_runner/reporter/spec.js	96.26% <100.00%> (+0.07%)	⬆️
lib/internal/test_runner/tests_stream.js	91.66% <100.00%> (+0.04%)	⬆️
lib/internal/test_runner/test.js	96.95% <0.00%> (-0.02%)	⬇️
lib/internal/test_runner/reporter/utils.js	95.00% <77.77%> (-1.71%)	⬇️

... and 45 files with indirect coverage changes

[cjihrig]

Requesting changes since this already has an approval. This also needs docs and tests.

[cjihrig]

Let's remove the debug level. The reporter stream is not a generic logger, and we have other ways (NODE_DEBUG) of adding debug output.

[hpatel292-seneca]

I will remove that. And I have a question so this CI https://github.com/nodejs/node/actions/runs/11983534043/job/33427085217?pr=55964 failed and it's for First commit message adheres to guidelines / lint-commit-message (pull_request) so should I change commit history??

[cjihrig]

test_runner: add level to diagnostics

[hpatel292-seneca]

So I am asking if I should change the commit history and force push.

[cjihrig]

Ah. Yes, please. You'll need to avoid merge commits too, as the tooling does not handle them well.

[pmarchini]

I suggest adding a test to test-runner-output.mjs.
This file contains a list of snapshot tests, some of which use colors.

An alternative (and probably more cohesive) approach could be to add a test that checks this specific behavior by setting the following environment variable:

node/lib/internal/util/colors.js

Line 19 in 4f62ab5

if (process.env.FORCE_COLOR !== undefined) {

to force the color.

One note: Since you changed the spec reporter, your test will need to use that specific reporter.

[hpatel292-seneca]

@pmarchini Ok so I am taking reference from this tests
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.mjs
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.snapshot

and based on that I wrote this test.
fixtures/test-runner/output/spec_reporter_diagnostic_levels.mjs

// Flags: --test-reporter=spec

import { test } from 'node:test';
import { TestsStream } from '../../../lib/internal/test_runner/tests_stream.js';

process.env.FORCE_COLOR = '3';

const testsStream = new TestsStream();

test('Diagnostic Levels Color Output', () => {
  testsStream.diagnostic(1, {}, 'Info-level message', 'info');
  testsStream.diagnostic(1, {}, 'Warning-level message', 'warn');
  testsStream.diagnostic(1, {}, 'Error-level message', 'error');
});

and add this in test-runner-output.mjs

{
  name: 'test-runner/output/spec_reporter_diagnostic_levels.mjs',
  transform: specTransform,
  tty: true,
}

now I am running tools/test.py test/parallel/test-runner-output.mjs --snapshot but it's not creating snap for added test.
How I can create snaps??

[hpatel292-seneca]

I got this when I tried using test with release/node

[pmarchini]

I forgot you were working in a Windows environment. Considering this, I would suggest picking a different approach. You could add a test to node/test/parallel/test-runner-coverage-thresholds.js that forces the colors via the environment variable in the spawn, and check that the error message 'coverage does not meet...' is displayed in red.
Note: you need to set the reporter to spec.

This is also because test-runner-output.mjs is intended for 'e2e' testing of the test runner's output under specific circumstances.

[hpatel292-seneca]

@pmarchini @cjihrig Thank you so much for your help. It wouldn't be possible without your help.
I ran test cases and here is output
I printed raw output in test case and here is output

and here is test case:

test(`test failing ${coverage.flag} with red color`, () => {
    const result = spawnSync(process.execPath, [
      '--test',
      '--experimental-test-coverage',
      `${coverage.flag}=99`,
      '--test-reporter', 'spec',
      fixture,
    ], {
      env: { ...process.env, FORCE_COLOR: '3' },
    });

    const stdout = result.stdout.toString();
    const redColorRegex = /\u001b\[31mℹ Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/;
    assert.match(stdout, redColorRegex, 'Expected red color code not found in diagnostic message');
    assert.strictEqual(result.status, 1);
    assert(!findCoverageFileForPid(result.pid));
  });

If it looks good to you I can commit it and you can review the whole PR.

[hpatel292-seneca]

Hi @pmarchini @cjihrig,
The PR is ready for review.

[hpatel292-seneca]

Hi @pmarchini @cjihrig, The PR is ready for review.

Hi, @pmarchini @cjihrig I will add one more commit for fixing the lint error.
Fix:
I must add // eslint-disable-next-line no-control-regex for that line as I need that character for \u001b[31m for checking the response if red color.
It is also used at other places like

node/test/fixtures/test-runner/output/spec_reporter_cli.js

Line 9 in 585f7bc

// eslint-disable-next-line no-control-regex

Should I go ahead and send commit?

[pmarchini]

Hi @pmarchini @cjihrig, The PR is ready for review.

Hi, @pmarchini @cjihrig I will add one more commit for fixing the lint error.
Fix:
I must add // eslint-disable-next-line no-control-regex for that line as I need that character for \u001b[31m for checking the response if red color.
It is also used at other places like

node/test/fixtures/test-runner/output/spec_reporter_cli.js

Line 9 in 585f7bc

// eslint-disable-next-line no-control-regex

Should I go ahead and send commit?

Sure 🚀

[hpatel292-seneca]

Hi @pmarchini @cjihrig, The PR is ready for review.

Hi, @pmarchini @cjihrig I will add one more commit for fixing the lint error.
Fix:
I must add // eslint-disable-next-line no-control-regex for that line as I need that character for \u001b[31m for checking the response if red color.
It is also used at other places like

node/test/fixtures/test-runner/output/spec_reporter_cli.js

Line 9 in 585f7bc

// eslint-disable-next-line no-control-regex

Should I go ahead and send commit?

Sure 🚀

@pmarchini I did. Thanks