Test Runner does not display test runner warnings when TestDox result printer is used

[Rezyan]

Q	A
PHPUnit version	10.0.4
PHP version	8.1.2
Installation Method	Composer

Summary

I've just migrated to v10.0.4 and since then, an invisible warning appears when it was not there before. I tried to set the new setting displayDetailsOnTestsThatTriggerWarnings="true" in the XML file, but it doesn't change anything. It worked fine under v9.6.3.

Similar issue: #5136

Current behavior

A ghost warning appears since v10.0.4. See the cmd part:

How to reproduce

composer.json:

{
    "require": {
        "phpunit/phpunit": "^10.0"
    }
}

phpunit.xml.dist:

<?xml version="1.0" encoding="utf-8"?>
<phpunit
	cacheResult="true"
	cacheResultFile="var/phpunit/result.json"
	testdox="true"
	colors="true"
	displayDetailsOnIncompleteTests="true"
	displayDetailsOnSkippedTests="true"
	displayDetailsOnTestsThatTriggerDeprecations="true"
	displayDetailsOnTestsThatTriggerErrors="true"
	displayDetailsOnTestsThatTriggerNotices="true"
	displayDetailsOnTestsThatTriggerWarnings="true">
	<testsuites>
		<testsuite name="unit">
			<directory>src/Test</directory>
		</testsuite>
	</testsuites>

	<logging>
		<junit outputFile="var/phpunit/junit.xml"/>
		<teamcity outputFile="var/phpunit/teamcity.txt"/>
		<testdoxXml outputFile="var/phpunit/testdox.xml"/>
		<testdoxHtml outputFile="var/phpunit/testdox.html"/>
		<testdoxText outputFile="var/phpunit/testdox.txt"/>
		<text outputFile="var/phpunit/log.txt"/>
	</logging>
</phpunit>

src/Test/FooTest.php:

<?php

namespace Test;

use PHPUnit\Framework\TestCase;

final class FooTest extends TestCase
{
	/**
	 * @testWith
	 * [1, 1]
	 * [2, 2]
	 */
	public function test(int $expected, int $actual): void
	{
		$this->assertSame($expected, $actual);
	}
}

Expected behavior

No warning appears in v9.6.3. See the cmd part:

[sebastianbergmann]

Your XML configuration file does not conform to PHPUnit 10's schema: the logging/testdoxXml and logging/text elements no longer exist.

Here is the event log for the reproducing example:

PHPUnit Started (PHPUnit 10.0.4 using PHP 8.2.2 (cli) on Linux)
Test Runner Triggered Warning (Test results may not be as expected because the XML configuration file did not pass validation:

  Line 21:
  - Element 'testdoxXml': This element is not expected.
)
Test Runner Configured
Test Suite Loaded (2 tests)
Event Facade Sealed
Test Runner Started
Test Suite Sorted
Test Runner Execution Started (2 tests)
Test Suite Started (2 tests)
Test Suite Started (unit, 2 tests)
Test Suite Started (Test\FooTest, 2 tests)
Test Suite Started (Test\FooTest::test, 2 tests)
Test Preparation Started (Test\FooTest::test#0)
Test Prepared (Test\FooTest::test#0)
Assertion Succeeded (Constraint: is identical to 1, Value: 1)
Test Passed (Test\FooTest::test#0)
Test Finished (Test\FooTest::test#0)
Test Preparation Started (Test\FooTest::test#1)
Test Prepared (Test\FooTest::test#1)
Assertion Succeeded (Constraint: is identical to 2, Value: 2)
Test Passed (Test\FooTest::test#1)
Test Finished (Test\FooTest::test#1)
Test Suite Finished (Test\FooTest::test, 2 tests)
Test Suite Finished (Test\FooTest, 2 tests)
Test Suite Finished (unit, 2 tests)
Test Suite Finished (2 tests)
Test Runner Execution Finished
Test Runner Finished
PHPUnit Finished (Shell Exit Code: 1)

The bug here is that the test runner does not report warnings, for instance those triggered by the XML configuration file loader, when TestDox output is used (testdox="true").

[Rezyan]

Your XML configuration file does not conform to PHPUnit 10's schema: the logging/testdoxXml and logging/text elements no longer exist.

Indeed, sorry, I hadn't paid enough attention to these specific changes during the migration!

The bug here is that the test runner does not report warnings, for instance those triggered by the XML configuration file loader, when TestDox output is used (testdox="true").

My problem is solved, but we leave the issue open because the bug I found is actually a bug with the warning reports, right?

[sebastianbergmann]

My problem is solved, but we leave the issue open because the bug I found is actually a bug with the warning reports, right?

Right. I have fixed the issue locally, but before I commit/push the fix I will need to implement a regression test. I'll do that first thing tomorrow.

[Rezyan]

Right. I have fixed the issue locally, but before I commit/push the fix I will need to implement a regression test. I'll do that first thing tomorrow.

Take the time you need, thanks for your fast help!

[Rezyan]

@sebastianbergmann By the way, i think the v10 documentation should be updated: https://phpunit.readthedocs.io/en/10.0/configuration.html#the-logging-element. Because the two removed elements are still used in the example. I warn, in case it is not planned.

[sebastianbergmann]

I did not have the time yet to update the entire documentation. This is work for the next weekends ...