Tests from test class with same non-fully qualified class name as another test class are sometimes not discovered

[Kallys]

Q	A
PHPUnit version	10.0.16
PHP version	8.2.3
Installation Method	Composer

Summary

Not sure to understand what happen, but since I updated one of my project to PHPUnit 10, I got some conflicts with test classes having same names but in different namespaces (I'm using some obscur static mecanisms).

Let say class \MyTests\A\MyClassTest and \MyTests\B\MyClassTest having common \MyTests\C\MyTestCasse parent for example.

I tried to investigate a little bit, and found this line is overriding $suiteClassName using first loaded class having same name, whatever namespace is. Is it expected ?

Thanks!

[sebastianbergmann]

Thank you for your report.

Please provide a minimal, self-contained, reproducing test case that shows the problem you are reporting.

Without such a minimal, self-contained, reproducing test case I will not be able to investigate this issue.

[Kallys]

Took me a while to isolate the problem, but here it is: phpunit10_bug.zip

composer install
vendor/bin/phpunit tests/

Tests are passing while \AppTest\B\MyClassTest should not.

[sebastianbergmann]

The event stream confirms that AppTest\C\MyClassTest::test is run twice and that MyTests\B\MyClassTest is not run:

./vendor/bin/phpunit --no-output --log-events-text php://stdout tests
PHPUnit Started (PHPUnit 10.0.17 using PHP 8.2.4 (cli) on Linux)
Test Runner Configured
Test Suite Loaded (3 tests)
Event Facade Sealed
Test Runner Started
Test Suite Sorted
Test Runner Execution Started (3 tests)
Test Suite Started (/home/sb/phpunit10_bug/tests, 3 tests)
Test Suite Started (AppTest\A\AnotherClassTest, 1 test)
Test Suite Started (AppTest\A\AnotherClassTest::test, 1 test)
Test Preparation Started (AppTest\A\AnotherClassTest::test#0)
Test Prepared (AppTest\A\AnotherClassTest::test#0)
Assertion Succeeded (Constraint: is an instance of class AppTest\C\MyClassTest, Value: AppTest\C\MyClassTest Object #305 (
    'backupGlobals' => null
    'backupGlobalsExcludeList' => Array &0 ()
    'backupStaticProperties' => null
    'backupStaticPropertiesExcludeList' => Array &1 ()
    'snapshot' => null
    'runClassInSeparateProcess' => null
    'runTestInSeparateProcess' => null
    'preserveGlobalState' => false
    'inIsolation' => false
    'expectedException' => null
    'expectedExceptionMessage' => null
    'expectedExceptionMessageRegExp' => null
    'expectedExceptionCode' => null
    'providedTests' => Array &2 (
        0 => PHPUnit\Framework\ExecutionOrderDependency Object #304 (
            'className' => 'AppTest\C\MyClassTest'
            'methodName' => 'a'
            'shallowClone' => false
            'deepClone' => false
        )
    )
    'data' => Array &3 ()
    'dataName' => ''
    'name' => 'a'
    'groups' => Array &4 ()
    'dependencies' => Array &5 ()
    'dependencyInput' => Array &6 ()
    'iniSettings' => Array &7 ()
    'locale' => Array &8 ()
    'mockObjectGenerator' => null
    'mockObjects' => Array &9 ()
    'registerMockObjectsFromTestArgumentsRecursively' => false
    'status' => PHPUnit\Framework\TestStatus\Unknown Object #303 (
        'message' => ''
    )
    'numberOfAssertionsPerformed' => 0
    'testResult' => null
    'output' => ''
    'outputExpectedRegex' => null
    'outputExpectedString' => null
    'outputBufferingActive' => false
    'outputRetrievedForAssertion' => false
    'doesNotPerformAssertions' => false
    'customComparators' => Array &10 ()
    'testValueObjectForEvents' => null
    'wasPrepared' => false
))
Test Passed (AppTest\A\AnotherClassTest::test#0)
Test Finished (AppTest\A\AnotherClassTest::test#0)
Test Suite Finished (AppTest\A\AnotherClassTest::test, 1 test)
Test Suite Finished (AppTest\A\AnotherClassTest, 1 test)
Test Suite Started (AppTest\C\MyClassTest, 1 test)
Test Preparation Started (AppTest\C\MyClassTest::test)
Test Prepared (AppTest\C\MyClassTest::test)
Assertion Succeeded (Constraint: is true, Value: true)
Test Passed (AppTest\C\MyClassTest::test)
Test Finished (AppTest\C\MyClassTest::test)
Test Suite Finished (AppTest\C\MyClassTest, 1 test)
Test Suite Started (AppTest\C\MyClassTest, 1 test)
Test Preparation Started (AppTest\C\MyClassTest::test)
Test Prepared (AppTest\C\MyClassTest::test)
Assertion Succeeded (Constraint: is true, Value: true)
Test Passed (AppTest\C\MyClassTest::test)
Test Finished (AppTest\C\MyClassTest::test)
Test Suite Finished (AppTest\C\MyClassTest, 1 test)
Test Suite Finished (/home/sb/phpunit10_bug/tests, 3 tests)
Test Runner Execution Finished
Test Runner Finished
PHPUnit Finished (Shell Exit Code: 0)

The problem does not exist in PHPUnit 9.6:

./vendor/bin/phpunit --debug tests
PHPUnit 9.6.5 by Sebastian Bergmann and contributors.

Test 'AppTest\A\AnotherClassTest::test with data set #0 (AppTest\C\MyClassTest Object (...))' started
Test 'AppTest\A\AnotherClassTest::test with data set #0 (AppTest\C\MyClassTest Object (...))' ended
Test 'AppTest\B\MyClassTest::test' started
Test 'AppTest\B\MyClassTest::test' ended
Test 'AppTest\C\MyClassTest::test' started
Test 'AppTest\C\MyClassTest::test' ended


Time: 00:00.015, Memory: 4.00 MB

There was 1 failure:

1) AppTest\B\MyClassTest::test
Failed asserting that false is true.

/home/sb/phpunit10_bug/tests/B/MyClassTest.php:11

FAILURES!
Tests: 3, Assertions: 3, Failures: 1.

[sebastianbergmann]

This is related to the data provider in AppTest\A\AnotherClassTest that uses AppTest\C\MyClassTest. If I remove that ...

 class AnotherClassTest extends TestCase
 {
-    public static function provide(): array
+    public function test(): void
     {
-        return [[new MyClassTest('a')]];
-    }
-
-    /** @dataProvider provide */
-    public function test(mixed $className): void
-    {
-        $this->assertInstanceOf(MyClassTest::class, $className);
+        $this->assertTrue(true);
     }
 }

... then all three test methods are found and run as intended.

[sebastianbergmann]

Before I invest time to further investigate this issue, however, I would like to know whether there is a real use case that triggers this issue. Creating a TestCase object in a data provider method is not a valid use case.

This issue is likely just one more reason to use static analysis instead of dynamic reflection for test discovery. Such a change is a long(er) term project, though.

[sebastianbergmann]

@realFlowControl In the meantime, can I bother you to have a look? Maybe you have an idea what should be done here. Thanks!

[sebastianbergmann]

I added a test case for this in b7569ec.

[Kallys]

My real use case is using a public static method (instead of dummy class creation):

class AnotherClassTest extends TestCase
{
    public static function provide(): array
    {
-        return [[new MyClassTest('a')]];
+        return [[MyClassTest::getVal()]];
    }

    /** @dataProvider provide */
    public function test(mixed $className): void
    {
        $this->assertEquals('', $className);
    }
}

[Kallys]

That was quick, thanks!