pytest 8.0.0 ignores autouse fixtures in doctest modules when collecting packages

[mgedmin]

pyspacewar has a test suite written back when I still thought doctest modules were neat. It uses --doctest-modules and relies on some module-global setup/teardown specified in a @pytest.fixture(autouse=True).

This broke with the pytest 8.0.0 release. If I run

pytest src/pyspacewar/tests/ --setup-show

I can see that the autouse fixtures defined in my conftest.py are being called, but the autouse fixture in src/pyspacewar/tests/test_main.py (fake_game_ui) is not. This breaks the tests or makes them hang:

src/pyspacewar/tests/test_main.py 
        src/pyspacewar/tests/test_main.py::pyspacewar.tests.test_main.doctest_main (fixtures used: setUp)

Curiously, if I run just a single file or all the test files in the directory, this doesn't happen:

pytest src/pyspacewar/tests/*.py --setup-show

shows

...
src/pyspacewar/tests/test_main.py 
    SETUP    M fake_game_ui
        src/pyspacewar/tests/test_main.py::pyspacewar.tests.test_main.doctest_main (fixtures used: fake_game_ui, setUp).
    TEARDOWN M fake_game_ui
...

which is also what pytest 7.4.4 used to do.

I've verified by pip install "pytest<8" and pip install -U pytest that pytest being 7.4.4 vs 8.0.0 is the only difference between a working and a not-working tox environment. A full pip list is here:

Package    Version
---------- ----------
iniconfig  2.0.0
mock       5.1.0
packaging  23.2
pip        23.2.1
pluggy     1.4.0
pygame     2.5.2
pyspacewar 1.2.0.dev0
pytest     8.0.0

OS: Ubuntu 23.10.

I have not yet tried to produce a separate minimal example, but you should be able to quickly reproduce this if you

git clone https://github.com/mgedmin/pyspacewar
tox -e py312 --notest
.tox/py312/bin/pip install -U pytest    # because I pinned to <8 as a workaround for now
.tox/py312/bin/pytest src/pyspacewar/tests/ --setup-show

[bluetech]

Thanks for the report. Bisected to ab63ebb. Will check out what exactly is causing the issue.

[bluetech]

Figured out what's going on. We have the following collection tree:

<Dir pyspacewar>
  <Dir src>
    <Package pyspacewar>
      <Package tests>
        <DoctestModule test_main.py>
          <DoctestItem pyspacewar.tests.test_main.doctest_main>

And the test_main.py contains an autouse fixture (fake_game_ui) that doctest_main needs in order to run properly. The fixture doesn't run! It doesn't run because nothing collects the fixtures from (calls parsefactories() on) the test_main.py DoctestModule.

How come it only started happening with commit ab63ebb? Turns out it mostly only worked accidentally. Each DoctestModule is also collected as a normal Module, with the Module collected after the DoctestModule. For example, if we add a non-doctest test to test_main.py, the collection tree looks like this:

<Dir pyspacewar>
  <Dir src>
    <Package pyspacewar>
      <Package tests>
        <DoctestModule test_main.py>
          <DoctestItem pyspacewar.tests.test_main.doctest_main>
        <Module test_main.py>
          <Function test_it>

Now, Module does collect fixtures. When autouse fixtures are collected, they are added to the _nodeid_autousenames dict.

Before ab63ebb, DoctestItem consults _nodeid_autousenames at setup time. At this point, the Module has collected and so it ended up picking the autouse fixture (this relies on another "accident", that the DoctestModule and Module have the same node ID).

After ab63ebb, DoctestItem consults _nodeid_autousenames at collection time (= when it's created). At this point, the Module hasn't collected yet, so the autouse fixture is not picked out.

The fix is simple -- have DoctestModule.collect() call parsefactories. From some testing I've done it shouldn't have negative consequences (I hope).

[nicoddemus]

Great explanation! I think it would be nice to include it in the commit message of the fix itself. 👍