pytest_generate_tests conflicts with conftest.py fixtures

[vlcinsky]

Dedicated repository for exhibiting the problem and with detailed description see https://github.com/vlcinsky/pytest_generate_tests-conflicts

From "summary" in repository README.md

Duplicate values

As shown in 003_conftest_fixt1, when pytest_generate_tests provided a value for test case
parameter and at the same time conftest.py provides a fixture with the same name, it wrongly
attempts to define a test case call also with the value from conftest.py fixture.

Expected behaviour is, that once a test function gets value for a parameter from
pytest_generate_tests, then further collection of values for this parameter shall stop.
Alternatively it shall resolve the duplication by giving the pytest_generate_tests parameter value
preference.

Collection collects tests multiple times

As shown in 005_conftest_fixt2, when pytest_generate_tests provides a value for a test case
parameter and at the same time in conftest.py there exist a fixture, which is dependent on another
one, it results in collecting the same test call (having the same combination of parameter values)
multiple times, this time it goes around Duplicate values by generating new test id using
parameter value from conftest.py fixture.

Expected behaviour is the same as described in "Duplicate values" above.

[nicoddemus]

@vlcinsky thanks a lot for all the effort you put into creating a repository and documenting each case, we really appreciate it.

[vlcinsky]

@nicoddemus This is my contribution to my beloved testing framework. Good quality issue description is rather easy comparing to reading pytest internals what I gave up being happy, others take care of it.

[fconil]

Hi,
It seem I may have the same problem.

• os : ubuntu 22.04
• using a virtual environment
• Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
• pytest 7.4.0

I have a fake fixture function in conftest.py to explore testing possibilities

@pytest.fixture(
    params=[
        "samples/video1.mp4",
        "samples/video2.mp4",
        "samples/video3.mp4",
    ]
)
def video_input(request):
    return request.param

def pytest_addoption(parser):
    parser.addoption(
        "--video_path",
        action="append",
        default=[],
        help="list of video path to pass to test functions",
    )

And pytest_generate_tests is declared in test_hook.py in the same folder :

def get_video_duration(video):
    return 20

def pytest_generate_tests(metafunc):
    if "video_input" in metafunc.fixturenames:
        metafunc.parametrize("video_input", metafunc.config.getoption("video_path"))

def test_get_dynamic_video_duration(video_input):
    video = video_input
    assert get_video_duration(video) == 20

Here is the ValueError / duplicate I got when I launch the test :

$ pytest -v --no-header --video_path="samples/video4.mp4" tests/ex6/test_hook.py
=========================================================================================== test session starts ===========================================================================================
collected 0 items / 1 error                                                                                                                                                                               

================================================================================================= ERRORS ==================================================================================================
____________________________________________________________________________________ ERROR collecting ex6/test_hook.py ____________________________________________________________________________________
.venv/lib/python3.10/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
.venv/lib/python3.10/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.venv/lib/python3.10/site-packages/_pytest/python.py:534: in collect
    return super().collect()
.venv/lib/python3.10/site-packages/_pytest/python.py:455: in collect
    res = ihook.pytest_pycollect_makeitem(
.venv/lib/python3.10/site-packages/pluggy/_hooks.py:433: in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
.venv/lib/python3.10/site-packages/pluggy/_manager.py:112: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
.venv/lib/python3.10/site-packages/_pytest/python.py:271: in pytest_pycollect_makeitem
    return list(collector._genfunctions(name, obj))
.venv/lib/python3.10/site-packages/_pytest/python.py:498: in _genfunctions
    self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
.venv/lib/python3.10/site-packages/pluggy/_hooks.py:489: in call_extra
    return self._hookexec(self.name, hookimpls, kwargs, firstresult)
.venv/lib/python3.10/site-packages/pluggy/_manager.py:112: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
.venv/lib/python3.10/site-packages/_pytest/fixtures.py:1570: in pytest_generate_tests
    metafunc.parametrize(
.venv/lib/python3.10/site-packages/_pytest/python.py:1347: in parametrize
    newcallspec = callspec.setmulti(
.venv/lib/python3.10/site-packages/_pytest/python.py:1152: in setmulti
    raise ValueError(f"duplicate {arg!r}")
E   ValueError: duplicate 'video_input'
========================================================================================= short test summary info =========================================================================================
ERROR tests/ex6/test_hook.py - ValueError: duplicate 'video_input'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================ 1 error in 0.41s =============================================================================================

[RonnyPfannschmidt]

i believe this can be summarized as parametrize working unconditionally, so params and generate_tests are "mutually exclusive"

both are implemented in terms of invoking metafunc.parametrize

fixing would to add support for either way would require overriding parametriting only if a parameter is not overridden

[RonnyPfannschmidt]

a temporary workaround would be to remove the params on the fixture, and passing that value as default in generate_tests if no input was given

[fconil]

Thanks for you prompt reply.

In the Python testing with pytest, 2nd ed book, the example uses pytest_generate_tests on a fixture that uses params.

You can look at chapter 5 source code, test_fix_param.py and test_gen.py.

Surprisingly the fixture code is not in conftest.py but in another test file.

If I put my fixture in another test file in the same folder than test_hook.py, it works.

[RonnyPfannschmidt]

indeed, it works if the individual hooks/configurations only apply to specific test files
in the book example each file has a self contained variant

if one moved all the implementations over to a conftest,
one would start to see the conflicts as they would now apply to all files instead of individual files

[fconil]

I realized with this problem that I did not fully understand fixture loading / usage.

In the book example, pytest_generate_tests parametrize a fixture - start_state - that is not "defined" earlier. start_state does not come from test_fix_param.py as I wrongly believed (thanks --fixtures-per-test).

This is the same for stringinput in the online documentation, https://docs.pytest.org/en/stable/how-to/parametrize.html#pytest-generate-tests

I note that I should not use @pytest.fixture(params=...) and pytest_generate_tests at the same time.

Thanks for your kind help !