Surprising parametrized fixtures order

[smarie]

Hi,

The example below seems to always lead to the same tests execution order, in both pytest 2, 3, and 4. However the order is counter intuitive:

import pytest

@pytest.fixture(params=[(1, 2), (3, 4)], ids=str)
def arg1_arg2_root(request):
    return request.param

@pytest.fixture
def arg1(arg1_arg2_root):
    return arg1_arg2_root[0]

@pytest.fixture
def arg2(arg1_arg2_root):
    return arg1_arg2_root[1]

@pytest.fixture(params=[5, 6])
def arg3(request):
    return request.param

def test_order(arg1, arg2, arg3):
    pass

leads to

<..>::test_order[5-(1, 2)] 
<..>::test_order[5-(3, 4)] 
<..>::test_order[6-(1, 2)] 
<..>::test_order[6-(3, 4)] 

However it seems counter-intuitive to me. Indeed all the fixtures are function-scopes fixtures, and the function uses arg1 (so, arg1_arg2_root) first, then arg2 (same root), then arg3. I would therefore expect:

<..>::test_order[(1, 2)-5] 
<..>::test_order[(1, 2)-6] 
<..>::test_order[(3, 4)-5] 
<..>::test_order[(3, 4)-6] 

Do you agree ? If so that's maybe a bug happening when the fixture closure is computed?

[nicoddemus]

Hi @smarie, thanks for the report.

Fixture order due to parametrization is a "hot topic" in the sense that there's been some work been done on it, but is a non-trivial issue because fixture setup order might greatly affect test suite performance. To read more about it please see #3551.

I'm closing this for now, feel free to join the existing discussion on the topic. 👍

[smarie]

Ok. IMO all kind of optimization are highly welcome, as long as users can decide whether to use the optimized order or not. I suspect that a fair share of users might favor more simple but more readable execution order, especially when their tests do not require resources that are expensive to setup/teardown.

An extra API parameter would certainly be necessary to make this choice explicit.

See also #3393 and #2846.