Unexpected order of tests using parameterized fixtures #2846

rbierbasz · 2017-10-17T16:00:06Z

Consider following code:

import pytest

@pytest.fixture(scope="function", params=(1, 2, 3))
def f1(request):
    pass
    
@pytest.fixture(scope="function", params=('a', 'b', 'c'))
def f2(request):
    pass

def test(f1, f2):
    pass

The order of the tests meets my expectations:

============================= test session starts =============================
platform win32 -- Python 2.7.12, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- c:\users\rbierbasz\.virtualenvs\core\scripts\python.exe
cachedir: .cache
rootdir: C:\Users\rbierbasz\repos\galaxy-core, inifile:
collected 9 items

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[3-a] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-c] PASSED

========================== 9 passed in 0.04 seconds ===========================

But when I changed fixtures' scope from function to module:

import pytest

@pytest.fixture(scope="module", params=(1, 2, 3))
def f1(request):
    pass
    
@pytest.fixture(scope="module", params=('a', 'b', 'c'))
def f2(request):
    pass

def test(f1, f2):
    pass

it seems to get random:

============================= test session starts =============================
platform win32 -- Python 2.7.12, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- c:\users\rbierbasz\.virtualenvs\core\scripts\python.exe
cachedir: .cache
rootdir: C:\Users\rbierbasz\repos\galaxy-core, inifile:
collected 9 items

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[3-c] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-a] PASSED

========================== 9 passed in 0.04 seconds ===========================

The same happens for class and session scopes. It leads to fixtures being set up and torn down more times than necessary.

The text was updated successfully, but these errors were encountered:

nicoddemus · 2017-10-18T21:13:57Z

Definitely it seems a bug, thanks for the report.

Our parametrization code has some standing bugs, it would be nice for us to focus on that part of the code when we get the chance.

bilderbuchi · 2017-10-24T20:08:54Z

Have you got any pointers where to start looking?

nicoddemus · 2017-10-24T22:09:41Z

I would start debugging here:

pytest/_pytest/python.py

Lines 731 to 732 in 5631a86

    
           def parametrize(self, argnames, argvalues, indirect=False, ids=None, 
        
                           scope=None):

And here:

pytest/_pytest/fixtures.py

Lines 166 to 192 in 5631a86

    
           def reorder_items(items): 
        
               argkeys_cache = {} 
        
               for scopenum in range(0, scopenum_function): 
        
                   argkeys_cache[scopenum] = d = {} 
        
                   for item in items: 
        
                       keys = OrderedDict.fromkeys(get_parametrized_fixture_keys(item, scopenum)) 
        
                       if keys: 
        
                           d[item] = keys 
        
               return reorder_items_atscope(items, set(), argkeys_cache, 0) 
        
           def reorder_items_atscope(items, ignore, argkeys_cache, scopenum): 
        
               if scopenum >= scopenum_function or len(items) < 3: 
        
                   return items 
        
               items_done = [] 
        
               while 1: 
        
                   items_before, items_same, items_other, newignore = \ 
        
                       slice_items(items, ignore, argkeys_cache[scopenum]) 
        
                   items_before = reorder_items_atscope( 
        
                       items_before, ignore, argkeys_cache, scopenum + 1) 
        
                   if items_same is None: 
        
                       # nothing to reorder in this scope 
        
                       assert items_other is None 
        
                       return items_done + items_before 
        
                   items_done.extend(items_before) 
        
                   items = items_same + items_other 
        
                   ignore = newignore

ghost · 2017-10-24T22:24:58Z

Hi,
I would like to provide a fix for this, but want to make sure I understood the problem correctly.

From preliminary research it seems like tests order is determined in reorder_items function in fixtures.py, which invokes reorder_items_atscope, which in turn calls slice_items.

If comments in the code stand true, the desired order of tests that use parametrized fixtures with different scopes is as follows: session > module > class > function?
Is this the only requirement for determining tests order?

I am asking this because current solution is not easy to grasp and seems somewhat overly complicated if we just want to have tests ordered by their scope.

@nicoddemus Any clarification on the subject would be more than welcome.

nicoddemus · 2017-10-24T23:39:38Z

@kchomski-reef thanks for the interest.

You are half right: the purpose is to reorder test itemswithin a scope to minimize fixture setup/teardown, when those fixtures are parametrized.

For example, consider this session fixture:

@pytest.fixture(scope='session', params=[1, 2])
def session_fix(request):
    # let's pretend this actually does some heavy weight setup/teardown 
    return request.param

Consider this test file:

def test_normal_1(): pass

def test_uses_fix_1(session_fix): pass
def test_uses_fix_2(session_fix): pass

def test_normal_2(): pass

If we don't do any reordering, the tests would execute in this order:

test_normal_1
test_uses_fix_1[session_fix(1)]
test_uses_fix_1[session_fix(2)]
test_uses_fix_2[session_fix(1)]
test_uses_fix_2[session_fix(2)]
test_normal_2

IOW, this would cause session_fix to be created with parameter 1 for test_uses_fix_1, then be destroyed so a new session_fix can be created with parameter 2 and execute test_uses_fix_1 again with the new fixture. The same happens for test_uses_fix_2 and for every other test which uses those fixtures.

A key fact to understand all this and that is not clear in the documentation is that at any given time only a single instance of session_fix can exist, that's why it needs to be destroyed between each test function call which parametrizes it.

The reorder algorithm thus tries to reorder the tests to minimize fixture setup/teardown:

test_normal_1
test_uses_fix_1[session_fix1]
test_uses_fix_2[session_fix1]
test_uses_fix_1[session_fix2]
test_uses_fix_2[session_fix2]
test_normal_2

This order now allow us to create session_fix(1) once, execute test_uses_fix_1 and test_uses_fix_2 with that fixture, then create session_fix(2) and execute the tests, and so on.

The same is valid for the other scopes other than function.

Let me know if something is not clear on my explanation.

bilderbuchi · 2017-10-25T06:45:47Z

From looking at the code the first time, you could try this hunch of mine: The source of the non-determinism you saw could be that you are iterating through dicts.
This call could maybe use a sorted wrapped around the generator expression in the argument, otherwise you are feeding the contents of a dict to OrderedDict:

pytest/_pytest/fixtures.py

Line 206 in 5631a86

newargkeys = OrderedDict.fromkeys(k for k in argkeys if k not in ignore)

Alternatively, you could replace this {} (which becomes the argkeys above afaict) with an OrderedDict:

pytest/_pytest/fixtures.py

Line 169 in 5631a86

argkeys_cache[scopenum] = d = {}

bilderbuchi · 2017-10-25T06:56:18Z

OK, I just put my money where my mouth is and tested this myself - it doesn't work. 😞
Also, I observe the exact same (wrong) order with the repro above, and it stays the same on successive pytest runs, so it's probably not a dict issue as I guessed above.

ghost · 2017-10-25T14:11:45Z

@nicoddemus Thank you for such a detailed explanation.
I'll try to do some more debugging and I'll get back to you if I come up with a possible solution or if I'll have more questions.

nicoddemus · 2017-10-25T14:27:13Z

Sounds good, thanks!

ghost · 2017-10-25T16:50:52Z

@nicoddemus: After reading your explanation, looking more into the code and checking back OP's examples there doesn't seem to be a bug at all.

Taking into account what you wrote:

the purpose is to reorder test items within a scope to minimize fixture setup/teardown, when those fixtures are parametrized.

and

A key fact to understand all this and that is not clear in the documentation is that at any given time only a single instance of session_fix can exist, that's why it needs to be destroyed between each test function call which parametrizes it.

the 2nd example posted by OP seems to be perfectly valid - the order seems random at first, but after a closer look it can be seen that order of tests is optimal to avoid unnecessary fixtures setups and teardowns:

# OP 2nd example results
test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[3-c] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-a] PASSED

f1 is created 4 times and f2 is created 6 times for a total of 10.

If we would like to have the same order for class, module and session fixtures as we have for function scoped fixtures:

# OP 1st example results
test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[3-a] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-c] PASSED

then it's not optimal, because f1 is created only 3 times, but f2 is created 9 times, for a total of 12. It's OK for function scoped fixtures because they get setup and tore down every time, so we simply don't care, but for higher scoped fixtures current solution seems to work perfectly fine and as expected.

bilderbuchi · 2017-10-25T17:04:54Z

Ha, awesome reasoning! Didn't see the forest for the trees!

nicoddemus · 2017-10-25T21:54:16Z

@kchomski-reef indeed your reasoning seems correct to me, thanks for taking a look into this.

@rbierbasz-gog what do you think?

rbierbasz · 2017-10-26T09:15:44Z

@kchomski-reef thanks for the great insight. I haven't checked reorder_items code yet, but your reasoning seems correct. I only think that the assumptions on which this functionality is based are naive - that all paremetrized fixtures have the same weight. In real world f1 fixture from my example can have much more expensive setup/teardown, or there can be more fixtures depending on it. It would be great if I could control the order somehow. I tried indirect parametrization:

import itertools
import pytest

f1_params = (1, 2, 3)
@pytest.fixture(scope="module", params=f1_params)
def f1(request):
    pass
    
f2_params = ('a', 'b', 'c')
@pytest.fixture(scope="module", params=f2_params)
def f2(request):
    pass

@pytest.mark.parametrize('f1,f2', itertools.product(f1_params, f2_params), indirect=True)
def test(f1, f2):
    pass

But it breaks fixtures' scopes (#570). The only working solutions for me now is moving one parametrized fixture to higher scope:

import pytest

@pytest.fixture(scope="session", params=(1, 2, 3))
def f1_meta(request):
    return request.param

@pytest.fixture(scope="module")
def f1(f1_meta):
    pass
    
@pytest.fixture(scope="module", params=('a', 'b', 'c'))
def f2(request):
    pass

def test(f1, f2):
    pass

But it seems hacky and it's not scalable.
Is there any way of achieving it? Some hook maybe?

ghost · 2017-10-26T10:27:05Z

@rbierbasz-gog I totally agree. I came up to the same conclusion while investigating this issue.

It would be really nice to have a possibility to indicate the importance/weight of the fixture and reorder tests based on this importance. Something like additional keyword argument with default value maybe.

@nicoddemus do you think it is a possible feature candidate?

nicoddemus · 2017-10-26T10:36:39Z

I think we can cook up a hook that allows users to completely customize fixture ordering, we just have to think of a good interface for that.

Anybody up for opening an issue with an initial proposal for discussion?

rbierbasz · 2017-10-26T16:30:43Z

Setting aside a new hook for the moment - pytest_collection_modifyitems allows to reorder tests, which is enough for my needs:

def pytest_collection_modifyitems(session, config, items):
    items.sort(key=lambda x: x._genid)

It's still hacky (accessing private attribute of Item object) 😄

bilderbuchi · 2017-11-03T18:51:56Z

In any case I think this issue can be closed.

nicoddemus · 2017-11-03T20:03:05Z

Closing then.

Anybody up for opening an issue with an initial proposal for discussion?

That invitation still stands though! We would need a few use cases to come up with a hook that is useful for those cases.

satishmovvar · 2018-01-26T02:07:22Z

@nicoddemus @kchomski-reef, All,

I seems to have a use case where I wanted to control how the tests get created with parametrized fixtures; The main reason I need this feature is because my test results are dependent on previous test result. At time I need to save previous tests result and then compare it

I want to sequence the tests in a predictable order as originally described in this issue even when the fixtures scopes are at class/module level.

OP 1st example results

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[3-a] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-c] PASSED

satishmovvar · 2018-01-26T13:50:52Z

Below is a little more concrete scenario why my tests are need to be in a particular sequence->
I am testing volume function on an amplifier. I need to check change in volume level has changed the output gain; I have different speakers outputs to test across.
I store output level measured with previous test and compare it with next test. So now the order of fixture usage/test execution is very important and I want to control so that I can predict the outcome.

setInputVoltage & creat ListenerObj
setInputFreq
setVolume
listenAudio
4.a check the output freq of the audio compare with input (test all channels -varChannel)
4.b test if previous volumeLevel was present then test (test all channels -varChannel)
else go to next test and compare previous setting

@pytest.fixture(scope="module", autouse=True)
def listener(request):
	# listenr object is capable of listening
	listenerObj = listener()
	request.module.listener = listenerObj
	
	return listener

@pytest.fixture(scope="module", params=[110, 220], autouse=True)
def setInputVoltage(request):
	#set input voltage
	return request.param


@pytest.fixture(scope="class", params=[100, 200, 500], autouse=True)
def setInputFreq(request):
	# set input frequency
	return request.param
	
@pytest.fixture(scope="class", params=[10, 20, 30, 40])
def setVolume(request):
	# set volume 
	return request.param

@pytest.fixture(scope="class")
def listenAudio(request, listener):
	# measures the output all channels out as list
	listener.listen()
	def fin():
		listener.store()
	request.addfinalizer(fin)

	
@pytest.fixture(scope="function", params=[1,2,3])
def varChannel(request):
	# declaring it as fixture to create muliple tests for each channel
	return request.param

def test_amplifierIsOn():
	
@pytest.mark.usefixtures("listenAudio", "setVolume")
class Test_AudioLevel:
	def test_outputFreq(self, setInputFreq):
		assert 
		
	def test_volumeLevel(request, setVolume, varChannel):
		if (listner.storedData):
			assert listener.currentdata[varChannel] > listner.storedData[varChannel]

gavincyi · 2018-04-12T02:17:39Z

Totally agree with @rbierbasz-gog. The equal-weighted cost of setup/teardown all fixtures is not a pragmatic assumption while the users would like to customize the ordering if they really consider the performance.

Could we have one more parameter, e.g. priority (between 0 and 100), for the users the customize the cost of setup/teardown fixture? For example,

@pytest.fixture(scope="session", params=[1, 2, 3], priority=100)
def f1(request):
    pass

@pytest.fixture(scope="session", params=[a, b, c], priority=50)
def f2(request):
    pass

def test_f(f1, f2):
    pass

The expected order is then

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[3-a] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-c] PASSED

nicoddemus · 2018-04-12T02:22:15Z

@satishmovvar and gavincyi perhaps we should create a new issue?

gavincyi · 2018-04-12T06:35:16Z

Good point to have a new issue rather than stuck in a closed ticket. #3393 created.

nicoddemus added topic: parametrize related to @pytest.mark.parametrize type: bug problem that needs to be addressed labels Oct 18, 2017

nicoddemus closed this as completed Nov 3, 2017

pytestbot mentioned this issue Apr 4, 2018

Applying multiple skipifs to parametrized test using param(marks=[]) is order dependent #3368

Closed

This was referenced Apr 12, 2018

Duplicated fixture setup even though the previous cached fixture can be used #3392

Closed

Customize the fixture ordering on the same scope level #3393

Open

smarie mentioned this issue Dec 19, 2018

Explicitly handle session- and module- scoped fixtures with @saved_fixture smarie/python-pytest-harvest#17

Closed

smarie mentioned this issue Dec 21, 2018

Provide a parameter to guarantee fixture uniqueness in @fixture smarie/python-pytest-cases#18

Open

This was referenced May 20, 2019

Surprising parametrized fixtures order #5054

Closed

[WIP] Minimize calls to session, module, and class-scoped fixtures by changing test order #3551

Closed

smarie mentioned this issue Jun 18, 2019

"Optimal" tests reordering - synthesis & strategy #5460

Open

YS-L mentioned this issue Feb 18, 2023

[DEV-1113] Refactor integration tests to be more easily extensible to multiple backends featurebyte/featurebyte#781

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unexpected order of tests using parameterized fixtures #2846

Unexpected order of tests using parameterized fixtures #2846

rbierbasz commented Oct 17, 2017 •

edited

Loading

nicoddemus commented Oct 18, 2017

bilderbuchi commented Oct 24, 2017

nicoddemus commented Oct 24, 2017

ghost commented Oct 24, 2017

nicoddemus commented Oct 24, 2017

bilderbuchi commented Oct 25, 2017

bilderbuchi commented Oct 25, 2017

ghost commented Oct 25, 2017

nicoddemus commented Oct 25, 2017

ghost commented Oct 25, 2017

bilderbuchi commented Oct 25, 2017

nicoddemus commented Oct 25, 2017 •

edited

Loading

rbierbasz commented Oct 26, 2017

ghost commented Oct 26, 2017 •

edited by ghost

Loading

nicoddemus commented Oct 26, 2017

rbierbasz commented Oct 26, 2017

bilderbuchi commented Nov 3, 2017

nicoddemus commented Nov 3, 2017

satishmovvar commented Jan 26, 2018

satishmovvar commented Jan 26, 2018 •

edited

Loading

gavincyi commented Apr 12, 2018 •

edited

Loading

nicoddemus commented Apr 12, 2018

gavincyi commented Apr 12, 2018

Unexpected order of tests using parameterized fixtures #2846

Unexpected order of tests using parameterized fixtures #2846

Comments

rbierbasz commented Oct 17, 2017 • edited Loading

nicoddemus commented Oct 18, 2017

bilderbuchi commented Oct 24, 2017

nicoddemus commented Oct 24, 2017

ghost commented Oct 24, 2017

nicoddemus commented Oct 24, 2017

bilderbuchi commented Oct 25, 2017

bilderbuchi commented Oct 25, 2017

ghost commented Oct 25, 2017

nicoddemus commented Oct 25, 2017

ghost commented Oct 25, 2017

bilderbuchi commented Oct 25, 2017

nicoddemus commented Oct 25, 2017 • edited Loading

rbierbasz commented Oct 26, 2017

ghost commented Oct 26, 2017 • edited by ghost Loading

nicoddemus commented Oct 26, 2017

rbierbasz commented Oct 26, 2017

bilderbuchi commented Nov 3, 2017

nicoddemus commented Nov 3, 2017

satishmovvar commented Jan 26, 2018

OP 1st example results

satishmovvar commented Jan 26, 2018 • edited Loading

gavincyi commented Apr 12, 2018 • edited Loading

nicoddemus commented Apr 12, 2018

gavincyi commented Apr 12, 2018

rbierbasz commented Oct 17, 2017 •

edited

Loading

nicoddemus commented Oct 25, 2017 •

edited

Loading

ghost commented Oct 26, 2017 •

edited by ghost

Loading

satishmovvar commented Jan 26, 2018 •

edited

Loading

gavincyi commented Apr 12, 2018 •

edited

Loading