Support generator cases

[smarie]

Currently if we wish to perform some setup/teardown activity on a particular case, we have to create a distinct fixture to do so and reference it in the case.

Generator case functions could directly be supported out of the box instead.

[eddiebergman]

Seems like some fun and something I was looking for before, any pointers?

[smarie]

A case function is "just" a pytest_cases.lazy_value fed to a @pytest_cases.parametrize, or a fixture. It is turned into one or the other depending on whether it requires parametrization.

Below you can find info on lazy values, but the more I think about it, the more I think that the simple way is to detect generator cases and force-turn them into fixtures (the same way as for parametrized case functions or case functions requiring fixtures). Much simpler and requires no pytest hack since fixtures suport this.

-- Old post, probably bad idea

Source : https://github.com/smarie/python-pytest-cases/blob/main/src/pytest_cases/common_pytest_lazy_values.py#L486

There is an example in the API reference : https://smarie.github.io/python-pytest-cases/pytest_goodies/#parametrize
You can remove everything but the lazy_value, and put a breakpoint in the whatfun function to see the stack calling it.

From what I remember there is a get_lazy_args function that gets called from all entry points (fixtures and tests): https://github.com/smarie/python-pytest-cases/blob/main/src/pytest_cases/common_pytest_lazy_values.py#L533

I have no clue on where and when (which pytest step) we should interact with lazy values to have this setup/teardown thing. Maybe this is a bad idea ?

[smarie]

Below you can find info on lazy values, but the more I think about it, the more I think that the simple way is to detect generator cases and force-turn them into fixtures (the same way as for parametrized case functions or case functions requiring fixtures). Much simpler and requires no pytest hack since fixtures suport this.

This is done in case_to_argvalues:

python-pytest-cases/src/pytest_cases/case_parametrizer_new.py

Line 388 in 501f9fb

def case_to_argvalues(host_class_or_module, # type: Union[Type, ModuleType]

You see that there is one if branch that creates _LazyValueCaseParamValue and another one that creates a fixture and then refers to it using a _FixtureRefCaseParamValue. Detecting a generator case could happen before the if in order to force the second situation.

[eddiebergman]

Hi @smarie, I'll take a look throughout this week, thanks for the detailed write up :)

[jenstroeger]

Thank you guys for an interesting package!

I think I came across a problem that aligns with this discussion. Assume a test file test_numbers.py with the following content:

import typing
import pytest_cases

def case_one() -> typing.Iterator[int]:
    yield 1

def case_two() -> typing.Iterator[int]:
    yield 2

@pytest_cases.parametrize_with_cases("number", cases=".")
def test_number(number: int) -> None:
    assert isinstance(number, int)

This fails with

tests/test_number.py::test_number[one] FAILED
tests/test_number.py::test_number[two] FAILED
...
  File "/path/to/tests/test_number.py", line 14, in test_number
    assert isinstance(number, int)
AssertionError: assert False
 +  where False = isinstance(<generator object case_one at 0x104e69c40>, int)

I would have expected that the value passed into the test is an int.

However, I think this becomes more interesting if the test case produces more than one value, e.g.

def case_numbers() -> typing.Iterator[int]:
   yield from range(10)

Curious to get your thoughts on this!

[smarie]

Hi @jenstroeger thanks for the kind words ! Sorry for answering very late.

Yes today the case functions are normal functions, therefore if you provide a generator (a function containing a yield statement) then pytest-cases will not handle it particularly well. This is because not all case functions are turned into pytest fixtures today. The discussion above concerns supporting yield statements in case functions, BUT ONLY ONE (like pytest fixtures). This can be done by forcing the conversion of case functions containing yield to a fixture, following the writeup in #229 (comment)

Supporting case functions with multiple yield statements (as you suggest) was, ironically, supported in the very first versions of pytest-cases, with the @cases_generator decorator. I also created another plugin pytest-steps performing this feature for normal pytest tests.

pytest main principle is that all test MUST be independent. Therefore pytest-steps is maybe not a very good idea. Yet, in pytest-cases we handle parameters, not tests. And parameters can be dependent, as long as a failed test for the previous parameter does not modify the test with the next parameter.

Therefore I agree that this could be a good idea. Still, how to make this elegant while generic ? There are implementation details to handle, that can make things very complex.

• What about pytest marks ? marking a single subcase (1 yield) / marking all subcases (all of the yields) ?
• Should we consider the "return" of a generator ?
• How to manage pytest ids one by one ? All at once ?
• How to manage the compliance with the @case decorator ? (ids, filters, tags, etc.). Maybe the best way is to not create a new decorator but to have extra arguments in this decorator ?

If you are still willing to discuss all of this :) I suggest that you recopy this post into a dedicated new issue so that we take the discussion there.

Thanks !