Hierarchical pairs of fixtures and values to check against (like a zipped for loop over fixtures and "check" dictionaries)

[sgbaird]

I have two fixtures that each return a class instance where the classes are different from each other, and I want to compare the attributes of each class instance for a fixed set of inputs to a fixed set of expected outputs. The fixed inputs are the same between the two fixtures, but the list of attributes to check and the associated values are different between the two classes.

This question is pretty similar:

If I have this list of tuples:

[(['a', 'b', 'c'], [1, 2, 3]),

 (['d', 'e', 'f'], [4, 5, 6])]

How can I parametrize a test function, so the following pairs are tested:

[('a', 1), ('a', 2), ('a', 3),
 ('b', 1), ('b', 2), ('b', 3),
 ('c', 1), ('c', 2), ('c', 3),

 ('d', 4), ('d', 5), ('d', 6),
 ('e', 4), ('e', 5), ('e', 6),
 ('f', 4), ('f', 5), ('f', 6)]

In my case, since I'd like to "loop" through fixtures, it seemed like I'd need to either use pytest-cases or some custom workaround. I had trouble getting this kind of behavior using two @parametrize decorators, so I went with the solution mentioned above of creating a flat list of the combinations. I set indirect=True so that I evaluate it list-wise, but this throws an error:

..\..\..\..\miniconda3\envs\matbench-genmetrics\lib\site-packages\pytest_cases\fixture_parametrize_plus.py:831: in _parametrize_plus
    raise ValueError("Setting `indirect=True` is not yet supported when at least a `fixure_ref` is present in "
E   ValueError: Setting `indirect=True` is not yet supported when at least a `fixure_ref` is present in the `argvalues`.

Here's the function I mocked up for this use-case:

from typing import Callable
import numpy as np
from numpy.typing import ArrayLike
from pytest_cases import parametrize
@parametrize(
    fixture=flat_fixtures, attr=flat_attributes, check_value=flat_values, indirect=True
)
def test_numerical_attributes(fixture: Callable, attr: str, check_value: ArrayLike):
    """Verify that numerical attributes match the expected values.

    Note that scalars are converted to numpy arrays before comparison.

    Parameters
    ----------
    fixture : Callable
        a pytest fixture that returns an instantiated class operable with getattr
    attr : str
        the attribute to test, e.g. "match_rate"
    check_value : np.ndarray
        the expected value of the attribute checked via ``assert_array_equal``, e.g. [1.0]

    Examples
    --------
    >>> test_numerical_attributes(dummy_gen_metrics, "match_count", expected)
    """
    value = getattr(fixture, attr)
    value = np.array(value) if not isinstance(value, np.ndarray) else value

    assert_array_equal(
        value,
        np.array(check_value),
        err_msg=f"bad value for {dummy_gen_matcher.__class__.__name__}.{attr}",
    )

Maybe I could pass a tuple of (fixture, attr, check_value) with indirect=False. Assuming that works, will I be losing the benefit of using fixtures in the first place?

How would you suggest dealing with this situation? I've spent a long time searching and messing around, so if you have a suggestion or a canonical answer I think I'll go with that.

Related:

• Indirect parametrization of fixture with multiple params with parametrize_with_cases #150

[sgbaird]

Maybe I could pass a tuple of (fixture, attr, check_value) with indirect=False. Assuming that works, will I be losing the benefit of using fixtures in the first place?

Not working either, as the fixture is left as a callable inside the test function.

[smarie]

Thanks @sgbaird for your question.

A simple way to tackle the issue would be to revert the problem: you first create a parametrize fixture that returns a pair of objects (it will therefore return the "zip" directly), and then you define your two "independent" fixtures so that they dependn on the above, and take only the first or second element.

Would that solve your problem ?

[smarie]

See also #284