pytest.mark not evaluated with pytest_generate_tests

[spaceone]

cat > test_foo.py

import pytest


def test_foo(a, b):
        assert a == 'a'
        assert b == 'b'


def pytest_generate_tests(metafunc):
        data = [[pytest.mark.xfail('reason=foo')('A'), 'b']]
        metafunc.parametrize('a,b', data)

results in:

# py.test test_foo.py                                                                                                                                                                                                                                            
============================================================================================================================== test session starts ==============================================================================================================================
platform linux2 -- Python 2.7.9 -- py-1.4.25 -- pytest-2.6.3
collected 1 items 

test_foo.py F

=================================================================================================================================== FAILURES ====================================================================================================================================
________________________________________________________________________________________________________________________________ test_foo[a0-b] _________________________________________________________________________________________________________________________________

a = <MarkDecorator 'xfail' {'args': ('reason=foo', 'A'), 'kwargs': {}}>, b = 'b'

    def test_foo(a, b):
>       assert a == 'a'
E    assert <MarkDecorator 'xfail' {'args': ('reason=foo', 'A'), 'kwargs': {}}> == 'a'

test_foo.py:5: AssertionError
=========================================================================================================================== 1 failed in 0.02 seconds ===========================================================================================================================

Expected result:
test is marked as XFAILED.

[RonnyPfannschmidt]

That's expected behavior, the mark must wrap around all parameters in the parameter set

In pytest 3.1 we will ship a better mechanism for marking parametersets

[spaceone]

I build up a workaround:

def xfail(set_):
    fails = [x for x in set_ if isinstance(x, type(pytest.mark.xfail()))]
    if fails:
        data = [x.args[0] if isinstance(x, type(pytest.mark.xfail())) else x for x in set_]
        set_ = pytest.mark.xfail(**fails[0].kwargs)(data)
    return set_

data = [xfail(x) for x in data]

[spaceone]

but I can't differentiate a condition from a value :-/ it's both stored in "args".

[RonnyPfannschmidt]

@spaceone one you need to use actual tuples as arguments to encode the args that get extracted

the method used in pytest < 3.1 is pretty fragile

which is why we introduce https://github.com/pytest-dev/pytest/blob/features/_pytest/mark.py#L14 to support declaring it as pytest.param(my, arg, values, marks=[pytest.mark.mymark(...)

the linked code also contains the extraction logic which should give you some hints on how to write the workaround for current py.test

[RonnyPfannschmidt]

but bascially, the extraction mechanism presumes that when given a mark, the last element of the args tuple is the parameter list for the parameterize call

and it will recurse on that

[spaceone]

As you are working on it in further releases, this issue can be closed. I'll find my workaround.