WIP Change outcome to 'passed' for xfail unless it's strict #1795

hackebrot · 2016-08-05T18:31:22Z

I am not entirely sure if this is the right way to do it.

This is very much WIP as the tests will be failing.

Can anybody help me out with this one, please 😃

Resolves #1546

nicoddemus · 2016-08-05T23:04:32Z

I think you are in the right direction; I took a quick look on the failing junitxml tests, but I believe they were not entirely correct in the first place, or at least the new meaning makes more sense.

I will review this more carefully tomorrow as I'm short on time right now.

hackebrot · 2016-08-05T23:09:52Z

Thanks @nicoddemus! 🙇

Have a nice evening. 😄

nicoddemus · 2016-08-09T21:28:21Z

_pytest/skipping.py

@@ -260,7 +266,7 @@ def pytest_report_teststatus(report):
    if hasattr(report, "wasxfail"):
        if report.skipped:
            return "xfailed", "x", "xfail"
-        elif report.failed:
+        elif report.passed:


@fabioz I think this will mean you no longer have to do special handling for this in pydev_runfiles_pytest2.py, correct?

@nicoddemus do you think we need to check for strict_xfail here too?

No need: strict xfails now have report set to failed and will appear as a normal failure.

okay, that's what I thought. Thanks!

nicoddemus · 2016-08-09T21:54:12Z

We should also add somewhere in the docs that the skipping plugin adds a wasxfail attribute to TestReport objects, although I'm not sure where the best place to do that would be.

nicoddemus · 2016-08-09T22:22:36Z

_pytest/skipping.py

@@ -245,7 +246,12 @@ def pytest_runtest_makereport(item, call):
                rep.outcome = "skipped"
                rep.wasxfail = evalxfail.getexplanation()
        elif call.when == "call":
-            rep.outcome = "failed"  # xpass outcome
+            strict_default = item.config.getini('xfail_strict')


Hmm I think this is the more correct implementation of "strict xfail" than the one I did... with this, I think we don't even need check_strict_xfail (which is used in pytest_pyfunc_call) in this module anymore at all, because failure/passing will be handled here which seems more appropriate now. 😄

hackebrot · 2016-08-09T22:27:04Z

Hey @nicoddemus! Thank you so much for your thorough code review.

I am currently on vacation (on some Scottish island 😉). Feel free to pick this up.

That'd be awesome, so get we can get 3.0 released soon-ish 😁

hackebrot · 2016-08-12T23:02:35Z

Hey @nicoddemus, I've added some changes. Please let me know your thoughts! 😄

nicoddemus · 2016-08-15T22:24:46Z

@hackebrot thanks! The changes look good, we just have to fix the junitxml tests. They broke because we have changed the semantics slightly: "xpass" for non-strict xfail now became a "passing test", instead of a "skipped test". I'm not sure, it seems sensible to me.

hackebrot · 2016-08-15T22:54:09Z

Should an XPASS in non-strict mode be:

a skipped test
a passed test

(It's a failure in strict-mode)

@pytest-dev/core what do you think? 😄

The-Compiler · 2016-08-16T08:46:44Z

XPASS being a skipped test sounds really weird - what's the rationale of that? The test did run (and passed), so surely it wasn't skipped? So +1 for passed.

RonnyPfannschmidt · 2016-08-16T09:12:38Z

its clearly a "passed" test when being non-strict

hackebrot · 2016-08-16T10:02:12Z

I don't think it should be skipped, but that looks to be what is reported on master, see https://github.com/pytest-dev/pytest/blob/master/testing/test_junitxml.py#L103

RonnyPfannschmidt · 2016-08-16T10:04:57Z

that would be a bug then ^^

hackebrot · 2016-08-16T14:03:09Z

Okay, cool 😄 so to conclude:

Non-strict mode (default)

xfail > test fails > xfailed (passed)
xfail > test passes > xpass (passed)

Strict mode

xfail(strict=True) > test fails > xfailed (passed)
xfail(strict=True) > test passes > failed (failed)

Everyone happy?

nicoddemus · 2016-08-16T14:07:53Z

I don't think it should be skipped, but that looks to be what is reported on master, see

That's what I meant by:

They broke because we have changed the semantics slightly: "xpass" for non-strict xfail now became a "passing test", instead of a "skipped test".

😁

Okay, cool 😄 so to conclude:
(snip table)

On that table, what does > xfailed (passed) mean exactly? That report.outcome == "passed" (or IOW report.passed is True)? Just to clarify.

hackebrot · 2016-08-16T16:11:07Z

Test Code

# test_nope.py

import pytest

@pytest.mark.xfail
def test_fail_non_strict():
    assert False

@pytest.mark.xfail
def test_pass_non_strict():
    assert True

@pytest.mark.xfail(strict=True)
def test_fail_strict():
    assert False

@pytest.mark.xfail(strict=True)
def test_pass_strict():
    assert True

# conftest.py

def pytest_runtest_logreport(report):
    print()
    print("report.nodeid   '{}'".format(report.nodeid))
    print("report.when     '{}'".format(report.when))
    print("report.outcome  '{}'".format(report.outcome))
    print("report.wasxfail '{}'".format(getattr(report, 'wasxfail', 'NOT SET')))
    print()

hackebrot · 2016-08-16T16:16:00Z

master

============================= test session starts ==============================
collected 4 items

test_nope.py 
report.nodeid   'test_nope.py::test_fail_non_strict'
report.when     'setup'
report.outcome  'passed'
report.wasxfail 'NOT SET'

x
report.nodeid   'test_nope.py::test_fail_non_strict'
report.when     'call'
report.outcome  'skipped'
report.wasxfail ''


report.nodeid   'test_nope.py::test_fail_non_strict'
report.when     'teardown'
report.outcome  'passed'
report.wasxfail 'NOT SET'


report.nodeid   'test_nope.py::test_pass_non_strict'
report.when     'setup'
report.outcome  'passed'
report.wasxfail 'NOT SET'

X
report.nodeid   'test_nope.py::test_pass_non_strict'
report.when     'call'
report.outcome  'failed'
report.wasxfail ''


report.nodeid   'test_nope.py::test_pass_non_strict'
report.when     'teardown'
report.outcome  'passed'
report.wasxfail 'NOT SET'


report.nodeid   'test_nope.py::test_fail_strict'
report.when     'setup'
report.outcome  'passed'
report.wasxfail 'NOT SET'

x
report.nodeid   'test_nope.py::test_fail_strict'
report.when     'call'
report.outcome  'skipped'
report.wasxfail ''


report.nodeid   'test_nope.py::test_fail_strict'
report.when     'teardown'
report.outcome  'passed'
report.wasxfail 'NOT SET'


report.nodeid   'test_nope.py::test_pass_strict'
report.when     'setup'
report.outcome  'passed'
report.wasxfail 'NOT SET'

F
report.nodeid   'test_nope.py::test_pass_strict'
report.when     'call'
report.outcome  'failed'
report.wasxfail 'NOT SET'


report.nodeid   'test_nope.py::test_pass_strict'
report.when     'teardown'
report.outcome  'passed'
report.wasxfail 'NOT SET'


=========================== short test summary info ============================
XFAIL test_nope.py::test_fail_non_strict
XFAIL test_nope.py::test_fail_strict
XPASS test_nope.py::test_pass_non_strict 

=================================== FAILURES ===================================
_______________________________ test_pass_strict _______________________________
[XPASS(strict)] 
================ 1 failed, 2 xfailed, 1 xpassed in 0.01 seconds ================

nicoddemus · 2016-08-17T01:10:05Z

Here's the output using @hackebrot's branch (only for when == call):

============================= test session starts =============================
platform win32 -- Python 3.5.0, pytest-2.9.3.dev0, py-1.4.31, pluggy-0.3.1
rootdir: X:\pytest, inifile: tox.ini
plugins: hypothesis-3.1.0, xdist-1.14
collected 4 items

tmp\test_nope.py x
report.nodeid   'tmp/test_nope.py::test_fail_non_strict'
report.outcome  'skipped'
report.wasxfail ''

X
report.nodeid   'tmp/test_nope.py::test_pass_non_strict'
report.outcome  'passed'
report.wasxfail ''

x
report.nodeid   'tmp/test_nope.py::test_fail_strict'
report.outcome  'skipped'
report.wasxfail ''

F
report.nodeid   'tmp/test_nope.py::test_pass_strict'
report.outcome  'failed'
report.wasxfail 'NOT SET'

=========================== short test summary info ===========================
XFAIL tmp/test_nope.py::test_fail_non_strict
XFAIL tmp/test_nope.py::test_fail_strict
XPASS tmp/test_nope.py::test_pass_non_strict
================================== FAILURES ===================================
______________________________ test_pass_strict _______________________________
[XPASS(strict)]
=============== 1 failed, 2 xfailed, 1 xpassed in 0.02 seconds ================

And here's the junit XML file:

<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="pytest" skips="2" tests="4" time="0.020">
    <testcase classname="tmp.test_nope" file="tmp\test_nope.py" line="2" name="test_fail_non_strict" time="0.0">
        <skipped message="expected test failure"></skipped>
    </testcase>
    <testcase classname="tmp.test_nope" file="tmp\test_nope.py" line="6" name="test_pass_non_strict" time="0.0"></testcase>
    <testcase classname="tmp.test_nope" file="tmp\test_nope.py" line="10" name="test_fail_strict" time="0.0">
        <skipped message="expected test failure"></skipped>
    </testcase>
    <testcase classname="tmp.test_nope" file="tmp\test_nope.py" line="14" name="test_pass_strict" time="0.0">
        <failure message="[XPASS(strict)] ">[XPASS(strict)]</failure>
    </testcase>
</testsuite>

Parsing the XML file, we can see:

test_fail_non_strict: SKIPPED
test_pass_non_strict: PASSED
test_fail_strict: SKIPPED
test_pass_strict: FAILED

IMHO the XML now looks better because a non-strict xpass now reports PASSED, so the tests in test_junitxml.py should be updated to reflect this.

For comparison, here's the same summary for the master branch:

test_fail_non_strict: SKIPPED
test_pass_non_strict: SKIPPED
test_fail_strict: SKIPPED
test_pass_strict: FAILED

The-Compiler · 2016-08-17T06:53:29Z

That sounds right to me - so what's left to do here? I'm guessing @hackebrot needs to adjust the existing tests and then this is ready for review?

hackebrot · 2016-08-17T07:36:06Z

I find it slightly confusing thatxfail results in SKIPPED. The test item is being run, so I don't quite see how this should result in skipped.

I'd much rather expect the following:

test_fail_non_strict: PASSED
test_pass_non_strict: PASSED
test_fail_strict: PASSED
test_pass_strict: FAILED

The-Compiler · 2016-08-17T08:09:34Z

@hackebrot Hmm, you do have a point there. On a second thought, I agree (except with xfail(run=False) I guess...)

Change outcome to 'passed' for xfail unless it's strict

4fc20d0

nicoddemus reviewed Aug 9, 2016
View reviewed changes

hackebrot added 4 commits August 12, 2016 23:18

Update unittest test to expect failure for an unexpected success

10a6ed1

Treat unittest.expectedFailure pass as a failure

296f42a

Extend test to verify longrepr in stdout

14a4dd0

Set wasxfail only for xpass w/o strict and else set longrepr

225341c

Update test_junitxml.py to interpret XPASS as passed

55ec1d7

hackebrot mentioned this pull request Aug 16, 2016

use of 'pytest' in setup.cfg collides with 'pytest' distutils command #567

Closed

nicoddemus added this to the 3.0 milestone Aug 16, 2016

bluetech mentioned this pull request Jan 12, 2021

Unittest unexpected success should be translated to XPASS #7393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP Change outcome to 'passed' for xfail unless it's strict #1795

WIP Change outcome to 'passed' for xfail unless it's strict #1795

hackebrot commented Aug 5, 2016

nicoddemus commented Aug 5, 2016

hackebrot commented Aug 5, 2016

nicoddemus Aug 9, 2016

hackebrot Aug 15, 2016

nicoddemus Aug 15, 2016 •

edited

Loading

hackebrot Aug 15, 2016

nicoddemus commented Aug 9, 2016

nicoddemus Aug 9, 2016

hackebrot commented Aug 9, 2016

hackebrot commented Aug 12, 2016

nicoddemus commented Aug 15, 2016

hackebrot commented Aug 15, 2016

The-Compiler commented Aug 16, 2016

RonnyPfannschmidt commented Aug 16, 2016

hackebrot commented Aug 16, 2016

RonnyPfannschmidt commented Aug 16, 2016

hackebrot commented Aug 16, 2016

nicoddemus commented Aug 16, 2016

hackebrot commented Aug 16, 2016

hackebrot commented Aug 16, 2016

nicoddemus commented Aug 17, 2016

The-Compiler commented Aug 17, 2016

hackebrot commented Aug 17, 2016

The-Compiler commented Aug 17, 2016

WIP Change outcome to 'passed' for xfail unless it's strict #1795

WIP Change outcome to 'passed' for xfail unless it's strict #1795

Conversation

hackebrot commented Aug 5, 2016

nicoddemus commented Aug 5, 2016

hackebrot commented Aug 5, 2016

nicoddemus Aug 9, 2016

Choose a reason for hiding this comment

hackebrot Aug 15, 2016

Choose a reason for hiding this comment

nicoddemus Aug 15, 2016 • edited Loading

Choose a reason for hiding this comment

hackebrot Aug 15, 2016

Choose a reason for hiding this comment

nicoddemus commented Aug 9, 2016

nicoddemus Aug 9, 2016

Choose a reason for hiding this comment

hackebrot commented Aug 9, 2016

hackebrot commented Aug 12, 2016

nicoddemus commented Aug 15, 2016

hackebrot commented Aug 15, 2016

The-Compiler commented Aug 16, 2016

RonnyPfannschmidt commented Aug 16, 2016

hackebrot commented Aug 16, 2016

RonnyPfannschmidt commented Aug 16, 2016

hackebrot commented Aug 16, 2016

Non-strict mode (default)

Strict mode

nicoddemus commented Aug 16, 2016

hackebrot commented Aug 16, 2016

Test Code

hackebrot commented Aug 16, 2016

master

nicoddemus commented Aug 17, 2016

The-Compiler commented Aug 17, 2016

hackebrot commented Aug 17, 2016

The-Compiler commented Aug 17, 2016

nicoddemus Aug 15, 2016 •

edited

Loading