pytest: test re-run should also include resetup of all the test-scoped fixtures

[bubenkoff]

This is obviously required if fixtures are mutable
For example: test logs in using a 'browser' fixture and checks some content on the page loaded by ajax
It can be flaky if there's some timeout for ajax to load
But with this plugin it will not work as test will not be able to log in again as it's already logged in

[bubenkoff]

This is what i would see this plugin does:

def call_and_report(plugin, rerun_info, terminalreporter, item, when, log=True, **kwds):
    """Skip falure reporting if test run is not last."""
    call = plugin.call_runtest_hook(item, when, **kwds)
    if not rerun_info['last']:
        if call.excinfo:
            rerun_info['excinfo'] = call.excinfo
            call.excinfo = None
            log = False
            if terminalreporter:
                terminalreporter.write_line('Retrying potentially flaky test...')
    hook = item.ihook
    report = hook.pytest_runtest_makereport(item=item, call=call)
    if log:
        hook.pytest_runtest_logreport(report=report)
    if plugin.check_interactive_exception(call, report):
        hook.pytest_exception_interact(node=item, call=call, report=report)
    return report


def pytest_runtest_protocol(item, nextitem):
    """Rerun flaky tests."""
    runner = item.config.pluginmanager.getplugin("runner")
    old_call_and_report = runner.call_and_report
    terminalreporter = item.config.pluginmanager.getplugin('terminalreporter')
    max_runs = item.config.getoption('max_runs')
    try:
        def run():
            runner.pytest_runtest_protocol(item, nextitem)

        for index in xrange(max_runs):
            rerun_info = dict(last=index == max_runs - 1)
            runner.call_and_report = partial(call_and_report, runner, rerun_info, terminalreporter)
            run()
            if 'excinfo' not in rerun_info:
                return True
    finally:
        runner.call_and_report = old_call_and_report

this works as expected with setup/teardown of the fixtures

[Jeff-Meadows]

Thanks for the report. I think you're right, that it should recreate fixtures. I'll work on a fix.

[Jeff-Meadows]

Hi - I haven't forgotten about this, but I haven't made a ton of progress yet. I'll keep this issue updated when I do.

[bubenkoff]

i tried as well, without changes to the core of pytest it's not possible,
im affraid
and changes are not at all cosmetic

On Fri, Aug 28, 2015 at 8:44 PM Jeff Meadows [email protected]
wrote:

Hi - I haven't forgotten about this, but I haven't made a ton of progress
yet. I'll keep this issue updated when I do.

—
Reply to this email directly or view it on GitHub
#53 (comment).

[paurullan]

I patched the flaky_pytest_plugin.py with the code above and it seems to work. Altought flaky's own tests fail afterwards, so I will have to check what is going on.

Have you made any advance in this issue?

[Jeff-Meadows]

@bubenkoff @mangeld Please check PR #76 to see if it fixes the issues you're having with flaky.