Test run summary bar is yellow but not clear why, after updating to v2.8.2

[edmorley]

STR:

1. In an existing project ([1]), update from pytest 2.7.2 to pytest 2.8.2
2. Run py.test tests/
3. Observe the test run including the summary bar message and colour

Expected:
If the summary bar under pytest v2.7.2 was green, and is now yellow after updating to pytest v2.8.2:

• the cause should be clear in the console output
• the pytest v2.8.x changelog should explain all of the criteria for turning yellow

Actual:
The summary bar is yellow, and reads:

======================== 438 passed, 17 skipped, 1 xfailed, 2 pytest-warnings in 120.96 seconds ========================

The test run console output does not show what the pytest-warnings were (for full test run output see https://emorley.pastebin.mozilla.org/8851873).

The changlog states (there's actually two almost-duplicate entries):

* Summary bar now is colored yellow for warning situations such as: all tests either were skipped or xpass/xfailed, or no tests were run at all (this is a partial fix for issue500).
...
* Summary bar now is colored yellow for warning situations such as: all tests either were skipped or xpass/xfailed, or no tests were run at all (related to issue500). Thanks Eric Siegerman.

I don't believe I meet those criteria (tests were run, and they weren't all skipped/failed), as such either the yellow summary bar is bogus, or else it's due to the pytest-warnings.

If the latter is the case (ie it's due to the pytest-warnings):

1. It would be good to add this to the changelog
2. IMO the warnings should be output to the console without needing any additional parameters

Many thanks :-)

[1] On the offchance someone wants to repro, it's: https://github.com/mozilla/treeherder revision 169a3d8b854bb3ce5b9914f35aa2011d2d1a1763 - Vagrant setup instructions at http://treeherder.readthedocs.org/installation.html - then update to pytest 2.8.2

[The-Compiler]

I agree that either the bar should be green without -rw even if there are warnings, or the warnings should be shown by default.

[nicoddemus]

Thanks @edmorley for taking the time to write such a detailed description.

I think at some point it was decided almost as an afterthought to also make the bar yellow if there are warnings, but in retrospect I think this caused more confusion than helped. I think the bar should be green if there are warnings, but the warnings themselves should not be shown by default: I guess most of them will be related to things outside of the control of the end users, for example warnings about plugins using obsolete hookwrappers.

@RonnyPfannschmidt, what are your thoughts?

This is loosely related to #1173.

[RonnyPfannschmidt]

I'm starting to think thearnings systemneeds to differentiate between different kinds of warning

[hpk42]

let's keep it simple. I think we can just turn back to 2.7 behaviour -- don't regard warnings as a serious enough issue to change the color. We could start to differentiate warnings but i think we don't need to invest the effort, really. Stuff breaks. If you ignore warnings, things might at some point break. Such is life. But if your test suite otherwise passes fine, the fact that it with some later release break is not immediately important. We could say that "--strict" turns warnings into errors (and thus things will be read) and generally only promise to be compatible with minor or major releases if the test run was previously fine under "--strict".

FWIW i have also a problem because i am using "black foreground on white background" and yellow is hardly readable.

[The-Compiler]

We could say that "--strict" turns warnings into errors

-1 on that - I unfortunately still have plugins which raise deprecation warnings and ignore my PRs to fix them so far (pytest-dev/pytest-bdd#163 and pytest-dev/pytest-bdd#164), and I'd like to continue using --strict to catch marker typos. IMHO at the very least, it should be possible to pass a list of things to --strict, like --strict=markers,warnings, and --strict would turn everything on.

[Zac-HD]

Closing this issue as out of date, now that Pytest uses Python's standard warnings mechanisms. It's not perfect by any means, but we've replaced the old problems with (fewer!) new ones 😄