I'm submitting a ...

• bug report
• feature request
• support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

My team relies heavily on custom categorization of test results to make reviewing reports easier for our development teams. Currently, you cannot categorize "Passed" tests as status details are only included for broken or failed tests that have an exception.

Closes #580

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

What is the expected behavior?

We would like to be able to also categorize passed tests without having to mark those tests as failed in order to get a status details attached to the result

What is the motivation / use case for changing the behavior?

We have tagging processes in place for our team to flag tests with known product issues and known test issues. Sometimes these known issues are the result of a flaky test or product issue that does not reproduce all of the time.

• It is important for test reviewers to know when a test fails with a known issue so they can focus on any new failures instead.
• It's also important for test developers to known when a passing test has a known issue because it isn't always communicated back to us when a bug gets fixed (ie we need to remove the test tag)

To accomplish this we currently have to group scenarios with known issues into separate categories -- "failed" or "passed" (like below). However this means we have to mark both sets of scenarios as "failed" so that the exception is included in the resulting report allowing us to categorize the results. This is less than ideal because when reviewing the history of a flaky issue, it's not clear how frequently the test actually passes / fails without clicking into each result to view the category

Suggested change

One lightweight solution would be to also consider the error_message when getting a scenarios status details. This would allow test developers to attach errors messages to other tests that did not fail with an exception. The cool thing about this is the test developer could also include skip reasons for skipped tests by simply setting the skip reason as the error message

# attach message to first step for passed test with known issue
if scenario.status == Status.passed and test_has_known_issue:
    list(scenario.all_steps)[0].error_message = "Test passed with known issues"

Please tell us about your environment:

Allure version: 2.15.0
Test framework: behave=1.2.6
Allure adaptor: allure-behave==2.8.24

Other information