[Feature request] Include tests marked with @org.junit.Ignore in JUnitReport.xml

[plnice]

We often use @org.junit.Ignore annotation to mark tests we plan to fix later. When we are doing multiple changes in the specific area and we predict that some other changes will happen very soon and will break the tests again, for us it's better to mark the test as ignored for some time and go back to it when all the changes are introduced.

Such tests are (correctly) not taken into account when running test suite on FTL. However, we'd like to include ignored tests in JUnitReport.xml so that our CI environment (when the XML report gets parsed) can present ignored tests. In such way we can track which tests were ignored in the given test run.

The end result could be similar to how regular Android unit tests report looks like - sometimes we also have some unit tests marked with @Ignore, they land in the XML report and are correctly parsed by the CI.

[plnice]

Here's an example of an Android unit test results XML with one test marked with @Ignore:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="8" skipped="1" failures="0" errors="0" ...>
  ...
  <testcase name="testName" classname="com.test.ClassName" time="0.0">
    <skipped/>
  </testcase>
  ...
</testsuite>

[pawelpasterz]

@plnice Thanks for pointing this out!

@bootstraponline, current flank implementation filters tests annotated with @Ignore while creating test matrices (they are not sent to TL). As I see it, TL skips ignored tests anyway but they are included in the final report.

My proposal is to introduce new flag, --include-ignored, which would be false by default. User will decide if ignored tests are needed within final xml results, or not.

Let me know what you think.

[bootstraponline]

@pawelpasterz I think Flank already has many flags, ideally we'll look to change default behavior when possible. For this specific use case, it's probably best to always include ignored tests in the JUnit XML. I believe JUnit does this by default for local runs.

[pawelpasterz]

@pawelpasterz I think Flank already has many flags, ideally we'll look to change default behavior when possible. For this specific use case, it's probably best to always include ignored tests in the JUnit XML. I believe JUnit does this by default for local runs.

Sounds like a plan, thanks for answer