Skipped Tests with DataProvider appear as failed

[jmoreira18]

TestNG Version

7.7.1

How to reproduce?

Context: I don't want a DataProvider to execute if the test is going to be skipped

1. Create a TestListener that implements both IDataProviderListener and ITestListener
2. Override the onTestStart method to skip if certain conditions are matched (ex: the test contains a specific group)
3. Also override the beforeDataProviderExecution method to skip the DataProvider if the test contains a skip condition
4. Create a TestFile that uses TestListener as Listener
5. Add a test with a DataProvider that meets the skip condition
6. (Optional) Add a test without a DataProvider that meets the skip condition to compare
7. Run the test/s

Expected behavior

Skipping a test with a DataProvider should skip the test and the DataProvider, the test result should show that the test was Skipped.
The test with the DP and one without it should behave the same, everything should be skipped

Actual behavior

The test with the DataProvider is shown as failed with a SkipException

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Here you have a small IntelliJ project containing a test file with two tests, one has the dumb DataProvider:
https://github.com/jmoreira18/TestNG-DataProvider-issue-demo

Code comments

This part of the code is not taking into account that it could be a SkipException, then it's treating it as a failure and failing the test

boolean bubbleUpFailures =
        m_configuration.isPropagateDataProviderFailureAsTestFailure() || bag.isBubbleUpFailures();

if (throwable instanceof TestNGException || bubbleUpFailures) {
        tr.setStatus(ITestResult.FAILURE);
        m_notifier.addFailedTest(arguments.getTestMethod(), tr);
} else {
        tr.setStatus(ITestResult.SKIP);
        m_notifier.addSkippedTest(arguments.getTestMethod(), tr);
}

Thanks in advance!

[krmahadevan]

@jmoreira18 Would you like to create a pull request for this ? We can help you with review and merge.

[jmoreira18]

@krmahadevan Sure! Sounds cool

[krmahadevan]

@jmoreira18 - Thank you for taking the time to create a PR and contribute. We look forward to more contributions from you :)