[BUG] Test failure exists but Gradle 'check' task passes

[tlfeng]

Describe the bug
The Gradle check task may pass, even if test failure occurred.

I'm not sure if this is by design, not to fail the task in the following scenario:

• test failure occurs in YAML REST API BWC test
  example: Add request parameter 'cluster_manager_timeout' and deprecate 'master_timeout' - in Index APIs except index template APIs #2660 (comment)
• test failed by unexpected deprecation warning occurs in YAML REST API test and High-Level-Rest-Client test
  example: Add request parameter 'cluster_manager_timeout' and deprecate 'master_timeout' - in Index Template APIs #2678 (comment) and Added a new line linter #2875 (comment)

Maybe the kind of tests is not critical, so that it's not failing the build.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
If not to fail the Gradle check task is by design, then keep it as it is. Otherwise, fail the check task when there are test failures in the above situation.

Plugins
none.

Screenshots
none.

Host/Environment (please complete the following information):
n/a

Additional context
none.

[owaiskazi19]

@tlfeng can we close this issue?

[tlfeng]

Hi @owaiskazi19, before closing the issue, I would like to find out the code or the reason that not to fail the Gradle build when test failure occurs in these kinds of tests (YAML REST API test and High-Level-Rest-Client test).

[dreamer-89]

This issue can result in failure to identify changes, which actually break backword compatibility. Thus, adding high-priority label.

[andrross]

This is actually the desired behavior that was added in #2547

The idea is to do limited retries with the same seed. If it passes on a retry then it is a flaky test, but we don't necessarily want to fail the build in such a case.

[dblock]

I think this is working as expected. I'm closing this, reopen if you can show an example where all retries failed and we still reported success.