Integration test job succeeds despite of test failures

[nibix]

What is the bug?

Even if PRs show a green CI check mark ("all check have passed"), inspecting the logs of the integration_tests job usually shows that there were test failures:

2024-10-03T13:58:29.2325946Z 
2024-10-03T13:58:29.2327437Z > Task :integrationTest
2024-10-03T13:58:29.2329044Z 
2024-10-03T13:58:29.2330461Z Tests with failures:
2024-10-03T13:58:29.2333698Z  - org.opensearch.security.api.DefaultApiAvailabilityIntegrationTest.securityConfigIsNotAvailableByDefault
2024-10-03T13:58:29.2335895Z  - org.opensearch.security.api.CertificatesRestApiIntegrationTest.timeoutTest
2024-10-03T13:58:29.2338895Z  - org.opensearch.security.api.InternalUsersScoreBasedPasswordRulesRestApiIntegrationTest.canNotCreateUsersWithPassword
2024-10-03T13:58:29.2340664Z 
2024-10-03T13:58:29.2341034Z 493 tests completed, 3 failed
2024-10-03T13:58:29.3312359Z There were failing tests. See the report at: file:///home/runner/work/security/security/build/reports/tests/integrationTest/index.html
2024-10-03T13:58:31.7322529Z 
2024-10-03T13:58:31.7327146Z > Task :jacocoTestReport
2024-10-03T13:58:31.7331759Z 
2024-10-03T13:58:31.7334788Z Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
2024-10-03T13:58:31.7336911Z 
2024-10-03T13:58:31.7339452Z You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
2024-10-03T13:58:31.7342154Z 
2024-10-03T13:58:31.7344713Z For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
2024-10-03T13:58:31.7356207Z 
2024-10-03T13:58:31.7356423Z BUILD SUCCESSFUL in 9m 26s
2024-10-03T13:58:31.7356953Z 6 actionable tasks: 6 executed

(from just a randomly grabbed PR: #4752 ... the same tests also fail for others, so the issue seems to be already in main.)

The logs also show that the tests were tried twice - and both times they failed. So, these are no flakes.

IMO, the job should fail when these tests fail. At the moment, it is unclear to me why the job does not fail.

[nibix]

I think, I have misinterpreted the output - and that the info "493 tests completed, 3 failed - There were failing tests." actually indicates that tests were retried and then became successful.

Closing this for now.