False negatives in CI marked as "Flakes" while running PR tests

[Karm]

Describe the bug

Hello,

While working on #44348 it has been bewildering to me that the CI was all green while locally it was broken.
The way the PR was done, it could not have passed, it was missing Brotli library loading, yet the CI was green 🟢

The reason seems to be that Errors are marked as "Flakes", because 2nd run PASSED:

2024-11-09T14:15:53.4363212Z [ERROR]   Run 1: RESTEndpointsIT>RESTEndpointsTest.testDecompressors:120 » UnsatisfiedLink 'java.nio.ByteBuffer com.aayushatharva.brotli4j.encoder.EncoderJNI.nativeCreate(long[])'
2024-11-09T14:15:53.4365271Z [INFO]    Run 2: PASS

Full: https://karms.biz/pastebin/errors-labeled-as-flakes.txt

It seems that the way Re-Run is configured, it recycles the whole context and the second time around, the Brotli4jLoader got called?

It might hide, as this case shows, important issues with integration tests. Can we take a step back and rather fix tests than to enable this re-run feature en mass?

How about we enable re-runs only for a subset of tests that are notorious for race conditions, e.g. some Mongo or DB or messaging scenarios.

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Output of uname -a or ver

Linux amd64

Output of java -version

21

Quarkus version or git rev

main

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[Karm]

@zakkak FYI ^^^

[gsmet]

What I would like to understand is why this test would pass the second time if Brotli is not around at all? That would be interesting to know and could maybe help us improve.

Because sorry to say that but the problem here is not that we rerun the tests, the problem is that we have a failing test passing.