Improve test quality by either improving or removing unreliable test

[Jeffwan]

This actually looks really bad. We don't even know it's flaky tests or test failures.

1. Could maintainers make sure tests pass before merging the PR?
2. Could maintainers rerun the test for post-submit commits?
3. is there a way to open rerun permission for PR author?

[kevin85421]

Every PR will pass all tests before merging. There are a lot of questions (e.g. #618 #634 #638) in the existing compatibility tests. Some of them are fixed, e.g. #620 and #621. We still need more time to fix all of them.

Is there a way to open rerun permission for PR author?

This is a good idea. We can discuss this in the next community sync.

[Jeffwan]

For short term, I think it's better to rerun the tests for merged commits before the compatibility get fixed

[Jeffwan]

[DmitriGekhtman]

For short term, I think it's better to rerun the tests for merged commits before the compatibility get fixed

Makes sense!

Some other potential actions we considered:
Isolate flaky tests.
Move Fault Tolerance tests to a nightly pipeline.
Consider modifying test timeouts.