-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] RemoteStoreStatsIT.testStatsResponseFromAllNodes is flaky #9828
Comments
Tagging @BhumikaSaini-Amazon |
Thanks @shourya035 I tried to repro this locally: Tests with failures:
- org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseFromAllNodes {seed=[BF41DC8F5BDE48FB:EAF82628D3FF804A]}
- org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseFromAllNodes {seed=[BF41DC8F5BDE48FB:E70285B0347709C0]}
- org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseFromAllNodes {seed=[BF41DC8F5BDE48FB:5EF77F8498B1EB9F]}
200 tests completed, 3 failed
> Task :server:internalClusterTest FAILED The 3/200 failed instances are failing on asserting 0 failed segment uploads: assertEquals(0, stats.totalUploadsFailed); java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([BF41DC8F5BDE48FB:E70285B0347709C0]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.remotestore.RemoteStoreStatsIT.validateSegmentUploadStats(RemoteStoreStatsIT.java:627)
at org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseFromAllNodes(RemoteStoreStatsIT.java:81)
at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) Will check for more iterations and post if I am able to repro with tlog failures. |
I have run this test for around 1K iterations without failure once. Reran again and seeing failure this time. Assertion failure stack trace -
It looks to be happening due to below exception -
|
Describe the bug
RemoteStoreStatsIT.testStatsResponseFromAllNodes is failing
Gradle build: https://build.ci.opensearch.org/job/gradle-check/24535/
Seems like the failure is coming from:
OpenSearch/server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreStatsIT.java
Lines 639 to 641 in 742267d
The text was updated successfully, but these errors were encountered: