Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] The org.opensearch.indices.replication.SegmentReplicationStatsIT.testMultipleIndices is flaky #11454

Closed
reta opened this issue Dec 4, 2023 · 2 comments · Fixed by #12070
Closed

[BUG] The org.opensearch.indices.replication.SegmentReplicationStatsIT.testMultipleIndices is flaky #11454

reta opened this issue Dec 4, 2023 · 2 comments · Fixed by #12070
Assignees
@Poojita-Raj
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run

Comments

@reta
Copy link
Collaborator

reta commented Dec 4, 2023

Describe the bug
Test case org.opensearch.indices.replication.SegmentReplicationStatsIT.testMultipleIndices is flaky:

org.opensearch.indices.replication.SegmentReplicationStatsIT.testMultipleIndices

java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([5589E7C93CF2F754:4CA766F0F22AC433]:0)
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertNotNull(Assert.java:713)
	at org.junit.Assert.assertNotNull(Assert.java:723)
	at org.opensearch.indices.replication.SegmentReplicationStatsIT.testMultipleIndices(SegmentReplicationStatsIT.java:310)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

To Reproduce

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationStatsIT.testMultipleIndices" -Dtests.seed=5589E7C93CF2F754

Expected behavior
The test should always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • CI

Additional context
@reta reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Dec 4, 2023
@reta reta mentioned this issue Dec 4, 2023
Merged
@mch2 mch2 removed their assignment Jan 11, 2024
@Rishikesh1159 Rishikesh1159 mentioned this issue Jan 24, 2024
Merged
8 tasks
@Poojita-Raj
Copy link
Contributor

Doesn't show up in the last 1000 builds:

ruby flaky-test-finder.rb --s 31745 --e 32745 
Will crawl builds from 31745 to 32745
------------------
172 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing (31745,31758,31766,31771,31772,31772,31772,31776,31783,31806,31813,31814,31826,31827,31833,31835,31847,31849,31862,31871,31871,31871,31886,31890,31899,31899,31900,31915,31917,31917,31917,31940,31943,31944,31946,31953,31957,31963,31968,31974,31985,31990,31995,31998,32005,32009,32017,32033,32036,32049,32050,32052,32053,32053,32056,32057,32074,32078,32095,32103,32103,32105,32112,32113,32121,32123,32134,32136,32145,32150,32154,32155,32158,32163,32164,32168,32171,32172,32178,32189,32192,32193,32194,32216,32232,32233,32239,32242,32246,32248,32249,32253,32258,32260,32264,32273,32278,32282,32284,32286,32294,32297,32298,32300,32303,32318,32325,32328,32339,32340,32349,32367,32368,32391,32395,32415,32421,32425,32430,32432,32441,32441,32448,32449,32453,32459,32464,32464,32466,32471,32471,32471,32512,32516,32516,32518,32520,32526,32550,32553,32569,32575,32586,32598,32599,32599,32603,32607,32612,32614,32616,32625,32626,32634,32634,32642,32644,32650,32657,32666,32666,32684,32685,32687,32689,32697,32701,32716,32724,32728,32733,32741)
117 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshAfterCommit (31772,31820,31835,31900,31939,31962,31977,31995,32012,32026,32036,32039,32039,32043,32046,32049,32050,32053,32067,32086,32101,32103,32111,32115,32116,32121,32124,32139,32145,32155,32156,32157,32165,32168,32170,32172,32173,32178,32191,32192,32227,32239,32259,32273,32274,32284,32301,32305,32306,32309,32313,32317,32318,32322,32330,32335,32336,32338,32339,32340,32341,32362,32366,32367,32368,32389,32394,32398,32407,32425,32432,32436,32445,32459,32464,32466,32471,32478,32491,32504,32509,32511,32513,32516,32520,32526,32530,32534,32537,32550,32553,32579,32580,32581,32584,32589,32592,32594,32594,32596,32597,32600,32604,32607,32626,32642,32646,32651,32652,32656,32657,32659,32660,32673,32691,32698,32701)
107 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testAfterRefresh (31939,31995,32012,32026,32036,32039,32046,32049,32050,32053,32067,32086,32101,32103,32111,32115,32116,32121,32124,32139,32145,32155,32156,32157,32165,32168,32170,32172,32173,32191,32192,32227,32239,32240,32273,32274,32284,32301,32305,32309,32313,32317,32318,32322,32330,32335,32336,32338,32339,32340,32341,32362,32366,32367,32368,32389,32394,32398,32407,32425,32432,32433,32436,32445,32459,32464,32466,32471,32478,32491,32509,32511,32513,32516,32520,32526,32530,32534,32537,32550,32553,32579,32580,32581,32584,32589,32592,32594,32596,32597,32600,32604,32607,32626,32642,32646,32651,32651,32652,32656,32657,32659,32660,32673,32691,32698,32701)
107 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testAfterCommit (31939,31942,31995,32012,32026,32036,32039,32046,32049,32050,32053,32067,32086,32101,32103,32111,32115,32116,32121,32124,32139,32145,32155,32156,32157,32165,32168,32170,32172,32173,32191,32192,32227,32239,32240,32273,32274,32284,32301,32305,32309,32313,32317,32318,32322,32330,32334,32335,32336,32338,32339,32340,32341,32362,32366,32367,32368,32389,32394,32398,32407,32425,32432,32436,32445,32459,32464,32466,32471,32478,32491,32509,32511,32513,32516,32520,32526,32530,32534,32537,32550,32553,32579,32580,32581,32584,32589,32592,32594,32596,32597,32600,32604,32607,32626,32642,32646,32651,32652,32656,32657,32659,32660,32673,32691,32698,32701)
104 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testAfterMultipleCommits (31939,31995,32012,32026,32036,32039,32046,32049,32050,32053,32067,32086,32101,32103,32111,32115,32116,32121,32124,32139,32145,32155,32156,32157,32165,32168,32170,32172,32173,32191,32192,32227,32239,32273,32274,32284,32301,32305,32309,32313,32317,32318,32322,32330,32335,32336,32338,32339,32340,32341,32362,32366,32367,32368,32389,32394,32398,32407,32425,32432,32436,32445,32459,32464,32466,32471,32478,32491,32509,32511,32513,32516,32520,32526,32530,32534,32537,32550,32553,32579,32580,32581,32584,32589,32592,32594,32596,32597,32600,32604,32607,32626,32642,32646,32651,32652,32656,32657,32659,32660,32673,32691,32698,32701)
46 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod (31772,31772,31820,31820,31835,31835,31900,31900,31942,31942,31962,31962,31977,31977,32039,32039,32043,32043,32156,32156,32178,32178,32240,32240,32259,32259,32306,32306,32322,32322,32336,32336,32389,32389,32398,32398,32433,32433,32504,32504,32594,32594,32646,32646,32651,32651)
45 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled (31757,31767,31774,31799,31803,31849,31851,31853,31897,31909,31918,31926,31935,31944,32009,32067,32103,32111,32113,32148,32161,32173,32214,32283,32284,32312,32336,32410,32413,32458,32466,32471,32491,32509,32513,32516,32532,32549,32552,32554,32556,32573,32639,32641,32657)
23 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction (32236,32236,32338,32415,32436,32436,32511,32537,32537,32542,32542,32542,32550,32594,32596,32596,32603,32604,32650,32650,32650,32689,32701)
23 org.opensearch.index.shard.RemoteIndexShardTests.testNoFailuresOnFileReads (31939,31961,32046,32050,32067,32086,32157,32172,32284,32338,32340,32471,32537,32584,32592,32594,32600,32604,32651,32652,32660,32660,32691)
14 org.opensearch.search.SearchWeightedRoutingIT.testShardRoutingWithNetworkDisruption_FailOpenEnabled (31917,31917,31945,31994,32026,32278,32312,32349,32366,32399,32479,32572,32650,32689)
13 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards (31803,31913,31933,31953,32013,32124,32145,32203,32389,32575,32606,32611,32625)
11 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation (31821,31919,31998,32103,32210,32246,32428,32459,32530,32645,32683)
9 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs (31925,32191,32231,32317,32427,32435,32509,32582,32642)
9 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled (31935,32112,32112,32239,32319,32319,32319,32560,32579)
8 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.classMethod (31757,31757,32057,32057,32236,32236,32282,32282)
8 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadRangeBlobWithRetries (31832,31918,31960,32086,32413,32432,32432,32607)
6 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards (31819,32049,32086,32313,32478,32607)
5 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseAllShards (31928,32069,32168,32210,32464)
4 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimarySingleReplica (31955,32026,32165,32284)
4 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex (32136,32303,32359,32666)
4 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob (31891,31985,32254,32573)
4 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown (31757,32057,32236,32282)
4 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount (31859,31894,32425,32513)
3 org.opensearch.indices.replication.SegmentReplicationIT.classMethod (31962,32056,32410)
3 org.opensearch.index.reindex.UpdateByQueryBasicTests.testMultipleSources (32297,32297,32422)
3 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation (32335,32560,32657)
2 org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT.testPendingTasksWithClusterNotRecoveredBlock (32121,32453)
2 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringFetchPhase (31757,32078)
2 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness (31758,31813)
2 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testOverriddenBufferInterval (31777,32687)
2 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase (31835,32074)
2 org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureLastSuccessfulSettingsUpdate (31913,32011)
2 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"false"}} (31998,32657)
2 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityThrowsExceptionWhenRestrictSettingTrue (32010,32050)
2 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals} (32011,32619)
2 org.opensearch.search.SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString_FailOpenEnabled (32039,32604)
2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload (32074,32336)
2 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries (32078,32078)
2 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals} (32101,32466)
2 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadNonexistentBlobThrowsNoSuchFileException (32189,32739)
2 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=pit/10_basic/Delete all} (32192,32626)
2 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testReadRange (32248,32415)
2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttempt (32254,32651)
2 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseFromLocalNode (32274,32612)
2 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.classMethod (32398,32398)
2 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore (32471,32666)
2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testTrackerData (32594,32646)
2 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.classMethod (32666,32666)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles (31827)
1 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}} (32139)
1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsOnRemoteStoreRestore (32155)
1 org.opensearch.index.reindex.DeleteByQueryBasicTests.testMultipleSources (32618)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testRestartPrimary_NoReplicas (32491)
1 org.opensearch.index.reindex.ReindexFailureTests.testAbortOnVersionConflict (31796)
1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica (32572)
1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadBlobWithRetries (32254)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testWriteRead (32573)
1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotDeleteRelocatingPrimaryIndex (32259)
1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.classMethod (32259)
1 org.opensearch.snapshots.CloneSnapshotIT.testCloneAfterRepoShallowSettingEnabled (32579)
1 org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainerRetriesTests.testWriteLargeBlob (31767)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testContainerCreationAndDeletion (31945)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testRestartPrimary (31929)
1 org.opensearch.remotestore.CreateRemoteIndexIT.testDefaultRemoteStoreNoUserOverride (31976)
1 org.opensearch.action.admin.indices.create.CreateIndexIT.testCreateAndDeleteIndexConcurrently (31990)
1 org.opensearch.action.admin.indices.create.CreateIndexIT.classMethod (31990)
1 org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites (32309)
1 org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting (31750)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testDeleteBlobs (32339)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats (32349)
1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testRequestDurabilityWhenRestrictSettingImplicitFalse (32050)
1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityWhenRestrictSettingImplicitFalse (32050)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testIndexReopenClose (32065)
1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testRequestDurabilityWhenRestrictSettingExplicitFalse (32367)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication (32398)
1 org.opensearch.cluster.routing.allocation.IndexShardHotSpotTests.testSkewedClusterScaleOut (32731)
1 org.opensearch.snapshots.CloneSnapshotIT.testCloneShallowSnapshotIndex (31820)

@Poojita-Raj
Copy link
Contributor

Both the linked failures are Regressions that show an assertion failure on the response received by the segmentReplicationStatsRequest.

@Poojita-Raj Poojita-Raj mentioned this issue Jan 29, 2024
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Poojita-Raj Poojita-Raj

Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix flaky test SegmentReplicationStatsIT.testMultipleIndices Poojita-Raj/OpenSearch
4 participants
@reta @mch2 @peternied @Poojita-Raj