[BUG] o.o.i.s.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew is flaky #10303

dblock · 2023-10-02T15:51:14Z

Describe the bug

#10256 (comment)
https://build.ci.opensearch.org/job/gradle-check/26576/


REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew" -Dtests.seed=714C3129261F73E5 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=und -Dtests.timezone=Etc/GMT-3 -Druntime.java=20

org.opensearch.index.shard.RemoteIndexShardTests > testRepicaCleansUpOldCommitsWhenReceivingNew FAILED
    java.lang.AssertionError: 
    Expected: a collection with size <1>
         but: collection size was <0>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.index.shard.IndexShardTestCase.assertDocCount(IndexShardTestCase.java:1227)
        at org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew(RemoteIndexShardTests.java:307)

    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3503, name=opensearch[org.opensearch.index.shard.RemoteIndexShardTests][generic][T#3], state=RUNNABLE, group=TGRP-RemoteIndexShardTests]

        Caused by:
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([714C3129261F73E5]:0)
            at org.junit.Assert.fail(Assert.java:87)
            at org.junit.Assert.assertTrue(Assert.java:42)
            at org.junit.Assert.assertTrue(Assert.java:53)
            at org.opensearch.index.shard.IndexShardTestCase$4.onReplicationDone(IndexShardTestCase.java:1558)
            at org.opensearch.indices.replication.SegmentReplicationTargetService$SegmentReplicationListener.onDone(SegmentReplicationTargetService.java:464)
            at org.opensearch.indices.replication.common.ReplicationTarget.markAsDone(ReplicationTarget.java:146)
            at org.opensearch.indices.replication.common.ReplicationCollection.markAsDone(ReplicationCollection.java:221)
            at org.opensearch.indices.replication.SegmentReplicationTargetService$3.onResponse(SegmentReplicationTargetService.java:516)
            at org.opensearch.indices.replication.SegmentReplicationTargetService$3.onResponse(SegmentReplicationTargetService.java:512)
            at org.opensearch.indices.replication.SegmentReplicationTarget.lambda$startReplication$3(SegmentReplicationTarget.java:179)
            at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
            at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:126)
            at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
            at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341)
            at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120)
            at org.opensearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:82)
            at org.opensearch.action.StepListener.whenComplete(StepListener.java:95)
            at org.opensearch.indices.replication.SegmentReplicationTarget.startReplication(SegmentReplicationTarget.java:177)
            at org.opensearch.indices.replication.SegmentReplicationTargetService.start(SegmentReplicationTargetService.java:512)
            at org.opensearch.indices.replication.SegmentReplicationTargetService$ReplicationRunner.doRun(SegmentReplicationTargetService.java:498)
            at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
            at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
            at java.****/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
            at java.****/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            at java.****/java.lang.Thread.run(Thread.java:1623)

The text was updated successfully, but these errors were encountered:

ashking94 · 2023-10-26T03:40:11Z

#10939 (comment)

peternied · 2023-11-02T23:24:36Z

Impacted #11068, logs.

reta · 2024-01-05T23:00:43Z

The issue is no fixed: https://build.ci.opensearch.org/job/gradle-check/31797/testReport/junit/org.opensearch.index.shard/RemoteIndexShardTests/testRepicaCleansUpOldCommitsWhenReceivingNew/

org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew

java.lang.AssertionError: Replication should complete successfully expected:<0> but was:<1>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.opensearch.index.shard.IndexShardTestCase.replicateSegments(IndexShardTestCase.java:1669)
	at org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew(RemoteIndexShardTests.java:312)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.lang.AssertionError: 
Expected: <[doc{id='1 seqNo=0 primaryTerm=45 version=1 source= {}}]>
     but: was <[]>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:964)
		at org.junit.Assert.assertThat(Assert.java:930)
		at org.opensearch.index.replication.OpenSearchIndexLevelReplicationTestCase$ReplicationGroup.close(OpenSearchIndexLevelReplicationTestCase.java:607)
		at org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew(RemoteIndexShardTests.java:304)
		... 38 more

BhumikaSaini-Amazon · 2024-04-01T10:43:45Z

...

WARNING: System::setSecurityManager will be removed in a future r│
elease                                                           │
                                                                 │
BUILD SUCCESSFUL in 22s                                          │
53 actionable tasks: 1 executed, 52 up-to-date                   │
=================================================================│
==                                                               │
23530                                                            │
=================================================================│
==                                                               │
=======================================                          │
OpenSearch Build Hamster says Hello!                             │
  Gradle Version        : 8.7                                    │
  OS Info               : <redacted>                             │
  JDK Version           : 11 (Amazon Corretto JDK)               │
  JAVA_HOME             : /usr/lib/jvm/java-11-amazon-corretto   │
  Random Testing Seed   : 9C9CDE0BBD909DF0                       │
  In FIPS 140 mode      : false                                  │
=======================================                          │
                                                                 │
> Task :server:test                                              │
Apr 01, 2024 10:39:28 AM sun.util.locale.provider.LocaleProviderA│
dapter <clinit>                                                  │
WARNING: COMPAT locale provider will be removed in a future relea│
se

...

Unable to repro this on the latest main branch even after 23k+ unique master seeds. I don't see any recent reports for this issue either.

I will check if this could be somehow related to concurrent runs in Jenkins.

BhumikaSaini-Amazon · 2024-04-05T05:20:18Z

WARNING: Please consider reporting this to the maintainers of org│
.gradle.api.internal.tasks.testing.worker.TestWorker             │
WARNING: System::setSecurityManager will be removed in a future r│
elease                                                           │
                                                                 │
BUILD SUCCESSFUL in 14s                                          │
53 actionable tasks: 1 executed, 52 up-to-date                   │
=================================================================│
==                                                               │
45825                                                            │
=================================================================│
==                                                               │
=======================================                          │
OpenSearch Build Hamster says Hello!                             │
  Gradle Version        : 8.7                                    │
  OS Info               : Linux 5.10.210-

No repro in 45k+ iters. Nothing unusual in Jenkins.

chishui · 2024-07-25T09:58:07Z

Still see this issue happening:
https://build.ci.opensearch.org/job/gradle-check/42426/testReport/junit/org.opensearch.index.shard/RemoteIndexShardTests/testRepicaCleansUpOldCommitsWhenReceivingNew/

java.lang.AssertionError: Replication should complete successfully expected:<0> but was:<1>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.opensearch.index.shard.IndexShardTestCase.replicateSegments(IndexShardTestCase.java:1702)
	at org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew(RemoteIndexShardTests.java:313)

linuxpi · 2024-07-25T15:07:12Z

[Storage Triage - attendees 1 2 3 4 5 6 7 8]

Thanks for filing this issue, please feel free to raise PR to add a fix!

dblock added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Oct 2, 2023

dblock mentioned this issue Oct 2, 2023

[Meta] Segment Replication flaky test failures #8279

Closed

16 tasks

psychbot mentioned this issue Oct 3, 2023

[Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository #9839

Merged

6 tasks

Rishikesh1159 mentioned this issue Oct 17, 2023

[Segment Replication] Unmute Test SegmentReplicationTargetServiceTests.testShardAlreadyReplicating #10660

Merged

7 tasks

RamakrishnaChilaka assigned mch2 Oct 24, 2023

reta mentioned this issue Oct 31, 2023

[Backport] [2.x] Update bundled JDK to JDK-21.0.1 (#10576) #11003

Merged

peternied mentioned this issue Nov 2, 2023

Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents #11068

Merged

8 tasks

mch2 mentioned this issue Nov 6, 2023

Fix flaky test RemoteIndexShardtests.testRepicaCleansUpOldCommitsWhenReceivingNew #11111

Merged

8 tasks

dreamer-89 closed this as completed in #11111 Nov 6, 2023

AlexRuiz7 mentioned this issue Nov 17, 2023

Compatibility with OpenSearch 2.11.0 wazuh/wazuh-dashboard#113

Closed

4 tasks

reta reopened this Jan 5, 2024

reta mentioned this issue Jan 5, 2024

[AUTOCUT] Gradle Check Failure on push to main #11783

Closed

anasalkouz removed the untriaged label Jan 11, 2024

mch2 removed their assignment Jan 11, 2024

peternied mentioned this issue Feb 9, 2024

[AUTOCUT] Gradle Check Failure on push to main #12266

Closed

andrross added the Storage:Remote label Feb 21, 2024

github-project-automation bot added this to Storage Project Board Feb 21, 2024

github-project-automation bot moved this to 🆕 New in Storage Project Board Feb 21, 2024

BhumikaSaini-Amazon self-assigned this Mar 18, 2024

BhumikaSaini-Amazon closed this as not planned Won't fix, can't repro, duplicate, stale Apr 5, 2024

github-project-automation bot moved this from 🆕 New to ✅ Done in Storage Project Board Apr 5, 2024

peternied reopened this Jul 25, 2024

github-project-automation bot moved this from ✅ Done to 🏗 In progress in Storage Project Board Jul 25, 2024

github-actions bot added the untriaged label Jul 25, 2024

linuxpi removed the untriaged label Jul 25, 2024

linuxpi moved this from 🏗 In progress to 🆕 New in Storage Project Board Jul 25, 2024

linuxpi moved this from 🆕 New to Ready To Be Picked in Storage Project Board Jul 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] o.o.i.s.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew is flaky #10303

[BUG] o.o.i.s.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew is flaky #10303

dblock commented Oct 2, 2023

ashking94 commented Oct 26, 2023

peternied commented Nov 2, 2023

reta commented Jan 5, 2024 •

edited

Loading

BhumikaSaini-Amazon commented Apr 1, 2024

BhumikaSaini-Amazon commented Apr 5, 2024

chishui commented Jul 25, 2024

linuxpi commented Jul 25, 2024

[BUG] o.o.i.s.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew is flaky #10303

[BUG] o.o.i.s.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew is flaky #10303

Comments

dblock commented Oct 2, 2023

ashking94 commented Oct 26, 2023

peternied commented Nov 2, 2023

reta commented Jan 5, 2024 • edited Loading

BhumikaSaini-Amazon commented Apr 1, 2024

BhumikaSaini-Amazon commented Apr 5, 2024

chishui commented Jul 25, 2024

linuxpi commented Jul 25, 2024

reta commented Jan 5, 2024 •

edited

Loading