[BUG] org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicationAfterPrimaryRefreshAndFlush is flaky

[sachinpkale]

Describe the bug

org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicationAfterPrimaryRefreshAndFlush is flaky

  2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.SegmentReplicationRemoteStoreIT" -Dtests.seed=807DF35A5F17422D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-IN -Dtests.timezone=Asia/Kolkata -Druntime.java=19
  2> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=111, name=opensearch[node_t1][flush][T#1], state=RUNNABLE, group=TGRP-SegmentReplicationRemoteStoreIT]

        Caused by:
        java.lang.AssertionError: Expected non-empty readers
            at __randomizedtesting.SeedInfo.seed([807DF35A5F17422D]:0)
            at org.opensearch.index.translog.RemoteFsTranslog.deleteStaleRemotePrimaryTermsAndMetadataFiles(RemoteFsTranslog.java:417)
            at org.opensearch.index.translog.RemoteFsTranslog.trimUnreferencedReaders(RemoteFsTranslog.java:387)
            at org.opensearch.index.translog.InternalTranslogManager.rollTranslogGeneration(InternalTranslogManager.java:89)
            at org.opensearch.index.engine.InternalEngine.flush(InternalEngine.java:1846)
            at org.opensearch.index.shard.IndexShard.flush(IndexShard.java:1388)
            at org.opensearch.action.admin.indices.flush.TransportShardFlushAction.lambda$shardOperationOnPrimary$1(TransportShardFlushAction.java:110)
            at org.opensearch.action.ActionListener.completeWith(ActionListener.java:342)
            at org.opensearch.action.admin.indices.flush.TransportShardFlushAction.shardOperationOnPrimary(TransportShardFlushAction.java:109)
            at org.opensearch.action.admin.indices.flush.TransportShardFlushAction.shardOperationOnPrimary(TransportShardFlushAction.java:63)
            at org.opensearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1260)
            at org.opensearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:159)
            at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:581)
            at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:480)
            at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80)
            at org.opensearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$35(IndexShard.java:3669)
            at org.opensearch.action.ActionListener$3.onResponse(ActionListener.java:130)
            at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:309)
            at org.opensearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:254)
            at org.opensearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:3640)
            at org.opensearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1189)
            at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:477)
            at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
            at org.opensearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:416)
            at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106)
            at org.opensearch.transport.TransportService$8.doRun(TransportService.java:1060)
            at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
            at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            at java.base/java.lang.Thread.run(Thread.java:1589)

To Reproduce

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicationAfterPrimaryRefreshAndFlush" -Dtests.seed=807DF35A5F17422D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-IN -Dtests.timezone=Asia/Kolkata -Druntime.java=19

[sachinpkale]

This should be fixed with recent fixes (#8134). Ran multiple times on local without failing