[BUG] org.opensearch.index.reindex.ReindexBasicTests.testCopyMany is flaky

[soosinha]

Describe the bug

org.opensearch.index.reindex.ReindexBasicTests.testCopyMany is flaky
https://build.ci.opensearch.org/job/gradle-check/40471/

./gradlew ':modules:reindex:test' --tests "org.opensearch.index.reindex.ReindexBasicTests.testCopyMany" -Dtests.seed=932270563016D925 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=lv -Dtests.timezone=America/Argentina/San_Luis -Druntime.java=21

Error Message

java.lang.AssertionError:
Expected: updated matches <0L> and created matches <294L> and deleted matches <0L> and batches matches <59> and versionConflicts matches <0L> and failures size matches <0> and reason cancelled matches null and slices matches an empty collection
but: was <BulkByScrollResponse[took=4.5s,timed_out=false,sliceId=null,updated=0,created=90,deleted=0,batches=18,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[{"index":"source","shard":0,"node":"GdWRQ7YdTFeMaVvrCMX9EA","status":500,"reason":{"type":"concurrent_modification_exception","reason":"concurrent_modification_exception: null"}}]]>

Stacktrace

java.lang.AssertionError:
Expected: updated matches <0L> and created matches <294L> and deleted matches <0L> and batches matches <59> and versionConflicts matches <0L> and failures size matches <0> and reason cancelled matches null and slices matches an empty collection
but: was <BulkByScrollResponse[took=4.5s,timed_out=false,sliceId=null,updated=0,created=90,deleted=0,batches=18,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[{"index":"source","shard":0,"node":"GdWRQ7YdTFeMaVvrCMX9EA","status":500,"reason":{"type":"concurrent_modification_exception","reason":"concurrent_modification_exception: null"}}]]>
at __randomizedtesting.SeedInfo.seed([932270563016D925:39E4972BE34FD263]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:964)
at org.junit.Assert.assertThat(Assert.java:930)
at org.opensearch.index.reindex.ReindexBasicTests.testCopyMany(ReindexBasicTests.java:97)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)

Related component

Indexing

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Test should always pass

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[andrross]

[Triage - attendees 1 2 3 4 5]

Thanks for filing.