[BUG] Release 1.3.9 OS integ-test fail

[jordarlu]

What is the bug?
Release 1.3.9 OS integ-test fail

How can one reproduce the bug?

OS - tar is fail at index-management https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/3920/pipeline/91/

Suite: Test class org.opensearch.indexmanagement.rollup.runner.RollupRunnerIT
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.indexmanagement.rollup.runner.RollupRunnerIT.test continuous delay exclusion period" -Dtests.seed=B71EFE91ACEC5211 -Dtests.security.manager=false -Dtests.locale=el -Dtests.timezone=America/Lima -Druntime.java=11
  2> java.lang.AssertionError: Did not spend time searching
        at __randomizedtesting.SeedInfo.seed([B71EFE91ACEC5211:5FA8DA69A6D40082]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.indexmanagement.rollup.runner.RollupRunnerIT.test continuous delay exclusion period(RollupRunnerIT.kt:728)
  2> NOTE: leaving temporary files on disk at: /tmp/tmps512ievd/index-management/build/testrun/integTest/temp/org.opensearch.indexmanagement.rollup.runner.RollupRunnerIT_B71EFE91ACEC5211-001
  2> NOTE: test params are: codec=Asserting(Lucene87): {}, docValues:{}, maxPointsInLeafNode=1011, maxMBSortInHeap=7.7537671421221415, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=el, timezone=America/Lima
  2> NOTE: Linux 5.10.135-122.509.amzn2.x86_64 amd64/Eclipse Adoptium 11.0.15 (64-bit)/cpus=16,threads=1,free=302570352,total=536870912
  2> NOTE: All tests run in this JVM: [IndexManagementIndicesIT, ActionRetryIT, ActionTimeoutIT, AllocationActionIT, CloseActionIT, DeleteActionIT, ForceMergeActionIT, IndexPriorityActionIT, IndexStateManagementHistoryIT, OpenActionIT, ReadOnlyActionIT, ReadWriteActionIT, ReplicaCountActionIT, RolloverActionIT, RollupActionIT, SnapshotActionIT, TransitionActionIT, ManagedIndexCoordinatorIT, MigrationServicesIT, ISMTemplateRestAPIIT, IndexStateManagementRestApiIT, RestAddPolicyActionIT, RestChangePolicyActionIT, RestExplainActionIT, RestRemovePolicyActionIT, RestRetryFailedManagedIndexActionIT, ManagedIndexRunnerIT, RefreshSearchAnalyzerActionIT, RestRefreshSearchAnalyzerActionIT, FieldCapsFilterIT, RollupInterceptorIT, RestDeleteRollupActionIT, RestExplainRollupActionIT, RestGetRollupActionIT, RestIndexRollupActionIT, RestStartRollupActionIT, RestStopRollupActionIT, RollupRunnerIT]

248 tests completed, 1 failed, 6 skipped

OS - rpm is also fail at index-management https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/3921/pipeline/91

Suite: Test class org.opensearch.indexmanagement.transform.TransformRunnerIT
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.indexmanagement.transform.TransformRunnerIT.test continuous transform with wildcard indices" -Dtests.seed=292AEFD5A06F461B -Dtests.security.manager=false -Dtests.locale=en-MT -Dtests.timezone=Etc/GMT+9 -Druntime.java=11
  2> java.lang.AssertionError: Not the expected pages processed expected:<6> but was:<2>
        at __randomizedtesting.SeedInfo.seed([292AEFD5A06F461B:C34C214215BFDFFC]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.opensearch.indexmanagement.transform.TransformRunnerIT.test continuous transform with wildcard indices(TransformRunnerIT.kt:743)
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.indexmanagement.transform.TransformRunnerIT.test continuous transform picks up new documents" -Dtests.seed=292AEFD5A06F461B -Dtests.security.manager=false -Dtests.locale=en-MT -Dtests.timezone=Etc/GMT+9 -Druntime.java=11
  2> java.lang.AssertionError: Transform did not complete iteration or had incorrect number of documents processed expected:<5000> but was:<10000>
        at __randomizedtesting.SeedInfo.seed([292AEFD5A06F461B:2215D946B51DA861]:0)
...........
248 tests completed, 8 failed, 6 skipped

looks like there was a similar issue at here -> #90

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[jordarlu]

opensearch-project/opensearch-build#3195

[bowenlan-amzn]

Know flaky tests