Flaky RemoteClustersIT: Add assert busy to avoid race condition

[ankitkala]

Description

Avoid potential race condition in RemoteClustersIT.testHAProxyModeConnectionWorks

Related Issues

#1703

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change bde5021

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29462/
• CommitID: 6045f30
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ankitkala]

Known flaky test: #10154

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT" -Dtests.method="testRequestBreaker {p0={"search.concurrent_segment_search.enabled":"true"}}" -Dtests.seed=52B7BFDB40480C4E -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-Hans-CN -Dtests.timezone=Africa/Douala -Druntime.java=21

org.opensearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT > testRequestBreaker {p0={"search.concurrent_segment_search.enabled":"true"}} FAILED
    java.lang.AssertionError: Request breaker not reset to 0 on node: node_s3
    Expected: <0L>
         but: was <384L>
        at __randomizedtesting.SeedInfo.seed([52B7BFDB40480C4E:EAE5829BAF88DE04]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.test.InternalTestCluster.lambda$ensureEstimatedStats$42(InternalTestCluster.java:2705)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1086)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1059)
        at org.opensearch.test.InternalTestCluster.ensureEstimatedStats(InternalTestCluster.java:2703)
        at org.opensearch.test.TestCluster.assertAfterTest(TestCluster.java:106)
        at org.opensearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2761)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29653/
• CommitID: 781a317
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29695/
• CommitID: 781a317
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ankitkala]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29695/
• CommitID: 781a317
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Flaky test: #10704

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29706/
• CommitID: 50f5750
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/29709/
• CommitID: bde5021

[codecov]

Codecov Report

Merging #11057 (bde5021) into main (0ba5d58) will decrease coverage by 0.53%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##               main   #11057      +/-   ##
============================================
- Coverage     71.67%   71.14%   -0.53%     
+ Complexity    59187    58762     -425     
============================================
  Files          4883     4883              
  Lines        277139   277139              
  Branches      40284    40284              
============================================
- Hits         198643   197182    -1461     
- Misses        62247    63436    +1189     
- Partials      16249    16521     +272     

Files	Coverage Δ
.../opensearch/transport/ProxyConnectionStrategy.java	88.28% <100.00%> (ø)
...org/opensearch/transport/RemoteClusterService.java	86.27% <0.00%> (ø)

... and 474 files with indirect coverage changes