[Segment Replication] Wait for shard relocation before building node to shard allocation map

[dreamer-89]

Description

This change fixes the flakyness of recently introduced segment replication bwc test. For doc count assertion, this test first makes API call to index/_stats to build map of shardId to nodeID ([index][1] -> abcdXXX) (separate maps for primary and replica). Once map is built, a targeted search request is made which asserts same doc count for both primary and replica shards. The existing flakyness exists due to shard relocations which completes by the time search query hits the speicfic shard (now moved away to a different node). The fix here is to wait for relocations to complete before verifying doc count.

Related Issues

Resolves #7679

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16697/
• CommitID: f7a8af8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16737/
• CommitID: f7a8af8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/16739/
• CommitID: 431a6ae
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7876 (431a6ae) into main (b15ffc9) will increase coverage by 0.12%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #7876      +/-   ##
============================================
+ Coverage     70.75%   70.88%   +0.12%     
- Complexity    56244    56246       +2     
============================================
  Files          4689     4689              
  Lines        266340   266340              
  Branches      39098    39098              
============================================
+ Hits         188460   188790     +330     
+ Misses        61933    61520     -413     
- Partials      15947    16030      +83     

see 478 files with indirect coverage changes

[Poojita-Raj]

LGTM!