[Segment Replication] Fix flaky testReplicaHasDiffFilesThanPrimary test

[dreamer-89]

Description

Fixes the flakyness of this test by waiting for cluster to process the replica shard failure. Without this wait, doc ingestion & shard cancellation run in parallel. The test also uses existing utilities used for doc count assertion which waits for a longer time i.e. 1 minute. Ran test with fix 100 times without failure.

Issues Resolved

Resolves #6885

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13396/
• CommitID: 5ab7285
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13398/
• CommitID: 30582a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13398/
• CommitID: 30582a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

@kotwanikunal @peterzhuamazon : Is this a known issue ?

> Task :test:fixtures:krb5kdc-fixture:composeBuild
Building peppa
Service 'peppa' failed to build: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13422/
• CommitID: 30582a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/13426/
• CommitID: 4d6b655

[codecov-commenter]

Codecov Report

Merging #6979 (4d6b655) into main (95c6ed9) will decrease coverage by 0.05%.
The diff coverage is 65.50%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6979      +/-   ##
============================================
- Coverage     70.78%   70.73%   -0.05%     
- Complexity    59305    59326      +21     
============================================
  Files          4813     4823      +10     
  Lines        283781   284010     +229     
  Branches      40924    40949      +25     
============================================
+ Hits         200864   200886      +22     
- Misses        66420    66625     +205     
- Partials      16497    16499       +2     

Impacted Files	Coverage Δ
...ch/index/codec/customcodecs/CustomCodecPlugin.java	0.00% <0.00%> (ø)
...h/index/codec/customcodecs/CustomCodecService.java	0.00% <0.00%> (ø)
.../codec/customcodecs/CustomCodecServiceFactory.java	0.00% <0.00%> (ø)
...ustomcodecs/PerFieldMappingPostingFormatCodec.java	0.00% <0.00%> (ø)
...customcodecs/Lucene95CustomStoredFieldsFormat.java	25.00% <25.00%> (ø)
.../index/codec/customcodecs/Lucene95CustomCodec.java	60.00% <60.00%> (ø)
.../codec/customcodecs/ZstdNoDictCompressionMode.java	76.71% <76.71%> (ø)
...opensearch/index/codec/customcodecs/ZstdCodec.java	80.00% <80.00%> (ø)
...arch/index/codec/customcodecs/ZstdNoDictCodec.java	80.00% <80.00%> (ø)
.../index/codec/customcodecs/ZstdCompressionMode.java	84.09% <84.09%> (ø)

... and 480 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.