Fix flaky test SegmentReplicationWithRemoteStorePressureIT.testAddReplicaWhileWritesBlocked

[mch2]

Description

This test fails on ensureGreen after adding a replica. This is run inside of the try with resources that blocks operations. The block works by mocking transport calls to prevent segrep from completing until released. This will prevent force-sync recovery of the replica from completing, block recovery, and eventually result in ensureGreen timing out. Fixed by moving the ensureGreen until after releasing blockOperations.

This PR also reduces the doc count that is used while indexing down from max 200. Writes wit h the remote store version of this test take a much longer time to execute whe n performed serially, and we don't need this many docs indexed to create needed checkpoints.

Related Issues

Resolves #8887

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23274/
• CommitID: 67bd594
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot]

Compatibility status:

Checks if related components are compatible with change 5d3633c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23276/
• CommitID: 67bd594
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23274/
• CommitID: 67bd594
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Task :test:fixtures:krb5kdc-fixture:composeBuild FAILED

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23276/
• CommitID: 67bd594
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[org.opensearch.indices.replication.SegmentReplicationSuiteIT.testDropRandomNodeDuringReplication](https://build.ci.opensearch.org/job/gradle-check/23276/testReport/junit/org.opensearch.indices.replication/SegmentReplicationSuiteIT/testDropRandomNodeDuringReplication/)
[org.opensearch.indices.replication.SegmentReplicationSuiteIT.testFullRestartDuringReplication](https://build.ci.opensearch.org/job/gradle-check/23276/testReport/junit/org.opensearch.indices.replication/SegmentReplicationSuiteIT/testFullRestartDuringReplication/)
[org.opensearch.indices.replication.SegmentReplicationSuiteIT.testDeleteIndexWhileReplicating](https://build.ci.opensearch.org/job/gradle-check/23276/testReport/junit/org.opensearch.indices.replication/SegmentReplicationSuiteIT/testDeleteIndexWhileReplicating/)
[org.opensearch.indices.replication.SegmentReplicationSuiteIT.classMethod](https://build.ci.opensearch.org/job/gradle-check/23276/testReport/junit/org.opensearch.indices.replication/SegmentReplicationSuiteIT/classMethod/)

#9499

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/23309/
• CommitID: 67bd594

[codecov]

Codecov Report

Merging #9501 (67bd594) into main (ebdffbb) will decrease coverage by 0.08%.
Report is 5 commits behind head on main.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #9501      +/-   ##
============================================
- Coverage     71.20%   71.12%   -0.08%     
+ Complexity    57474    57466       -8     
============================================
  Files          4776     4776              
  Lines        270815   270815              
  Branches      39584    39584              
============================================
- Hits         192835   192625     -210     
- Misses        61741    61983     +242     
+ Partials      16239    16207      -32     

see 451 files with indirect coverage changes

[tlfeng]

Thanks for the fix with detailed explanation!