[Remote Segments] Retry segment uploads to remote store on failure

[ashking94]

Description

This PR does the following -

• Introduces retry on failed sync segment attempts during remote refresh.
• Uses dedicated threadpool for RemoteStoreRefreshListener.afterRefresh(didRefresh)
• Integ test that introduces random failures during remote store operations and retries to sync the segments from local to remote store.

Related Issues

Resolves #7363

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14824/
• CommitID: ea84f4f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

FYI - One of the other approach is to make changes in scheduleRefresh() method of IndexShard class.

OpenSearch/server/src/main/java/org/opensearch/index/shard/IndexShard.java

Lines 4147 to 4176 in 01e5c29

	public boolean scheduledRefresh() {
	verifyNotClosed();
	boolean listenerNeedsRefresh = refreshListeners.refreshNeeded();
	if (isReadAllowed() && (listenerNeedsRefresh || getEngine().refreshNeeded())) {
	if (listenerNeedsRefresh == false // if we have a listener that is waiting for a refresh we need to force it
	&& isSearchIdle()
	&& indexSettings.isExplicitRefresh() == false
	&& indexSettings.isSegRepEnabled() == false
	// Indices with segrep enabled will never wait on a refresh and ignore shard idle. Primary shards push out new segments only
	// after a refresh, so we don't want to wait for a search to trigger that cycle. Replicas will only refresh after receiving
	// a new set of segments.
	&& active.get()) { // it must be active otherwise we might not free up segment memory once the shard became inactive
	// lets skip this refresh since we are search idle and
	// don't necessarily need to refresh. the next searcher access will register a refreshListener and that will
	// cause the next schedule to refresh.
	final Engine engine = getEngine();
	engine.maybePruneDeletes(); // try to prune the deletes in the engine if we accumulated some
	setRefreshPending(engine);
	return false;
	} else {
	if (logger.isTraceEnabled()) {
	logger.trace("refresh with source [schedule]");
	}
	return getEngine().maybeRefresh("schedule");
	}
	}
	final Engine engine = getEngine();
	engine.maybePruneDeletes(); // try to prune the deletes in the engine if we accumulated some
	return false;
	}

In the first if condition can be changed to
if (isReadAllowed() && (listenerNeedsRefresh || getEngine().refreshNeeded()) || noRemoteSegmentsLag()) {
However, this is relatively more intrusive and will lead to complete refresh process getting triggered although most of it will be no-op. And on failures this will continue to get retried after every index.refresh_interval time which can be catastrophic if the retries induce more failures (throttling due to more retries).

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14869/
• CommitID: 27c2312
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/14876/
• CommitID: 7589820
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov-commenter]

Codecov Report

Merging #7400 (b7be6f2) into main (01e5c29) will decrease coverage by 0.12%.
The diff coverage is 75.53%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #7400      +/-   ##
============================================
- Coverage     70.70%   70.59%   -0.12%     
+ Complexity    59652    59552     -100     
============================================
  Files          4877     4877              
  Lines        285806   285847      +41     
  Branches      41165    41171       +6     
============================================
- Hits         202085   201793     -292     
- Misses        67144    67453     +309     
- Partials      16577    16601      +24     

Impacted Files	Coverage Δ
...opensearch/snapshots/mockstore/MockRepository.java	0.00% <0.00%> (ø)
...search/index/shard/RemoteStoreRefreshListener.java	81.88% <82.92%> (+0.57%)	⬆️
...ain/java/org/opensearch/threadpool/ThreadPool.java	81.89% <100.00%> (-0.72%)	⬇️

... and 503 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15026/
• CommitID: be43780
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting
Looks to be a flaky test, will create an issue for the same if it does not exist.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/15033/
• CommitID: f80f4ac

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15043/
• CommitID: 329de5e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/15044/
• CommitID: 843ea40
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Looks to be unrelated flaky test again.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/15047/
• CommitID: c0ac5dc

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/15089/
• CommitID: b7be6f2

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7400-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 769a0ea449845ff5a462020ed674a917386ad517
# Push it to GitHub
git push --set-upstream origin backport/backport-7400-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7400-to-2.x.