Make number of segment metadata files in remote segment store configurable

[sachinpkale]

Description

Current Behavior

• Remote segment store uploads a metadata file with each successful upload of segments post refresh.
• To read any segment data from remote segment store, we need to read metadata file first.
• All the flows that involves copying segments between 2 nodes, like replication, peer recovery are integrated with remote store.
• Following are the steps to cleanup stale segments (that are not live on local) from remote segment store:
  • On flush, we fetch all but latest 10 metadata files, let's call them stale metadata files.
  • Create a list of segments that are referenced by the stale metadata files and not currently live, let's call them stale segment files.
  • Delete stale segment files followed by deletion of stale metadata files.

Issue

• Replication and peer recovery flows are async. When replica (or new primary in case of primary relocation) starts downloading data from remote segment store, it reads the latest metadata file and starts downloading segment files referred by the metadata file.
• To avoid race condition where slow replica is taking a lot of time to download and the segment files became stale at primary (due to segment merges) and deleted from remote segment store, we kept a hard-coded value of 10 to keep last 10 metadata files.
• If the amount of data to be downloaded is high and if the flush (and merge) happens at primary during that time, it is possible that replica will not find a segment file in the remote store and recovery fails with following exception.

IndexShardRecoveryException[Exception while copying segment files from remote segment store]; 
nested: IOException[Error occurred when downloading segments from remote store]; 
nested: NoSuchFileException[Blob object [<Segment Path>/_6yt.cfs__KegSxIsBOrEU3DYZHraf] not found: 
The specified key does not exist]

Solution

• In this PR, we are mitigating the above mentioned issue by making last N metadata files to keep configurable.
• This is not a long term solution to the problem and we need to come up with a lock based solution (used by snapshot interop) but until then this change provides a way to avoid NoSuchFileException.
• We also made a change to peer recovery flow where, if downloading from remote segment store fails, recovery flow will continue with node-node segment copy. This is something that exists even today with remote store integration of peer recovery flow.

Related Issues

• [BUG] NoSuchFileException during peer recovery of a larger shard using remote store #11328

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 1a6cfad

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❌ Gradle check result for d5ac8ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ashking94]

Looks good, but interested in plan for this setting as it will be user facing. I can review once UTs and ITs are added.

[github-actions]

❌ Gradle check result for 71e133c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for cb81e86: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 98884c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 250fdcc: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (5bb6cae) 71.21% compared to head (250fdcc) 71.29%.
Report is 3 commits behind head on main.

❗ Current head 250fdcc differs from pull request most recent head 1a6cfad. Consider uploading reports for the commit 1a6cfad to get more accurate results

Files	Patch %	Lines
...ch/indices/recovery/PeerRecoveryTargetService.java	0.00%	4 Missing ⚠️
...earch/index/store/RemoteSegmentStoreDirectory.java	0.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11329      +/-   ##
============================================
+ Coverage     71.21%   71.29%   +0.08%     
- Complexity    58926    58984      +58     
============================================
  Files          4890     4890              
  Lines        277434   277451      +17     
  Branches      40313    40313              
============================================
+ Hits         197567   197812     +245     
+ Misses        63464    63237     -227     
+ Partials      16403    16402       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❕ Gradle check result for 1a6cfad: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11329-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ec5a0f99f89b82bbbd14cc0ae06ea8ea6ba08667
# Push it to GitHub
git push --set-upstream origin backport/backport-11329-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11329-to-2.x.