[Segment Replication] Allow segment replication with on disk files not referenced by reader with matching checksum

[dreamer-89]

Description

This change allows segment replication to proceed when local store contains files that are not referenced by active reader. This currently fails with FileAlreadyExistsException because for replica we only consider files referenced by active reader while evaluating the diff in order to request missing files. Since, these files are already copied the attempt to copy again fails with the exception. This change is better as it

1. Prevents duplicate file copy actions.
2. Prevent fatal exception resulting in replication failure.

Caused by: java.util.concurrent.ExecutionException: java.nio.file.FileAlreadyExistsException: /home/ec2-user/opensearch/data/nodes/0/indices/qSnoT1VTSiuipllqbCJfTA/27/index/_i0y.cfe
	at org.opensearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:286) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:260) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:125) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.10.0.jar:2.10.0]

Related Issues

Resolves #9556

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23778/
• CommitID: 80243e7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 80243e7

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Compatibility status:

Checks if related components are compatible with change dd0928d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Compatibility status:

Checks if related components are compatible with change 4354e25

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24030/
• CommitID: e2b3ce5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change e2b3ce5

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24030/
• CommitID: e2b3ce5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

https://build.ci.opensearch.org/job/gradle-check/24030/testReport/org.opensearch.index.shard/RemoteIndexShardTests/testNRTReplicaWithRemoteStorePromotedAsPrimaryRefreshRefresh/ known flaky test failure tracked in #9624

Also, rebased my branch with latest on main where RemoteIndexShardTests tests are retried. This ensures follow up checks do not fail on single failures from this test class.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 8806b8d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24035/
• CommitID: 8806b8d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24035/
• CommitID: 8806b8d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

https://build.ci.opensearch.org/job/gradle-check/24035/testReport/org.opensearch.remotestore/RemoteIndexRecoveryIT/testRerouteRecovery/ tracked in #9580

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24055/
• CommitID: 8806b8d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24064/
• CommitID: 8806b8d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24093/
• CommitID: 8806b8d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/24130/
• CommitID: 8806b8d

[codecov]

Codecov Report

Merging #9630 (8806b8d) into main (d66df10) will decrease coverage by 0.11%.
Report is 8 commits behind head on main.
The diff coverage is 78.80%.

@@             Coverage Diff              @@
##               main    #9630      +/-   ##
============================================
- Coverage     71.16%   71.05%   -0.11%     
- Complexity    57610    57657      +47     
============================================
  Files          4784     4800      +16     
  Lines        271422   272066     +644     
  Branches      39639    39705      +66     
============================================
+ Hits         193146   193306     +160     
- Misses        62062    62522     +460     
- Partials      16214    16238      +24     

Files Changed	Coverage Δ
...java/org/opensearch/common/crypto/DataKeyPair.java	0.00% <0.00%> (ø)
...n/blobstore/VerifyingMultiStreamBlobContainer.java	0.00% <0.00%> (ø)
...ch/repositories/blobstore/BlobStoreRepository.java	61.52% <0.00%> (+0.87%)	⬆️
...search/encryption/keyprovider/CryptoMasterKey.java	22.72% <22.72%> (ø)
.../java/org/opensearch/gateway/GatewayMetaState.java	66.10% <43.39%> (-8.18%)	⬇️
...arch/telemetry/tracing/sampler/RequestSampler.java	66.66% <66.66%> (ø)
.../indices/replication/SegmentReplicationTarget.java	84.87% <68.18%> (-4.13%)	⬇️
...ch/analysis/common/CommonAnalysisModulePlugin.java	92.09% <75.00%> (-0.24%)	⬇️
...arch/common/blobstore/stream/read/ReadContext.java	77.77% <77.77%> (ø)
...arch/gateway/remote/RemoteClusterStateService.java	78.47% <78.47%> (ø)
... and 15 more

... and 444 files with indirect coverage changes