Always create empty translog on replica for remote store enabled index

[sachinpkale]

Description

• In the restore from snapshot flow, primary shards are recovered from snapshot data and replica shards are recovered using peer-recovery flow.
• The changes in this PR address following issue which is applicable only to remote store enabled indices.

Issue

• If we execute following steps in order:
  • Create index
  • Ingest docs
  • Take a snapshot
  • Delete index
  • Created index with the same name
• The new index gets created with different translog UUID. The data in snapshot refers to older translog UUID.
• After recovering primary from snapshot, remote segment store is not updated with new data until the value of primaryMode is true.
• Before primaryMode changed to true, replica recovery is triggered. In peer recovery, we download data from remote segment store.
• As snapshot restored data is yet to be uploaded to remote segment store, replica fetches data that corresponds to the newly created index. So, replica has segments and translog corresponding to new index
• After the files are downloaded to replica, peer recovery checks the diff between primary and replica in terms of segment files and copies the diff. As primary has segments data from snapshot, it copies the diff to replica. At this point, replica has segments from snapshot and translog from new index.
• This fails when we open the engine as translog UUID referred by segments and actual translog files is different.

Solution

• To avoid the above issue, we create empty translog on replica whenever we download data from remote segment store. We use the same translog UUID that is referred by the downloaded segments.
• It is safe to create empty translog for replica of remote store enabled index as we use segment replication and translog is just used for durability purpose.
• Replica only needs to access translog at the time of failover which is already integrated with remote translog.

Related Issues

• [BUG] [Remote Store] TranslogCorruptedException in replica recovery if snapshot is restored to new index #10038

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change a33af59

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled

• URL: https://build.ci.opensearch.org/job/gradle-check/25469/
• CommitID: 8fc73ae
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10012 (a33af59) into main (921cd0c) will increase coverage by 0.67%.
Report is 1 commits behind head on main.
The diff coverage is 78.57%.

@@             Coverage Diff              @@
##               main   #10012      +/-   ##
============================================
+ Coverage     70.42%   71.09%   +0.67%     
- Complexity    57451    58094     +643     
============================================
  Files          4824     4824              
  Lines        273918   273927       +9     
  Branches      39918    39920       +2     
============================================
+ Hits         192894   194761    +1867     
+ Misses        64725    62855    -1870     
- Partials      16299    16311      +12     

Files Changed	Coverage Δ
.../org/opensearch/index/translog/TranslogHeader.java	81.48% <ø> (ø)
...in/java/org/opensearch/index/shard/IndexShard.java	69.40% <78.57%> (+0.35%)	⬆️

... and 585 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25502/
• CommitID: 72ab6ed
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries
      1 org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew

• URL: https://build.ci.opensearch.org/job/gradle-check/25551/
• CommitID: 6b0ab08
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25557/
• CommitID: b0ea4d8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25558/
• CommitID: 4f95968
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

• URL: https://build.ci.opensearch.org/job/gradle-check/25564/
• CommitID: a33af59
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.