[Backport 2.x] [Segment Replication] Handle failover in mixed cluster mode

[opensearch-trigger-bot]

Backport ff65403 from #9536.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 0dfcf49

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23895/
• CommitID: 0dfcf49
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23920/
• CommitID: 0dfcf49
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23930/
• CommitID: 0dfcf49
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[Poojita-Raj]

2 gradle check runs - flaky: #9624
1 gradle check run - flaky: #9580

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

• URL: https://build.ci.opensearch.org/job/gradle-check/23990/
• CommitID: 0dfcf49
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #9652 (0dfcf49) into 2.x (21da767) will increase coverage by 0.01%.
Report is 5 commits behind head on 2.x.
The diff coverage is 84.00%.

@@             Coverage Diff              @@
##                2.x    #9652      +/-   ##
============================================
+ Coverage     70.88%   70.89%   +0.01%     
- Complexity    57730    57782      +52     
============================================
  Files          4770     4770              
  Lines        272705   272721      +16     
  Branches      40147    40148       +1     
============================================
+ Hits         193303   193343      +40     
+ Misses        62844    62822      -22     
+ Partials      16558    16556       -2     

Files Changed	Coverage Δ
...main/java/org/opensearch/cluster/ClusterState.java	97.84% <ø> (-0.03%)	⬇️
...opensearch/action/get/TransportMultiGetAction.java	46.42% <50.00%> (+2.79%)	⬆️
.../org/opensearch/action/get/TransportGetAction.java	65.30% <83.33%> (+3.08%)	⬆️
...a/org/opensearch/cluster/routing/RoutingNodes.java	85.25% <83.33%> (-0.06%)	⬇️
...java/org/opensearch/cluster/metadata/Metadata.java	87.73% <100.00%> (-0.18%)	⬇️

... and 425 files with indirect coverage changes