[Backport 2.x] [Remote Translog] Trimming based on Remote Segment Store (#7383)

[ashking94]

• [Remote Translog] Set min referenced as per remote segment store only when enabled. Add IT for failover using remote translog as well

Description

This backports #7383

Related Issues

#7381
#6195
#5670

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16067/
• CommitID: 4d446ba
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.search.SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString
      1 org.opensearch.index.remote.RemoteRefreshSegmentPressureServiceTests.testValidateSegmentUploadLag

• URL: https://build.ci.opensearch.org/job/gradle-check/16073/
• CommitID: f8042b4
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7699 (f8042b4) into 2.x (b02bfd1) will increase coverage by 0.20%.
The diff coverage is 90.00%.

@@             Coverage Diff              @@
##                2.x    #7699      +/-   ##
============================================
+ Coverage     70.37%   70.57%   +0.20%     
+ Complexity    60159    56370    -3789     
============================================
  Files          4899     4663     -236     
  Lines        289124   267556   -21568     
  Branches      41989    39614    -2375     
============================================
- Hits         203461   188834   -14627     
+ Misses        68605    62445    -6160     
+ Partials      17058    16277     -781     

Impacted Files	Coverage Δ
.../org/opensearch/index/translog/TranslogWriter.java	74.30% <0.00%> (-0.80%)	⬇️
...rg/opensearch/index/translog/RemoteFsTranslog.java	75.00% <100.00%> (+0.30%)	⬆️
...n/java/org/opensearch/index/translog/Translog.java	83.19% <100.00%> (+0.84%)	⬆️
...dex/translog/transfer/TranslogTransferManager.java	82.82% <100.00%> (+0.21%)	⬆️

... and 691 files with indirect coverage changes