[Segment Replication] Prevent store clean up post reader close and refactor

[dreamer-89]

Description

1. Prevents the store clean up via cleanupAndPreserveLatestCommitPoint method on OpenSearchReaderManager close operation. Removes the cleanupTempFiles boolean flag from store cleanup as only files which are dec'ref to 0 are cleaned.
2. Refactors SegmentReplicationTarget finalize replication by moving to store.
3. Refactor Store clean up utilities.

Related Issues

Resolves #8286

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19210/
• CommitID: 0ef1de0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19345/
• CommitID: 1328c59
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19461/
• CommitID: f056610
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19462/
• CommitID: ec4eef8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19469/
• CommitID: fd2a05c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19469/
• CommitID: fd2a05c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Flaky test failure. Tracked in #6287

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.classMethod
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testPressureServiceStats

• URL: https://build.ci.opensearch.org/job/gradle-check/19501/
• CommitID: fd2a05c
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8463 (ae456c0) into main (4657fe7) will decrease coverage by 0.08%.
The diff coverage is 89.28%.

@@             Coverage Diff              @@
##               main    #8463      +/-   ##
============================================
- Coverage     70.89%   70.82%   -0.08%     
+ Complexity    56927    56854      -73     
============================================
  Files          4758     4758              
  Lines        269219   269222       +3     
  Branches      39407    39405       -2     
============================================
- Hits         190876   190687     -189     
- Misses        62268    62449     +181     
- Partials      16075    16086      +11     

Impacted Files	Coverage Δ
...arch/index/engine/NRTReplicationReaderManager.java	90.90% <ø> (ø)
...rc/main/java/org/opensearch/index/store/Store.java	80.83% <85.00%> (+2.74%)	⬆️
.../opensearch/index/engine/NRTReplicationEngine.java	71.79% <100.00%> (-4.90%)	⬇️
...g/opensearch/indices/recovery/MultiFileWriter.java	85.14% <100.00%> (+0.14%)	⬆️
.../indices/replication/SegmentReplicationTarget.java	84.34% <100.00%> (-2.10%)	⬇️

... and 470 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19507/
• CommitID: 2a375eb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[mch2]

lgtm, thanks for cleaning this up.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

• URL: https://build.ci.opensearch.org/job/gradle-check/19508/
• CommitID: 8d0bf23
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/19509/
• CommitID: 55497b6
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/19510/
• CommitID: ae456c0

[dreamer-89]

@mch2 : Is this fix also needed in 2.7 and 2.8 branch ?