Add support for async deletion in S3BlobContainer

[ashking94]

Description

In this PR, we are introducing the deletion using the S3 asyc client for repository-s3. We have also added a setting to control the async deletion in Snapshot code with default being false. We have also added multiple tests around the new code to ensure we are handling the edge cases.

Check List

• Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 7822739: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 639257a: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for 1f5a9cb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for ffc81c0: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 85.92593% with 19 lines in your changes missing coverage. Please review.

Project coverage is 71.88%. Comparing base (3937ccb) to head (5209835).
Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
...ch/repositories/blobstore/BlobStoreRepository.java	57.14%	10 Missing and 2 partials ⚠️
...bstore/AsyncMultiStreamEncryptedBlobContainer.java	0.00%	4 Missing ⚠️
...rg/opensearch/repositories/s3/S3BlobContainer.java	97.01%	1 Missing and 1 partial ⚠️
...rch/repositories/s3/async/S3AsyncDeleteHelper.java	97.14%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15621      +/-   ##
============================================
- Coverage     71.90%   71.88%   -0.02%     
+ Complexity    64392    64333      -59     
============================================
  Files          5278     5281       +3     
  Lines        300877   301004     +127     
  Branches      43478    43486       +8     
============================================
+ Hits         216351   216387      +36     
- Misses        66747    66873     +126     
+ Partials      17779    17744      -35     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❕ Gradle check result for 90548d2: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

✅ Gradle check result for 5013993: SUCCESS

[github-actions]

❌ Gradle check result for 96cd6f7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ashking94]

❌ Gradle check result for 96cd6f7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

> Task :server:compileTestJava
/var/jenkins/workspace/gradle-check/search/server/src/test/java/org/opensearch/cluster/routing/allocation/AllocationCommandsTests.java:751: error: incompatible types: cannot infer type arguments for HashSet<>
            new HashSet<>(
                       ^
    reason: inference variable E has incompatible bounds
      equality constraints: DiscoveryNodeRole
      lower bounds: T#1,E,Object,T#2,DiscoveryNodeRole
  where E,T#1,T#2 are type-variables:
    E extends Object declared in class HashSet
    T#1 extends Object declared in method <T#1>randomSubsetOf(Collection<T#1>)
    T#2 extends Object declared in method <T#2>asList(T#2...)

[github-actions]

❕ Gradle check result for 96cd6f7: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

✅ Gradle check result for 5209835: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15621-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 2b670cc11f52ada701cbf867ce8ad4527d8b6072
# Push it to GitHub
git push --set-upstream origin backport/backport-15621-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15621-to-2.x.