Add a setting to control auto release of OpenSearch Monitors managed index creation block

[RS146BIJAY]

Description

OpenSearch controls (applies and remove)create_index block (when all node are breaching high disk watermark) inside DiskThresholdMonitor. Now, if the user tries to apply this block from their end, it may be possible that DiskThresholdMonitor removes it (create_index gets removed if any of the node is not breaching high disk watermark). This makes this block to be usable only by OpenSearch monitors. User cannot control when these blocks get applied or removed.

We would suggest to add a configurable setting which will control when these blocks will be removed. This will ensure that when set to false, both user and OpenSearch monitor applies the respective blocks, but removal of the block is controlled by user.

Issues Resolved

#6269

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationStatsIT.testSegmentReplicationStatsResponse

• URL: https://build.ci.opensearch.org/job/gradle-check/10957/
• CommitID: 517a35d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov-commenter]

Codecov Report

Merging #6277 (517a35d) into main (dfe5d2b) will increase coverage by 0.11%.
The diff coverage is 40.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6277      +/-   ##
============================================
+ Coverage     70.62%   70.73%   +0.11%     
- Complexity    58786    58861      +75     
============================================
  Files          4789     4789              
  Lines        281788   281795       +7     
  Branches      40669    40670       +1     
============================================
+ Hits         199007   199335     +328     
+ Misses        66412    66024     -388     
- Partials      16369    16436      +67     

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	91.89% <ø> (ø)
...uster/routing/allocation/DiskThresholdMonitor.java	75.00% <25.00%> (-6.28%)	⬇️
...ster/routing/allocation/DiskThresholdSettings.java	92.90% <50.00%> (-1.73%)	⬇️
...pensearch/client/cluster/RemoteConnectionInfo.java	0.00% <0.00%> (-68.30%)	⬇️
...rch/test/junit/listeners/ReproduceInfoPrinter.java	9.52% <0.00%> (-68.26%)	⬇️
...a/org/opensearch/client/cluster/SniffModeInfo.java	0.00% <0.00%> (-58.83%)	⬇️
...ava/org/opensearch/action/NoSuchNodeException.java	0.00% <0.00%> (-50.00%)	⬇️
.../opensearch/client/indices/CloseIndexResponse.java	42.50% <0.00%> (-48.75%)	⬇️
...opensearch/persistent/PersistentTasksExecutor.java	22.22% <0.00%> (-44.45%)	⬇️
...nsearch/rest/action/cat/RestCatRecoveryAction.java	43.61% <0.00%> (-42.56%)	⬇️
... and 507 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/20_terms/string profiler via global ordinals}
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/10960/
• CommitID: 06d769d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[gbbafna]

LTGM Rishav.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationStatsIT.testSegmentReplicationStatsResponse

• URL: https://build.ci.opensearch.org/job/gradle-check/10984/
• CommitID: ff479f5
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10986/
• CommitID: 8f3416b

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/10987/
• CommitID: a8957cc
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.