[Backport 2.4] Handling OpenSearchRejectExecuteException Exception (#1004)

[sricharanvuppu]

• Handling OpenSearchRejectExecuteException Exception
• introduced writersPerShard setting.

(cherry picked from commit 448e7a7)

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #1019 (5675ccb) into 2.4 (864cdf5) will increase coverage by 1.43%.
Report is 3 commits behind head on 2.4.
The diff coverage is 58.74%.

❗ Current head 5675ccb differs from pull request most recent head 9100099. Consider uploading reports for the commit 9100099 to get more accurate results

@@             Coverage Diff              @@
##                2.4    #1019      +/-   ##
============================================
+ Coverage     73.27%   74.70%   +1.43%     
- Complexity      998     1025      +27     
============================================
  Files           141      141              
  Lines          4628     4724      +96     
  Branches        514      527      +13     
============================================
+ Hits           3391     3529     +138     
+ Misses          918      864      -54     
- Partials        319      331      +12     

Files	Coverage Δ
...in/org/opensearch/replication/ReplicationPlugin.kt	90.60% <100.00%> (+0.19%)	⬆️
.../org/opensearch/replication/ReplicationSettings.kt	100.00% <100.00%> (ø)
...on/resume/TransportResumeIndexReplicationAction.kt	86.95% <100.00%> (ø)
.../replication/repository/RemoteClusterRepository.kt	72.99% <100.00%> (+0.19%)	⬆️
...rch/replication/task/index/IndexReplicationTask.kt	69.14% <100.00%> (ø)
...rch/replication/task/shard/ShardReplicationTask.kt	75.18% <100.00%> (ø)
...org/opensearch/replication/ReplicationException.kt	11.11% <20.00%> (-3.18%)	⬇️
...tlin/org/opensearch/replication/util/Extensions.kt	63.73% <58.33%> (-1.74%)	⬇️
...action/stop/TransportStopIndexReplicationAction.kt	56.12% <40.00%> (-4.64%)	⬇️
...search/replication/task/shard/TranslogSequencer.kt	73.84% <60.97%> (-11.52%)	⬇️
... and 1 more

... and 8 files with indirect coverage changes