[Backport to main] add ML task timeout setting and clean up expired tasks from cache (#662)

[ylwu-amzn]

Description

Backport PR #662 to main branch

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #770 (38f13ad) into main (c02e437) will decrease coverage by 0.84%.
The diff coverage is 76.89%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main     #770      +/-   ##
============================================
- Coverage     84.78%   83.94%   -0.84%     
- Complexity      988     1012      +24     
============================================
  Files            92       93       +1     
  Lines          3542     3662     +120     
  Branches        326      342      +16     
============================================
+ Hits           3003     3074      +71     
- Misses          402      443      +41     
- Partials        137      145       +8     

Flag	Coverage Δ
ml-commons	83.94% <76.89%> (-0.84%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rg/opensearch/ml/plugin/MachineLearningPlugin.java	100.00% <ø> (ø)
...java/org/opensearch/ml/profile/MLModelProfile.java	46.29% <40.00%> (-0.65%)	⬇️
...earch/ml/action/load/TransportLoadModelAction.java	83.78% <42.85%> (-5.54%)	⬇️
...va/org/opensearch/ml/model/MLModelCacheHelper.java	92.06% <44.44%> (-3.70%)	⬇️
...ain/java/org/opensearch/ml/task/MLTaskManager.java	61.92% <66.66%> (-5.09%)	⬇️
...ain/java/org/opensearch/ml/model/MLModelCache.java	86.36% <71.42%> (-1.78%)	⬇️
...n/java/org/opensearch/ml/model/MLModelManager.java	78.20% <74.75%> (-4.33%)	⬇️
...arch/ml/action/forward/TransportForwardAction.java	90.76% <77.77%> (-0.14%)	⬇️
.../ml/action/syncup/TransportSyncUpOnNodeAction.java	89.79% <81.25%> (-4.15%)	⬇️
...java/org/opensearch/ml/utils/MLExceptionUtils.java	86.36% <86.36%> (ø)
... and 16 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.