Add model auto redeploy tutorial

[zane-neo]

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ylwu-amzn]

minor : There's -> There are

[zane-neo]

fixed.

[codecov]

Codecov Report

Merging #1175 (86841ea) into 2.x (5b4bd51) will decrease coverage by 0.05%.
Report is 1 commits behind head on 2.x.
The diff coverage is n/a.

❗ Current head 86841ea differs from pull request most recent head 57ffa5d. Consider uploading reports for the commit 57ffa5d to get more accurate results

@@             Coverage Diff              @@
##                2.x    #1175      +/-   ##
============================================
- Coverage     78.94%   78.89%   -0.05%     
+ Complexity     2118     2116       -2     
============================================
  Files           167      167              
  Lines          8633     8633              
  Branches        869      869              
============================================
- Hits           6815     6811       -4     
- Misses         1425     1428       +3     
- Partials        393      394       +1     

Flag	Coverage Δ
ml-commons	78.89% <ø> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

[dhrubo-os]

Can we also provide the value range for these variables?

[dhrubo-os]

How about:

The auto redeployment of models is designed to handle all cases involving node failures, but it does have its limitations. Under the hood, ml-commons uses a cron job to sync up the status of all models in a cluster. This cron job checks all the failed nodes and removes them from the internal model routing table (a mapping between model ID and operational node IDs) to ensure that requests won't be dispatched to crashed nodes.

However, there's one scenario that model auto redeployment cannot handle, which is a complete cluster restart. In this situation, if all nodes are shut down, the last live node's cron job will detect that all the models are not functioning correctly and update the models' status to DEPLOY_FAILED. The model auto redeploy won't check this status because it's not a valid redeployment status. In this case, the user will have to invoke the model deploy/load API.

For cases where only some nodes crash, once the plugins.ml_commons.model_auto_redeploy.enable configuration is set to true, the models will automatically redeploy on those crashed nodes.