Skip to content

Commit

Permalink
Change to recomanded statement
Browse files Browse the repository at this point in the history
Signed-off-by: zane-neo <[email protected]>
  • Loading branch information
zane-neo committed Aug 2, 2023
1 parent 86841ea commit 57ffa5d
Showing 1 changed file with 22 additions and 16 deletions.
38 changes: 22 additions & 16 deletions docs/tutorials/model_auto_redeploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,28 +12,34 @@ feature is pretty simple.
# Enable model auto redeploy
There are several configurations to control the model auto redeploy feature behavior including:
### plugins.ml_commons.model_auto_redeploy.enable
The default value of this configuration is false, once it's set to true, it means the model auto redeploy feature is enabled.
The default value of this configuration is false, value range is: [true, false], once it's set to true,
it means the model auto redeploy feature is enabled.
### plugins.ml_commons.model_auto_redeploy.lifetime_retry_times
This configuration means how many auto redeploy failure we can tolerate, default value is 3 which means once 3 times of
`failure auto redeploy` reached, the system will not retry auto redeploy anymore. But once auto redeploy is successful,
the retry time will be reset to 0.
This configuration means how many auto redeploy failure we can tolerate, value range is: [0, Integer.MAX_VALUE],
default value is 3 which means once 3 times of `failure auto redeploy` reached, the system will not retry auto
redeploy anymore. But once auto redeploy is successful, the retry time will be reset to 0.
### plugins.ml_commons.model_auto_redeploy_success_ratio
This configuration means how to determine if an auto redeployment is success or not. Since node failure is random, we can't
make sure model auto redeploy can be successful at any time in a cluster, so if most of the expected working nodes have
been successfully redeployed the model, the retry is considered successful. The default value of this is 0.8 which means
if 80% greater or equals 80% nodes successfully redeployed a model, that model's auto redeployment is success.
been successfully redeployed the model, the retry is considered successful. The value range is: [0, 1], and the default
value of this is 0.8 which means if 80% greater or equals 80% nodes successfully redeployed a model, that model's auto
redeployment is success.

# Limitation
Model auto redeploy is to handle all the failure node cases, but it still has limitation. Under the hood, ml-commons use
cron job to sync up all model's status in a cluster, the cron job checks all the failure nodes and remove the failure nodes
in the internal model routing table(this is a mapping between model id and working node ids) to make sure the request won't
be dispatched to crash nodes.
So one case model auto redeploy can't handle is the whole cluster restart, once all nodes are shut down, the last live
node's cron job will detect that all the models are not working correctly and update the model's status to `DEPLOY_FAILED`.
Model auto redeploy won't check this status since this is not a valid redeployment status. In this case, user has to invoke
the [model deploy/load API](https://opensearch.org/docs/latest/ml-commons-plugin/api/#deploying-a-model).
For partial nodes crash case, once the `plugins.ml_commons.model_auto_redeploy.enable` configuration is set to true, the
models will automatically redeploy on those crash nodes.
The auto redeployment of models is designed to handle all cases involving node failures, but it does have its limitations.
Under the hood, ml-commons uses a cron job to sync up the status of all models in a cluster.
This cron job checks all the failed nodes and removes them from the internal model routing
table (a mapping between model ID and operational node IDs) to ensure that requests won't be dispatched to
crashed nodes.

However, there's one scenario that model auto redeployment cannot handle, which is a complete cluster restart.
In this situation, if all nodes are shut down, the last live node's cron job will detect that all the models
are not functioning correctly and update the models' status to DEPLOY_FAILED.
The model auto redeploy won't check this status because it's not a valid redeployment status.
In this case, the user will have to invoke the model deploy/load API.

For cases where only some nodes crash, once the plugins.ml_commons.model_auto_redeploy.enable configuration
is set to true, the models will automatically redeploy on those crashed nodes.

# Example
An example to enable the model auto redeploy feature is via changing the configuration like below:
Expand Down

0 comments on commit 57ffa5d

Please sign in to comment.