avoid race condition in syncup model state refresh and handle NP of I… #2405
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Zhangxunmt
requested review from
b4sjoo,
dhrubo-os,
jngz-es,
model-collapse,
rbhavna,
ylwu-amzn,
zane-neo,
austintlee,
HenryL27,
sam-herman and
xinyual
as code owners
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Failure
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Error
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Error
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Error
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Failure
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Error
…sAutoDeployEnabled Signed-off-by: Xun Zhang <[email protected]>
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
ylwu-amzn
reviewed
May 3, 2024
| @@ -97,6 +99,9 @@ public void run() { | |||
| // gather running model/tasks on nodes | |||
| client.execute(MLSyncUpAction.INSTANCE, gatherInfoRequest, ActionListener.wrap(r -> { | |||
| List<MLSyncUpNodeResponse> responses = r.getNodes(); | |||
| if (r.failures() != null && r.failures().size() != 0) { | |||
| log.debug("Received {} failures in the sync up response on nodes", r.failures().size()); | |||
There was a problem hiding this comment.
Seems not very helpful if we don't log the error message
There was a problem hiding this comment.
updated in the refresh.
ylwu-amzn
reviewed
May 3, 2024
| if (!modelsToUndeploy.isEmpty()) { | ||
| // Undeploy expired models | ||
| undeployExpiredModels(modelsToUndeploy, modelWorkerNodes, deployingModels); | ||
| return; |
There was a problem hiding this comment.
Why return here ? If we have 10 models, modelsToUndeploy has 4 models. We should also call refreshModelState for the other 6, right?
There was a problem hiding this comment.
The refreshModelState is called in the listener of unDeployModels after the expired models are un-deployed. Check the undeployExpiredModels(), So the modelstate will be refreshed anyways.
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
ylwu-amzn
reviewed
May 4, 2024
| @@ -97,6 +99,9 @@ public void run() { | |||
| // gather running model/tasks on nodes | |||
| client.execute(MLSyncUpAction.INSTANCE, gatherInfoRequest, ActionListener.wrap(r -> { | |||
| List<MLSyncUpNodeResponse> responses = r.getNodes(); | |||
| if (r.failures() != null && r.failures().size() != 0) { | |||
| log.error("Received {} failures in the sync up response on nodes", r.failures().size()); | |||
There was a problem hiding this comment.
I think you misunderstood me. I mean we should log the error message in r.failures
There was a problem hiding this comment.
Added the messages in the logging.
Signed-off-by: Xun Zhang <[email protected]>
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Error
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Failure
Zhangxunmt
had a problem deploying
to
ml-commons-cicd-env
GitHub Actions
Error
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
ylwu-amzn
approved these changes
May 6, 2024
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
Zhangxunmt
temporarily deployed
to
ml-commons-cicd-env
GitHub Actions
Inactive
jngz-es
approved these changes
May 6, 2024
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
May 6, 2024
#2405) * avoid race condition in syncup model state refresh and handle NP of IsAutoDeployEnabled Signed-off-by: Xun Zhang <[email protected]> * log the error message from syncUp response Signed-off-by: Xun Zhang <[email protected]> * include the syncup response error messages as a string to help debug Signed-off-by: Xun Zhang <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> (cherry picked from commit 21bf079)
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
May 6, 2024
#2405) * avoid race condition in syncup model state refresh and handle NP of IsAutoDeployEnabled Signed-off-by: Xun Zhang <[email protected]> * log the error message from syncUp response Signed-off-by: Xun Zhang <[email protected]> * include the syncup response error messages as a string to help debug Signed-off-by: Xun Zhang <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> (cherry picked from commit 21bf079)
Zhangxunmt
added a commit
that referenced
this pull request
May 6, 2024
#2405) (#2408) * avoid race condition in syncup model state refresh and handle NP of IsAutoDeployEnabled Signed-off-by: Xun Zhang <[email protected]> * log the error message from syncUp response Signed-off-by: Xun Zhang <[email protected]> * include the syncup response error messages as a string to help debug Signed-off-by: Xun Zhang <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> (cherry picked from commit 21bf079) Co-authored-by: Xun Zhang <[email protected]>
Zhangxunmt
added a commit
that referenced
this pull request
May 6, 2024
#2405) (#2409) * avoid race condition in syncup model state refresh and handle NP of IsAutoDeployEnabled Signed-off-by: Xun Zhang <[email protected]> * log the error message from syncUp response Signed-off-by: Xun Zhang <[email protected]> * include the syncup response error messages as a string to help debug Signed-off-by: Xun Zhang <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> (cherry picked from commit 21bf079) Co-authored-by: Xun Zhang <[email protected]>
dhrubo-os
pushed a commit
to dhrubo-os/ml-commons
that referenced
this pull request
May 17, 2024
opensearch-project#2405) (opensearch-project#2408) * avoid race condition in syncup model state refresh and handle NP of IsAutoDeployEnabled Signed-off-by: Xun Zhang <[email protected]> * log the error message from syncUp response Signed-off-by: Xun Zhang <[email protected]> * include the syncup response error messages as a string to help debug Signed-off-by: Xun Zhang <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> (cherry picked from commit 21bf079) Co-authored-by: Xun Zhang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Avoid race condition in sync up job to un-deploy expired models and refresh model states. Without this change, the sync up job sometimes sets wrong Model State after model is un-deployed. Default AutoDeploy to enabled when it's a null value.
Verified and tested in local 2.14 cluster.
Issues Resolved
Related to #2376
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.