Skip to content

Commit

Permalink
add log for corner case
Browse files Browse the repository at this point in the history
Signed-off-by: Yaliang Wu <[email protected]>
  • Loading branch information
ylwu-amzn committed Jan 6, 2023
1 parent edf0732 commit f60641a
Showing 1 changed file with 10 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,16 @@ void cleanUpLocalCache() {
modelState = MLModelState.PARTIALLY_LOADED;
} else {
modelState = MLModelState.LOADED;
if (mlTask.getWorkerNodes().size() < workerNodes.length) {
log
.warn(
"Model loaded on more nodes than target worker nodes. taskId:{}, modelId: {}, workerNodes: {}, targetWorkerNodes: {}",
taskId,
modelId,
Arrays.toString(workerNodes),
Arrays.toString(mlTask.getWorkerNodes().toArray(new String[0]))
);
}
}
log.info("Reset model state as {} for model {}", modelState, modelId);
mlModelManager.updateModel(modelId, ImmutableMap.of(MLModel.MODEL_STATE_FIELD, modelState));
Expand Down

0 comments on commit f60641a

Please sign in to comment.