-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Monitoring non-draining node exits #4380
Conversation
@@ -215,6 +224,7 @@ func (n *Nodes) monitorDrainNode(ctx context.Context, nodeID string, index uint6 | |||
msg = Messagef(MonitorMsgLevelInfo, "Node %q has marked all allocations for migration", nodeID) | |||
} else { | |||
msg = Messagef(MonitorMsgLevelInfo, "No drain strategy set for node %s", nodeID) | |||
defer cancel() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why here and not before the if statement?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nickethier So by cancelling the shared context we can cancel the alloc watcher goroutine as well. We only want to do that if we have no expectation of allocations ever changing state because of a drain. That is only true in the else statement because the node has no drain strategy set and it never did. In the other case, allocations can still be transitioning.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yes I get it now, thanks!
command/node_drain_test.go
Outdated
if len(nodes) == 0 { | ||
return false, fmt.Errorf("missing node") | ||
} | ||
nodeID = nodes[0].ID |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this line and 270 can be removed.
api/nodes.go
Outdated
@@ -177,6 +177,14 @@ func (n *Nodes) monitorDrainMultiplex(ctx context.Context, cancel func(), | |||
select { | |||
case outCh <- msg: | |||
case <-ctx.Done(): | |||
|
|||
// If we are exiting but we have a message, attempt to send it | |||
// so we don't loose a message but do not block. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/loose/lose
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
No description provided.