-
Notifications
You must be signed in to change notification settings - Fork 900
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use systemd-notify for worker heartbeating #20840
Changes from all commits
d1242f2
d159dee
8eb0771
35b2558
91f98a9
01da008
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -27,13 +27,7 @@ def monitor_workers | |
|
||
cleanup_failed_workers | ||
|
||
# Monitor all remaining current worker records | ||
miq_workers.where(:status => MiqWorker::STATUSES_CURRENT_OR_STARTING).each do |worker| | ||
# Push the heartbeat into the database | ||
persist_last_heartbeat(worker) | ||
# Check the worker record for heartbeat timeouts | ||
validate_worker(worker) | ||
end | ||
monitor_active_workers | ||
|
||
do_system_limit_exceeded if self.kill_workers_due_to_resources_exhausted? | ||
end | ||
|
@@ -84,6 +78,22 @@ def cleanup_orphaned_worker_rows | |
end | ||
end | ||
|
||
def monitor_active_workers | ||
# When k8s or systemd is operating as the worker monitor then all of the | ||
# worker monitoring (liveness, memory threshold) is handled by those | ||
# systems. Only when workers are run as standalone processes does MiqServer | ||
# have to monitor the workers itself. | ||
return if podified? || systemd? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So looking at this method it
After this PR neither of these will be necessary neither on k8s (no access to heartbeat files and mem/cpu limits handled by k8s) nor on systemd (systemd-notify for heartbeating and cgroups for mem/cpu limits) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Side note, in a lot of places we ask There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Side note 2, I am seeing a few categories of "worker_monitor_types" that we make decisions on:
I'm sure there are others but maybe we can come up with good names for these concerns. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a good point, I've been assuming that any "external worker monitor" aka k8s or systemd will take care of all or most of the "every 15 seconds make sure the workers are okay" for us but some may provide most but not all. I'm thinking when we move to a more pluggable worker monitor class these won't be questions that have to be asked but rather methods that are implemented or not and should clean things up nicely (:crossed_fingers:) Unsure of readiness vs liveness, is readiness the "are there enough resources to start this new worker" check? Can you expand on "decisions about where a piece of work can run"? Is that "make MiqSever more like k8s and schedule workers across the appliances"? If so yeah that is a big one that I haven't even started to tackle yet, I think this is going to be a fundamental rethink of how we schedule workers (per server vs per region). We "got away with it" on podified because we went to a single There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
👍
liveness == heartbeat check
There are places that have expectations that the server and workers reside on the same process space / filesystem. For example: #20835 The workaround for these has been to make the right process fetch the environmental information for itself instead of doing it for someone else. Generally, it's been places where the assumption was that we needed to run on the miq server. Previously, any worker on that server was fine but doesn't work in pods so it makes sense to make everything queue things for the correct process instead of assuming one process can see other processes. Like this: jrafanie@1c3d8a7 Another example: #20290 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Ahh yes okay I see what you mean, basically anything doing "worker management" outside of MiqServer based on the assumption it can do it based on the PID (processInfo(pid) in the first example and kill(pid) in the second) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Exactly, and i believe we had a similar change for embedded ansible since that relied on the filesystem having the ansible repo checked out. It might have changed since but the idea was that you can't assume the "server with embedded ansible" can access the locally checked out ansible repo, only the process that's guaranteed to have checked it out. I don't know what to call this but the process isolation of process space (kill/processInfo(pid)) and filesystems (ansible repo as an example) are what I'm describing. |
||
|
||
# Monitor all remaining current worker records | ||
miq_workers.where(:status => MiqWorker::STATUSES_CURRENT_OR_STARTING).each do |worker| | ||
# Push the heartbeat into the database | ||
persist_last_heartbeat(worker) | ||
# Check the worker record for heartbeat timeouts | ||
validate_worker(worker) | ||
end | ||
end | ||
|
||
def cleanup_failed_workers | ||
check_not_responding | ||
check_pending_stop | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI, "A pure-Ruby implementation of sd_notify(3) that can be used to communicate state changes of Ruby programs to systemd." (no other dependencies and no c extensions 👏 )