[wip] notify users of killing workers when exceed memory #17640
Conversation
| MiqEvent.raise_evm_event_queue(w.miq_server, "evm_worker_memory_exceeded", :event_details => msg, :type => w.class.name) | ||
| notification_options = { | ||
| :name => w.type, | ||
| :memory_usage => usage.try!(:/, 1024*1024), |
There was a problem hiding this comment.
Can we leverage the number_to_human_size method instead?
[2] pry(main)> helper.number_to_human_size(10.megabytes)
=> "10 MB"
[3] pry(main)> helper.number_to_human_size(10.gigabytes)
=> "10 GB"
There was a problem hiding this comment.
@Fryguy was hoping to pass data. only have format for the messages, but wanted computers to be able to understand. I suppose the computer can parse these values
db/fixtures/notification_types.yml
Outdated
| @@ -269,3 +269,8 @@ | |||
| :expires_in: 24.hours | |||
| :level: :error | |||
| :audience: superadmin | |||
| - :name: evm_worker_memory_exceeded | |||
| :message: 'Killing Worker %{name} using %{memory_usage}mb exceeded limit of %{memory_threshold}mb' | |||
There was a problem hiding this comment.
The wording feels funny...perhaps "Killing worker %{name} due to excessive memory usage. %{memory_usage} used memory exceeds limit of %{memory_threshold}." ?
kbrock
force-pushed
the
monitor_notifier_v2
branch
from
2d0b750 to
b5b19d0
Compare
kbrock
changed the title
|
This was deemed too verbose when systems are not properly configured. If you want verbose server exceeded notifications, then add the following to - :name: evm_worker_memory_exceeded
:message: 'Killing worker %{name} due to excessive memory usage. %{memory_usage} used memory exceeds limit of %{memory_threshold}.'
:expires_in: 24.hours
:level: :error
:audience: superadmin |
kbrock
force-pushed
the
monitor_notifier_v2
branch
from
b5b19d0 to
c325336
Compare
|
This pull request is not mergeable. Please rebase and repush. |
- Provide more complete information for worker out of memory errors - Provide notification messages for out of memory errors - Introduce ability to disable notification https://bugzilla.redhat.com/show_bug.cgi?id=1535177
kbrock
force-pushed
the
monitor_notifier_v2
branch
from
c325336 to
f97d013
Compare
|
Checked commit kbrock@f97d013 with ruby 2.3.3, rubocop 0.52.1, haml-lint 0.20.0, and yamllint 1.10.0 |
|
defer to #17673 |
When a worker exceeds the memory threshold, it is killed and restarted.
But it is difficult for users to detect these events and fix the underlying problem.
https://bugzilla.redhat.com/show_bug.cgi?id=1535177
This change adds a notification to bring the problem to the attention of administrators. It includes the worker name and the current memory value to make action easier.
implementation notes:
The event stream record
after_createcallback currently creates the notification record.This lost some necessary data, so I skipped the after create callback and manually create the notification record.