Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Temporarily disable Kibana Rules #122573

Closed
wants to merge 4 commits into from
Closed

[DOC] Temporarily disable Kibana Rules #122573

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fe008ed
[DOC] Temporarily disable Kibana Rules
stefnestor Jan 10, 2022
0f049be
Move section from connectors to troubleshooting
lcawl Jan 18, 2022
e9fc3a5
Add link to settings
lcawl Jan 18, 2022
d91649e
Merge branch 'main' into stefnestor-patch-3
kibanamachine Mar 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/user/alerting/alerting-troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,23 @@ When diagnosing the health state of the task, you will most likely be interested

Investigating the underlying task can help you gauge whether the problem you’re seeing is rooted in the rule not running at all, whether it’s running and failing, or whether it is running, but exhibiting behavior that is different than what was expected (at which point you should focus on the rule itself, rather than the task).

[discrete]
[[alerting-kibana-disable]]
==== Temporarily disable

A cluster may become unresponsive or sluggish if too many or expensive {kib} rules
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not obvious what an "expensive" rule is. I recommend removing that term or else adding information about what makes a particular rule expensive:

Suggested change
A cluster may become unresponsive or sluggish if too many or expensive {kib} rules
A cluster may become unresponsive or sluggish if too many {kib} rules

are attempting to run. As a stop gap measure, you may consider temporarily overriding
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a stop gap measure, you may consider temporarily overriding

As a novice user, it's unclear to me from this text why/when you'd want to perform these steps as opposed to the steps described in https://www.elastic.co/guide/en/kibana/master/create-and-manage-rules.html#controlling-rules. Can we provide more context?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I feel like the framing here is more of a support perspective than that of the user.

Perhaps we can frame it like this (my phrasing is terrible, please do rephrase as needed):

If you are experiencing sluggishness in Kibana and wish to switch your rules off temporarily to ensure they are the cause, you can do so like this.

or something along those lines.

the {kib} Task Manager to gain breathing room to resolve your situation by restarting
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is "breathing room" plain enough? Seems kinda american-ish, but not sure. Maybe something more descriptive would be better anyway? Something like

As a stop gap measure, you may consider temporarily overriding the {kib} Task Manager to have it run fewer tasks less frequently. This should provide some time to make changes to the system to resolve your situation. The relevant {kib} configuration keys are:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together Stef!
I agree, we should definitely avoid figures of speech like that.
Perhaps @gchaps can help us find a good way of expressing this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can take a look. IMO we can make this more concise with something like this:

To temporarily reduce the workload, you can change the Task Manager settings to run fewer tasks less frequently:

{kib} after updating its config to:

[source, txt]
--------------------------------------------------
xpack.task_manager.max_workers: 1
xpack.task_manager.poll_interval: 1m
--------------------------------------------------
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should extend with two things:

  • what this is doing
  • remember to reset it after resolving the problem!

what this is doing

Setting xpack.task_manager.max_workers: 1 will limit each {kib} instance to claiming one task per polling cycle. The default is 10.

Setting xpack.task_manager.poll_interval: 1m sets the {kib} Task Manager polling cycle to 1 minute. The default is 3 seconds (3s)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a duplicate of what we already have in https://www.elastic.co/guide/en/kibana/master/task-manager-settings-kb.html

Perhaps instead we send them to those docs for more details, rather than explaining them here again?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a link to the settings page in e9fc3a5. However, I agree that it's not clear from a glance what these recommended changes accomplish. And I think we need to reiterate that you should revert these changes when your troubleshooting is complete.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Lisa! 👍


For more information about these settings, refer to <<task-manager-settings-kb>>.

In addition to the above methods, broadly used the next approaches and common issues:

* <<alerting-common-issues, Alerting common issues>>
Expand Down