Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Alerting][Docs] Adds Alerting & Task Manager Scalability Guidance & Health Monitoring #91171

Merged
merged 113 commits into from
Mar 4, 2021
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
c923f15
added troubleshooting to alerts prod considerations
gmmorris Feb 11, 2021
42f3aad
fixed links
gmmorris Feb 11, 2021
bf70427
fixed tm link
gmmorris Feb 11, 2021
c7db172
fixed tm link
gmmorris Feb 15, 2021
be4d424
added tm health api initial docs and linked fro malerting
gmmorris Feb 15, 2021
ab5379c
added tm long running task diagnosis
gmmorris Feb 17, 2021
ecd2f33
fixed headers
gmmorris Feb 17, 2021
61d7256
Merge branch 'master' into task-manager/docs-monitoring
gmmorris Feb 17, 2021
c6549c8
fixed merge conflict
gmmorris Feb 17, 2021
3d92701
split tm docs
gmmorris Feb 17, 2021
cc2f8be
added workload
gmmorris Feb 17, 2021
da1834c
move TM into own section
gmmorris Feb 17, 2021
a3d577e
formating
gmmorris Feb 17, 2021
239384b
remvoetm from alerting
gmmorris Feb 17, 2021
ceca142
added tm section
gmmorris Feb 17, 2021
2bc26aa
tm getting started page
gmmorris Feb 18, 2021
5c94c32
tm prod cons
gmmorris Feb 18, 2021
2f487dc
scaling
gmmorris Feb 18, 2021
385d305
tweaked tm
gmmorris Feb 18, 2021
78cec2f
added scaling
gmmorris Feb 18, 2021
b243248
typo
gmmorris Feb 18, 2021
2d03471
tweaked alerting troiubleshooting
gmmorris Feb 19, 2021
2113afe
fixed grammer and typos
gmmorris Feb 19, 2021
d56f4c9
capitalise
gmmorris Feb 19, 2021
7c990df
removed duplication from developer documentaion
gmmorris Feb 19, 2021
94ffbeb
typos and broken down troubleshooting
gmmorris Feb 19, 2021
fa807e9
Merge branch 'master' into task-manager/docs-monitoring
gmmorris Feb 19, 2021
7245f88
added api docs for tm
gmmorris Feb 19, 2021
979a362
broken troubleshooting down further
gmmorris Feb 19, 2021
c102c9e
added toc to troubleshooting
gmmorris Feb 19, 2021
bd39cab
Update docs/api/task-manager/health.asciidoc
gmmorris Feb 22, 2021
c30d2a8
Update docs/api/task-manager/health.asciidoc
gmmorris Feb 22, 2021
3621bf9
Update docs/developer/plugin-list.asciidoc
gmmorris Feb 22, 2021
da26e51
Merge branch 'master' into task-manager/docs-monitoring
kibanamachine Feb 22, 2021
24ccf37
typo
gmmorris Feb 22, 2021
f3d2e70
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Feb 22, 2021
c037175
plugin api
gmmorris Feb 22, 2021
e8aba63
ensure health monitoring api is always refered to in the same manner
gmmorris Feb 22, 2021
ad4fde6
added notes on delay
gmmorris Feb 23, 2021
759d1a6
Merge branch 'master' into task-manager/docs-monitoring
kibanamachine Feb 24, 2021
5b00ce1
improved scaling strategy step
gmmorris Feb 25, 2021
26bdc8d
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Feb 25, 2021
be72ebd
Update docs/developer/plugin-list.asciidoc
gmmorris Feb 25, 2021
ae46a41
Update docs/settings/task-manager-settings.asciidoc
gmmorris Feb 25, 2021
47d61cb
Update docs/user/alerting/alerting-production-considerations.asciidoc
gmmorris Feb 25, 2021
e2a4de4
Update docs/user/alerting/alerting-production-considerations.asciidoc
gmmorris Feb 25, 2021
43681a7
Update docs/user/alerting/alerting-production-considerations.asciidoc
gmmorris Feb 25, 2021
351e320
Update docs/user/alerting/alerting-production-considerations.asciidoc
gmmorris Feb 25, 2021
6d4112d
Update docs/user/alerting/alerting-production-considerations.asciidoc
gmmorris Feb 25, 2021
3aa5d14
Apply suggestions from code review
gmmorris Feb 25, 2021
36c4fd3
updated troubleshooting intro
gmmorris Feb 25, 2021
226deae
Merge branch 'master' into task-manager/docs-monitoring
gmmorris Feb 25, 2021
86fd368
intorduce top level prod considerations
gmmorris Feb 25, 2021
bb0a88f
fixed links
gmmorris Feb 25, 2021
b15ab3b
avoid refreshing by using useCallback
gmmorris Feb 25, 2021
ec6cf81
plugin docs
gmmorris Feb 25, 2021
8fcd898
moved production into user
gmmorris Feb 25, 2021
3b2a941
fixed topics
gmmorris Feb 25, 2021
e2ff3e7
moved prod after conf sec
gmmorris Feb 25, 2021
dc48758
Update docs/user/production-considerations/alerting-production-consid…
gmmorris Feb 26, 2021
86893f1
Update docs/user/production-considerations/alerting-production-consid…
gmmorris Feb 26, 2021
49fd8dc
Update docs/user/production-considerations/alerting-production-consid…
gmmorris Feb 26, 2021
5747fc1
Update docs/user/production-considerations/alerting-production-consid…
gmmorris Feb 26, 2021
ca154d1
Apply suggestions from code review
gmmorris Feb 26, 2021
f6101fc
grammer
gmmorris Feb 26, 2021
61bd58c
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Feb 26, 2021
e00e6ba
rephrased
gmmorris Feb 26, 2021
b6cc900
include TM in titles
gmmorris Feb 26, 2021
f20d0c2
removed a few links
gmmorris Feb 26, 2021
efda54b
fixed formatting
gmmorris Feb 26, 2021
2a37fdb
move TM articles under TM
gmmorris Feb 26, 2021
576e704
Apply suggestions from code review
gmmorris Feb 26, 2021
e0109af
merge sentances into a single paragraph
gmmorris Feb 26, 2021
7fef967
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Feb 26, 2021
14607b4
fixed broken TM link
gmmorris Feb 26, 2021
202530d
fixed link
gmmorris Feb 26, 2021
957f0cb
Merge branch 'master' into task-manager/docs-monitoring
kibanamachine Feb 26, 2021
f0695bb
removed docs link
gmmorris Feb 26, 2021
b56cc35
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Feb 26, 2021
78fa97e
Merge branch 'master' into task-manager/docs-monitoring
gmmorris Mar 1, 2021
764293c
updated plugin list
gmmorris Mar 1, 2021
ae3e373
fixed TM link
gmmorris Mar 1, 2021
adedebb
fixed TM link again
gmmorris Mar 1, 2021
2d6724a
improved prod con link from alerting
gmmorris Mar 1, 2021
6b21146
Merge branch 'master' into task-manager/docs-monitoring
gmmorris Mar 2, 2021
1626e82
Merge branch 'master' into task-manager/docs-monitoring
kibanamachine Mar 2, 2021
96fd086
Merge branch 'master' into task-manager/docs-monitoring
kibanamachine Mar 2, 2021
e264533
typo
gmmorris Mar 2, 2021
0a85947
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Mar 2, 2021
05a7882
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
33d9147
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
8b93230
Apply suggestions from code review
gmmorris Mar 3, 2021
81dc625
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
b808512
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
531002f
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
301523f
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
22acd67
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
cc3d776
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
6a169fc
Update docs/user/production-considerations/task-manager-troubleshooti…
gmmorris Mar 3, 2021
467ba46
applied PR review comments
gmmorris Mar 3, 2021
a324f66
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Mar 3, 2021
4ea2c9e
Apply suggestions from code review
gmmorris Mar 3, 2021
3ea0a86
applied PR review formatting comments
gmmorris Mar 3, 2021
deb8160
Merge branch 'task-manager/docs-monitoring' of github.com:gmmorris/ki…
gmmorris Mar 3, 2021
b0c15d5
Merge branch 'master' into task-manager/docs-monitoring
gmmorris Mar 3, 2021
f96cebe
boken down to bulltets
gmmorris Mar 3, 2021
92fe188
change question to statement
gmmorris Mar 3, 2021
bde118c
indent troubleshooting
gmmorris Mar 3, 2021
d934c8c
removed unneeded newline
gmmorris Mar 3, 2021
dc7ce41
Apply suggestions from code review
gmmorris Mar 4, 2021
f1b92ef
fixed space
gmmorris Mar 4, 2021
8fe9986
:wq!rge branch 'task-manager/docs-monitoring' of github.com:gmmorris/…
gmmorris Mar 4, 2021
6a3b8b3
Merge branch 'master' into task-manager/docs-monitoring
gmmorris Mar 4, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
<titleabbrev>Alerting</titleabbrev>
++++

{kib} Alerting run both alert checks and actions as persistent background tasks managed by the {kib} Task Manager.
Alerting runs both alert checks and actions as persistent background tasks managed by the Task Manager.

When relying on alerts and actions as mission critical services, make sure you follow the Task Manager <<task-manager-production-considerations>>.
When relying on alerts and actions as mission critical services, make sure you follow the <<task-manager-production-considerations, production considerations>> for Task Manager.

[float]
[[alerting-background-tasks]]
Expand All @@ -30,7 +30,7 @@ This is typically a symptom of the specific usage of the cluster in question.

You can address such issues by tweaking the {kib} <<task-manager,Task Manager>> or scaling the deployment to better suit your use case.

For detailed guidance, see <<alerting-troubleshooting,`Alerting Troubleshooting`>>.
For detailed guidance, see <<alerting-troubleshooting, Alerting Troubleshooting>>.
==============================================

[float]
Expand All @@ -44,7 +44,7 @@ When estimating the required task throughput, keep the following in mind:
* Each alert uses a single recurring task that is scheduled to run at the cadence defined by its <<defining-alerts-general-details, check interval>>.
* Each action uses a single task. However, because <<alerting-concepts-suppressing-duplicate-notifications, actions are taken per instance>>, alerts can generate a large number of non-recurring tasks.

It is difficult to predict how much throughput is needed to ensure all Alerts and Actions are executed at consistent schedules.
It is difficult to predict how much throughput is needed to ensure all alerts and actions are executed at consistent schedules.
By counting Alerts as recurring tasks and Actions as non-recurring tasks, a rough throughput <<task-manager-rough-throughput-estimation,can be estimated>> as a _tasks per minute_ measurment.
gmmorris marked this conversation as resolved.
Show resolved Hide resolved

Predicting the buffer required to account for Actions depends heavily on the Alert Types you use, the amount of Alert Instances they might detect and the number of actions you might choose to assign to action groups when defining your Alerts. With that in mind, we recommend regularly <<task-manager-health-monitoring,monitoring the health>> of your {kib} Task Managers instances.
Predicting the buffer required to account for actions depends heavily on the alert types you use, the amount of alert Instances they might detect, and the number of actions you might choose to assign to action groups. With that in mind, regularly <<task-manager-health-monitoring,monitor the health>> of your Task Manager instances.
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
== Task Manager health monitoring

++++
<titleabbrev>health monitoring</titleabbrev>
<titleabbrev>Health monitoring</titleabbrev>
++++

The {kib} Task Manager has an internal monitoring mechanism keeping track of a variety of metrics which can be consumed via both a health monitoring API, and the {kib} Server Log.
The Task Manager has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {kib} server log.

The health monitoring API is designed to be fast, providing a reliable endpoint which can be monitored.
Consuming this endpoint doesn't cause additional load, but rather it returns the latest health checks made by the system. This design enables consumption by external services at a regular cadance without the concern of additional load to the system.
The health monitoring API provides a reliable endpoint that can be monitored.
Consuming this endpoint doesn't cause additional load, but rather returns the latest health checks made by the system. This design enables consumption by external monitoring services at a regular cadence without additional load to the system.

Each {kib} instance exposes its own endpoint at:

Expand All @@ -23,17 +23,17 @@ Monitoring the `_health` endpoint of each {kib} instance in the cluster is the r

[float]
[[task-manager-configuring-health-monitoring]]
=== Configuring the Monitored Health Statistics
=== Configuring the monitored health statistics

The health monitoring api monitors Task Manager's performance out of the box, but certain performance considerination are deployment specific and are configurable as a result.
The health monitoring API monitors the performance of Task Manager out of the box. However, certain performance considerations are deployment specific and you can configure them.

A health threshold is the threshold for failed task executions. Once a task exceeds this threshold, a status of `warn` or `error` is set on the task type execution. To configure a health threshold, use the <<task-manager-health-settings,`xpack.task_manager.monitored_task_execution_thresholds`>> setting. You can apply this this setting to all task types in the system, or to a custom task type.

By default, this setting is configured to mark the health of every task type as `warning` when it exceeds 80% failed executions, and as `error` at 90%.
This value can be set to any number between 0 to 100, and a threshold is hit when the value *exceeds* this number.
This means that you can avoid setting the status to `error` by setting the threshold at 100, or hit `error` the moment any task fails by setting the threshold to 0 (as it will exceed 0 once a single failure occurs).
By default, this setting marks the health of every task type as `warning` when it exceeds 80% failed executions, and as `error` at 90%.
Set this value to a number between 0 to 100. The threshold is hit when the value *exceeds* this number.
To avoid a status of `error`, set the threshold at 100. To hit `error` the moment any task fails, set the threshold to 0.

Custom configurations allow you to set lower thresholds for task types you might consider critical, such as alerting tasks, which you might wish to detect sooner in an external monitoring service.
Create a custom configuration to set lower thresholds for task types you consider critical, such as alerting tasks that you want to detect sooner in an external monitoring service.

[source,yml]
----
Expand All @@ -46,17 +46,17 @@ xpack.task_manager.monitored_task_execution_thresholds:
error_threshold: 50
warn_threshold: 0
----
<1> default configuration, setting a system wide `warn` threshold at 50%, and `error` at 70% failure rate
<2> custom configuration for the `alerting:.index-threshold` task type, setting a system wide `warn` threshold at 0% (which will set a `warn` status the moment any task of that type fails), and `error` at 50% failure rate
<1> A default configuration that sets the system-wide `warn` threshold at a 50% failure rate, and `error` at 70% failure rate.
<2> A custom configuration for the `alerting:.index-threshold` task type that sets a system wide `warn` threshold at 0% (which sets a `warn` status the moment any task of that type fails), and `error` at a 50% failure rate.

[float]
[[task-manager-consuming-health-stats]]
=== Consuming Health Stats
=== Consuming health stats

The `health` api is best consumed by via the `/api/task_manager/_health` endpoint.
The health API is best consumed by via the `/api/task_manager/_health` endpoint.

Additionally, the metrics are logged out into the {kib} `DEBUG` logger at a regular cadence.
If you wish to enable Task Manager DEBUG logging in your {kib} instance, you will need to add the following to your `Kibana.yml`:
Additionally, the metrics are logged in the {kib} `DEBUG` logger at a regular cadence.
To enable Task Manager DEBUG logging in your {kib} instance, add the following to your `kibana.yml`:
```
gmmorris marked this conversation as resolved.
Show resolved Hide resolved
logging:
loggers:
Expand All @@ -65,13 +65,13 @@ logging:
level: debug
```

Please bear in mind that these stats are logged as often as your <<task-manager-settings,`xpack.task_manager.poll_interval`>> setting, which means it could add substantial noise to your logs.
These stats are logged based the number of milliseconds set in your <<task-manager-settings,`xpack.task_manager.poll_interval`>> setting, which means it could add substantial noise to your logs.

We would recommend only enabling this level of logging temporarily.
Only enable this level of logging temporarily.

[float]
[[making-sense-of-task-manager-health-stats]]
=== Making Sense of Task Manager Health Stats
=== Making sense of Task Manager health stats

The health monitoring API exposes three sections: `configuration`, `workload` and `runtime`:

Expand All @@ -80,20 +80,20 @@ The health monitoring API exposes three sections: `configuration`, `workload` an

a| Configuration

| This section summarizes Task Manager's current configuration, including dynamic configurations which change over time, such as `poll_interval` and `max_workers` which can adjust in reaction to changing load on the system.
| This section summarizes the current configuration of Task Manager. This includes dynamic configurations that change over time, such as `poll_interval` and `max_workers`, which can adjust in reaction to changing load on the system.

a| Workload

| This section summarizes the work load across the cluster, listing the tasks in the system, their types and what their current status is.
| This section summarizes the work load across the cluster, including the tasks in the system, their types, and current status.

a| Runtime

| This section tracks excution performance of Task Manager, tracking task _drift_, worker _load_, and execution stats broken down by type, including duration and execution results
| This section tracks excution performance of Task Manager, tracking task _drift_, worker _load_, and execution stats broken down by type, including duration and execution results.

|===

Each section has a `timestamp` and a `status` which indicates when the last update to this setion took place and whether the health of this section was evaluated as `OK`, `Warning` or `Error`.
Each section has a `timestamp` and a `status` that indicates when the last update to this section took place and whether the health of this section was evaluated as `OK`, `Warning` or `Error`.

The root has its own `status` which indicate the `status` of the system overall.
The root `status` indicates the `status` of the system overall.

By monitoring the `status` of the system overall, and the `status` of specific task types of interest, it is easy to evaluate the health of the {kib} Task Management system.
By monitoring the `status` of the system overall, and the `status` of specific task types of interest, you can evaluate the health of the {kib} Task Management system.