[APM]: Replace error occurrence watchers with Kibana Alerting #46547

dgieselaar · 2019-09-25T06:50:48Z

This is a proof of concept that replaces the APM error occurrence Watcher (an Elasticsearch feature) with the new Kibana alerting and actions plugin. The Slack action succesfully fires, but I haven't bothered with the Email action because the approach is pretty similar, and we decided to timebox this to two days.

Here's the gist:

Define an Alert Type for error occurrences. An Alert Type is essentially a function (called an executor) that, given a set of parameters, decides whether (and which) actions need to be triggered. Actions are triggered via an Alert Instance, which captures state of previous executions (for the purpose of this POC, I don't think we need that state).
When the user configures a new alert, we create an Alert object on the server. An Alert is a configuration object for an Alert Type, that tells it at what interval it and with what parameters it should execute.
An Alert also configures the action groups that can be triggered from the Alert Type executor. For this POC, we create Slack and Email actions based on the user's input when configuring an alert, and add them to the default group.
In the executor of the Alert Type, that now runs at the configured interval for the Alert that was created, we run a query for the number of error occurrences and determine whether the threshold was exceeded. If so, we fire the default actions for the Alert, which can be either the Slack or Email action, or both.
To make formatting easier, we pre-process some of the output from the query and pass this as context to the actions that are triggered.

To test this, make sure to explicitly enable both required plugins in your kibana config file:

xpack.actions.enabled: true
xpack.alerting.enabled: true

Notes:

Currently an Alert only supports interval as a scheduling option. That means that we cannot run the executor at a given time each day, which is something that our current implementation does support. I've been told expanding the scheduling options is on the roadmap.
To determine the timeframe for the query we run in the executor, we need the interval parameter. However, it doesn't seem like that's automatically available in the executor, so we pass it as a parameter instead. Maybe there's a nicer way to solve this.
We have to create actions in addition to each Alert that we create (AFAICT), because we allow our users to configure the Slack webhook URL and email addresses in the form. This could present some maintenance issues. It might make more sense to expect the user to create specific Slack/Email actions first, and then provide them with a UI to select those predefined actions. This would limit the amount of maintenance/cleanup of orphaned actions that would be needed in our current implementation.
The Alerting plugin still uses template rendering, but it's a lot easier now that we can pre-process the output from the query with JavaScript, and then pass it to a Mustache renderer. One thing that could me off guard was that Mustache escapes by default, which broke the formatting of the Slack message initially.

Some questions/suggestions for the Alerting/Actions team:

Some of the configuration options for the email and slack actions are now seemingly moved to the secrets param, but this is not reflected in the documentation (https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/actions/README.md)
As explained above, would be nice to have the interval available in the executor as well (if it's not already there and I missed it). Seems like it's a common use case.
I could not figure out what actionGroups is for in the Alert Type. It's seemingly not documented, but it is required.
The typings of registerType could be improved by having typed params (instead of Record<string, any>). Similar to what we did for APM in [APM] migrate to io-ts #42961 (happy to open a PR if welcome).
The kbn-action CLI tool has been super useful, thanks for taking the time to build that.

dgieselaar · 2019-09-25T06:51:30Z

Woops, this was supposed to be a draft PR. I'll update the description in a bit.

elasticmachine · 2019-09-25T07:59:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

dgieselaar · 2019-09-25T10:44:45Z

x-pack/legacy/plugins/apm/common/alerting/constants.ts

+ */
+
+export const ERROR_OCCURRENCE_ALERT_TYPE_ID = 'apm.error_occurrence';
+export const NOTIFICATION_EMAIL_ACTION_ID = 'apm.notificationEmail';


I don't think we need this right now. I was figuring out if we could create the email action we need when registering the alert type, but because I never got around to implementing email it's not being used.

dgieselaar · 2019-09-25T10:45:26Z

x-pack/legacy/plugins/apm/index.ts

+      'elasticsearch',
+      'xpack_main',
+      'apm_oss',
+      'alerting',


I'm not sure if these should be required, or if they are optional. If it's the latter, not sure how we can get access to these plugins on startup.

Really good question for platform how to handle optional dependencies. I think it has come up before but can't remember the answer.

New platform allows for optional dependencies, but legacy platform does not. If you are depending on something in a legacy plugin that could be disabled, you need to make sure that you implement isEnabled and check that the dependency is enabled, otherwise Kibana will crash.

dgieselaar · 2019-09-25T10:46:07Z

...egacy/plugins/apm/public/components/app/ServiceDetails/ServiceIntegrations/WatcherFlyout.tsx

-          .map(email => email.trim())
-          .filter(email => !!email)
-      : [];
+    const email = this.state.actions.email ? this.state.emails : '';


I've simplified this because we can just pass a csv to the action, because it uses nodemailer under the hood. Never tested it though.

dgieselaar · 2019-09-25T10:46:46Z

x-pack/legacy/plugins/apm/server/lib/alerting/error_occurrence/create_alert.ts

+    })
+  );
+
+  return alertsClient.create({


Not sure if we're allowed to do this or if we have to use the API.

This is ok to do, all the business logic currently lives within the client and you're getting the client from the request (which is good). You will just have to handle validation until we have some within the client (if we decided to do so).

dgieselaar · 2019-09-25T10:48:28Z

x-pack/legacy/plugins/apm/server/routes/error_occurrence_alert.ts

+import { createRoute } from './create_route';
+import { createAlert } from '../lib/alerting/error_occurrence/create_alert';
+
+export const errorOccurrenceAlertRoute = createRoute(core => ({


I've created a route for this, because we have to execute several concurrent and sequential requests, and the server is a more robust environment for that kind of dependencies.

mikecote · 2019-09-27T19:16:06Z

...acy/plugins/apm/server/lib/alerting/error_occurrence/register_error_occurrence_alert_type.ts

+        }
+      };
+
+      const { callWithInternalUser } = elasticsearch.getCluster('admin');


You can also use services.callCluster(...). It will be in the context of the user who created the alert (security wise). The approach you have is fine as well if you don't want that.

elasticmachine · 2019-10-02T13:24:58Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-11-06T13:05:33Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: c6b3361

ogupte · 2019-11-13T08:49:31Z

...egacy/plugins/apm/public/components/app/ServiceDetails/ServiceIntegrations/WatcherFlyout.tsx

        this.props.onClose();
+        const id = 'id' in savedObject ? savedObject.id : NOT_AVAILABLE_LABEL;


Maybe I'm outmoded, but isn't the in operator generally discouraged since it's at risk for prototype hijacking?

ogupte · 2019-11-13T08:53:22Z

x-pack/legacy/plugins/apm/server/lib/alerting/error_occurrence/create_alert.ts

+              '<br/>' +
+              '{errorLogMessage}<br/>' +
+              '{errorCulprit}<br/>' +
+              '{docCount} occurrences<br/>',


do want to preserve new lines \n characters in email template like in the slack template?

dgieselaar · 2020-03-19T07:49:47Z

Closing in favor of #59566.

dgieselaar requested a review from a team as a code owner September 25, 2019 06:50

dgieselaar commented Sep 25, 2019

View reviewed changes

This was referenced Sep 26, 2019

[APM] Alerting POC #41032

Closed

[Observability] Alerting POC, metrics threshold alert #46511

Closed

mikecote reviewed Sep 27, 2019

View reviewed changes

dgieselaar force-pushed the alerting-poc branch from f01d96d to 087d6ec Compare October 2, 2019 12:46

Zacqary mentioned this pull request Oct 9, 2019

[Metrics] POC for Alerting on Metric Threshold #47165

Closed

smith added the WIP Work in progress label Oct 22, 2019

[APM]: Replace error occurrence watchers with Kibana Alerting

c6b3361

dgieselaar force-pushed the alerting-poc branch from 087d6ec to c6b3361 Compare November 6, 2019 12:22

ogupte reviewed Nov 13, 2019

View reviewed changes

andrewvc mentioned this pull request Nov 25, 2019

[Uptime] Alerting Implementation #51606

Closed

dgieselaar closed this Mar 19, 2020

dgieselaar deleted the alerting-poc branch March 19, 2020 07:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM]: Replace error occurrence watchers with Kibana Alerting #46547

[APM]: Replace error occurrence watchers with Kibana Alerting #46547

dgieselaar commented Sep 25, 2019 •

edited

Loading

dgieselaar commented Sep 25, 2019

elasticmachine commented Sep 25, 2019

dgieselaar Sep 25, 2019

dgieselaar Sep 25, 2019

sorenlouv Sep 25, 2019

bmcconaghy Sep 25, 2019

dgieselaar Sep 25, 2019

dgieselaar Sep 25, 2019

mikecote Sep 27, 2019

dgieselaar Sep 25, 2019

mikecote Sep 27, 2019

elasticmachine commented Oct 2, 2019

elasticmachine commented Nov 6, 2019

ogupte Nov 13, 2019

ogupte Nov 13, 2019 •

edited

Loading

dgieselaar commented Mar 19, 2020

		this.props.onClose();
		const id = 'id' in savedObject ? savedObject.id : NOT_AVAILABLE_LABEL;

[APM]: Replace error occurrence watchers with Kibana Alerting #46547

[APM]: Replace error occurrence watchers with Kibana Alerting #46547

Conversation

dgieselaar commented Sep 25, 2019 • edited Loading

dgieselaar commented Sep 25, 2019

elasticmachine commented Sep 25, 2019

💔 Build Failed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 2, 2019

💔 Build Failed

elasticmachine commented Nov 6, 2019

💔 Build Failed

Choose a reason for hiding this comment

ogupte Nov 13, 2019 • edited Loading

Choose a reason for hiding this comment

dgieselaar commented Mar 19, 2020

dgieselaar commented Sep 25, 2019 •

edited

Loading

ogupte Nov 13, 2019 •

edited

Loading