Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding content for the new Notifications plugin #586

Merged
merged 10 commits into from
May 26, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ collections:
monitoring-plugins:
permalink: /:collection/:path/
output: true
notifications-plugin:
permalink: /:collection/:path/
output: true
clients:
permalink: /:collection/:path/
output: true
Expand Down Expand Up @@ -90,6 +93,9 @@ just_the_docs:
monitoring-plugins:
name: Monitoring plugins
nav_fold: true
notifications-plugin:
name: Notifications plugin
nav_fold: true
clients:
name: Clients and tools
nav_fold: true
Expand Down
20 changes: 18 additions & 2 deletions _im-plugin/ism/policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ For information on writing cron expressions, see [Cron expression reference]({{s
## Error notifications

The `error_notification` operation sends you a notification if your managed index fails.
It notifies a single destination with a custom message.
It notifies a single destination or [notification channel]({{site.url}}{{site.baseurl}}/notifications-plugin/index) with a custom message.

Set up error notifications at the policy level:

Expand All @@ -459,7 +459,8 @@ Set up error notifications at the policy level:

Parameter | Description | Type | Required
:--- | :--- |:--- |:--- |
`destination` | The destination URL. | `Slack, Amazon Chime, or webhook URL` | Yes
`destination` | The destination URL. | `Slack, Amazon Chime, or webhook URL` | Yes if `channel` isn't specified
`channel` | A notification channel's ID | `string` | Yes if `destination` isn't specified
`message_template` | The text of the message. You can add variables to your messages using [Mustache templates](https://mustache.github.io/mustache.5.html). | `object` | Yes

The destination system **must** return a response otherwise the `error_notification` operation throws an error.
Expand Down Expand Up @@ -515,6 +516,21 @@ The destination system **must** return a response otherwise the `error_notificat
}
```

#### Example 4: Using a notification channel

```json
{
"error_notification": {
"channel": {
"id": "some-channel-config-id"
},
"message_template": {
"source": "The index {% raw %}{{ctx.index}}{% endraw %} failed during policy execution."
}
}
}
```

You can use the same options for `ctx` variables as the [notification](#notification) operation.

## Sample policy with ISM template
Expand Down
76 changes: 3 additions & 73 deletions _monitoring-plugins/alerting/monitors.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,76 +23,7 @@ Monitor | A job that runs on a defined schedule and queries OpenSearch indices.
Trigger | Conditions that, if met, generate *alerts*.
Alert | An event associated with a trigger. When an alert is created, the trigger performs *actions*, which can include sending a notification.
Action | The information that you want the monitor to send out after being triggered. Actions have a *destination*, a message subject, and a message body.
Destination | A reusable location for an action. Supported locations are Amazon Chime, Email, Slack, or custom webhook.
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved


---

## Create destinations

1. Choose **Alerting**, **Destinations**, **Add destination**.
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved
1. Specify a name for the destination so that you can identify it later.
1. For **Type**, choose Slack, Amazon Chime, custom webhook, or [email](#email-as-a-destination).

For Email, refer to the [Email as a destination](#email-as-a-destination) section below. For all other types, specify the webhook URL. See the documentation for [Slack](https://api.slack.com/incoming-webhooks) and [Amazon Chime](https://docs.aws.amazon.com/chime/latest/ug/webhooks.html) to learn more about webhooks.

If you're using custom webhooks, you must specify more information: parameters and headers. For example, if your endpoint requires basic authentication, you might need to add a header with a key of `Authorization` and a value of `Basic <Base64-encoded-credential-string>`. You might also need to change `Content-Type` to whatever your webhook requires. Popular values are `application/json`, `application/xml`, and `text/plain`.

This information is stored in plain text in the OpenSearch cluster. We will improve this design in the future, but for now, the encoded credentials (which are neither encrypted nor hashed) might be visible to other OpenSearch users.


### Email as a destination

To send or receive an alert notification as an email, choose **Email** as the destination type. Next, add at least one sender and recipient. We recommend adding email groups if you want to notify more than a few people of an alert. You can configure senders and recipients using **Manage senders** and **Manage email groups**.


#### Manage senders

Senders are email accounts from which the alerting plugin sends notifications.

To configure a sender email, do the following:

1. After you choose **Email** as the destination type, choose **Manage senders**.
1. Choose **Add sender**, **New sender** and enter a unique name.
1. Enter the email address, SMTP host (e.g. `smtp.gmail.com` for a Gmail account), and the port.
1. Choose an encryption method, or use the default value of **None**. However, most email providers require SSL or TLS, which require a username and password in OpenSearch keystore. Refer to [Authenticate sender account](#authenticate-sender-account) to learn more.
1. Choose **Save** to save the configuration and create the sender. You can create a sender even before you add your credentials to the OpenSearch keystore. However, you must [authenticate each sender account](#authenticate-sender-account) before you use the destination to send your alert.

You can reuse senders across many different destinations, but each destination only supports one sender.


#### Manage email groups or recipients

Use email groups to create and manage reusable lists of email addresses. For example, one alert might email the DevOps team, whereas another might email the executive team and the engineering team.

You can enter individual email addresses or an email group in the **Recipients** field.

1. After you choose **Email** as the destination type, choose **Manage email groups**. Then choose **Add email group**, **New email group**.
1. Enter a unique name.
1. For recipient emails, enter any number of email addresses.
1. Choose **Save**.


#### Authenticate sender account

If your email provider requires SSL or TLS, you must authenticate each sender account before you can send an email. Enter these credentials in the OpenSearch keystore using the CLI. Run the following commands (in your OpenSearch directory) to enter your username and password. The `<sender_name>` is the name you entered for **Sender** earlier.

```bash
./bin/opensearch-keystore add plugins.alerting.destination.email.<sender_name>.username
./bin/opensearch-keystore add plugins.alerting.destination.email.<sender_name>.password
```

Note: Keystore settings are node-specific. You must run these commands on each node.
{: .note}

To change or update your credentials (after you've added them to the keystore on every node), call the reload API to automatically update those credentials without restarting OpenSearch:

```json
POST _nodes/reload_secure_settings
{
"secure_settings_password": "1234"
}
```
Channel | A notifications channel to use in an action. See [notifications]({{site.url}}{{site.baseurl}}/notifications-plugin/index) for more information.
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved


---
Expand Down Expand Up @@ -321,7 +252,6 @@ Variable | Data Type | Description
:--- | :--- | : ---
`ctx.trigger.actions.id` | String | The action's ID.
`ctx.trigger.actions.name` | String | The action's name.
`ctx.trigger.actions.destination_id`| String | The alert destination's ID.
`ctx.trigger.actions.message_template.source` | String | The message to send in the alert.
`ctx.trigger.actions.message_template.lang` | String | The scripting language used to define the message. Must be Mustache.
`ctx.trigger.actions.throttle_enabled` | Boolean | Whether throttling is enabled for this trigger. See [adding actions](#add-actions) for more information about throttling.
Expand Down Expand Up @@ -356,7 +286,7 @@ If you don't want to receive notifications for alerts, you don't have to add act
{: .tip }

1. Specify a name for the action.
1. Choose a destination.
1. Choose a [notification channel]({{site.url}}{{site.baseurl}}/notifications-plugin/index).
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved
1. Add a subject and body for the message.

You can add variables to your messages using [Mustache templates](https://mustache.github.io/mustache.5.html). You have access to `ctx.action.name`, the name of the current action, as well as all [trigger variables](#available-variables).
Expand All @@ -367,7 +297,7 @@ If you don't want to receive notifications for alerts, you don't have to add act
{% raw %}{ "text": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue. - Trigger: {{ctx.trigger.name}} - Severity: {{ctx.trigger.severity}} - Period start: {{ctx.periodStart}} - Period end: {{ctx.periodEnd}}" }{% endraw %}
```

In this case, the message content must conform to the `Content-Type` header in the [custom webhook](#create-destinations).
In this case, the message content must conform to the `Content-Type` header in the [custom webhook]({{site.url}}{{site.baseurl}}/notifcations-plugin/index).
1. If you're using a bucket-level monitor, you can choose whether the monitor should perform an action for each execution or for each alert.

1. (Optional) Use action throttling to limit the number of notifications you receive within a given span of time.
Expand Down
Loading