Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding content for the new Notifications plugin #586

Merged
merged 10 commits into from
May 26, 2022
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ collections:
monitoring-plugins:
permalink: /:collection/:path/
output: true
notifications-plugin:
permalink: /:collection/:path/
output: true
clients:
permalink: /:collection/:path/
output: true
Expand Down Expand Up @@ -103,6 +106,9 @@ just_the_docs:
monitoring-plugins:
name: Monitoring plugins
nav_fold: true
notifications-plugin:
name: Notifications plugin
nav_fold: true
clients:
name: Clients and tools
nav_fold: true
Expand Down
20 changes: 18 additions & 2 deletions _im-plugin/ism/policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,7 @@ For information on writing cron expressions, see [Cron expression reference]({{s
## Error notifications

The `error_notification` operation sends you a notification if your managed index fails.
It notifies a single destination with a custom message.
It notifies a single destination or [notification channel]({{site.url}}{{site.baseurl}}/notifications-plugin/index) with a custom message.

Set up error notifications at the policy level:

Expand All @@ -526,7 +526,8 @@ Set up error notifications at the policy level:

Parameter | Description | Type | Required
:--- | :--- |:--- |:--- |
`destination` | The destination URL. | `Slack, Amazon Chime, or webhook URL` | Yes
`destination` | The destination URL. | `Slack, Amazon Chime, or webhook URL` | Yes if `channel` isn't specified
`channel` | A notification channel's ID | `string` | Yes if `destination` isn't specified
`message_template` | The text of the message. You can add variables to your messages using [Mustache templates](https://mustache.github.io/mustache.5.html). | `object` | Yes

The destination system **must** return a response otherwise the `error_notification` operation throws an error.
Expand Down Expand Up @@ -582,6 +583,21 @@ The destination system **must** return a response otherwise the `error_notificat
}
```

#### Example 4: Using a notification channel

```json
{
"error_notification": {
"channel": {
"id": "some-channel-config-id"
},
"message_template": {
"source": "The index {% raw %}{{ctx.index}}{% endraw %} failed during policy execution."
}
}
}
```

You can use the same options for `ctx` variables as the [notification](#notification) operation.

## Sample policy with ISM template for auto rollover
Expand Down
95 changes: 22 additions & 73 deletions _monitoring-plugins/alerting/monitors.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,75 +43,7 @@ Trigger | Conditions that, if met, generate *alerts*.
Tag | A label that can be applied to multiple queries to combine them with the logical OR operation in a per document monitor. You can't use tags with other monitor types.
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved
Alert | An event associated with a trigger. When an alert is created, the trigger performs *actions*, which can include sending a notification.
Action | The information that you want the monitor to send out after being triggered. Actions have a *destination*, a message subject, and a message body.
Destination | A reusable location for an action. Supported locations are Amazon Chime, Email, Slack, or custom webhook.
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved

---

## Create destinations

1. Choose **Alerting**, **Destinations**, **Add destination**.
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved
1. Specify a name for the destination so that you can identify it later.
1. For **Type**, choose Slack, Amazon Chime, custom webhook, or [email](#email-as-a-destination).

For Email, refer to the [Email as a destination](#email-as-a-destination) section below. For all other types, specify the webhook URL. See the documentation for [Slack](https://api.slack.com/incoming-webhooks) and [Amazon Chime](https://docs.aws.amazon.com/chime/latest/ug/webhooks.html) to learn more about webhooks.

If you're using custom webhooks, you must specify more information: parameters and headers. For example, if your endpoint requires basic authentication, you might need to add a header with a key of `Authorization` and a value of `Basic <Base64-encoded-credential-string>`. You might also need to change `Content-Type` to whatever your webhook requires. Popular values are `application/json`, `application/xml`, and `text/plain`.

This information is stored in plain text in the OpenSearch cluster. We will improve this design in the future, but for now, the encoded credentials (which are neither encrypted nor hashed) might be visible to other OpenSearch users.


### Email as a destination

To send or receive an alert notification as an email, choose **Email** as the destination type. Next, add at least one sender and recipient. We recommend adding email groups if you want to notify more than a few people of an alert. You can configure senders and recipients using **Manage senders** and **Manage email groups**.

#### Manage senders

You need to specify an email account from which the Alerting plugin can send notifications.

To configure a sender email, do the following:

1. After you choose **Email** as the destination type, choose **Manage senders**.
1. Choose **Add sender**, **New sender** and enter a unique name.
1. Enter the email address, SMTP host (e.g. `smtp.gmail.com` for a Gmail account), and the port.
1. Choose an encryption method, or use the default value of **None**. However, most email providers require SSL or TLS, which require a username and password in OpenSearch keystore. Refer to [Authenticate sender account](#authenticate-sender-account) to learn more.
1. Choose **Save** to save the configuration and create the sender. You can create a sender even before you add your credentials to the OpenSearch keystore. However, you must [authenticate each sender account](#authenticate-sender-account) before you use the destination to send your alert.

You can reuse senders across many different destinations, but each destination only supports one sender.


#### Manage email groups or recipients

Use email groups to create and manage reusable lists of email addresses. For example, one alert might email the DevOps team, whereas another might email the executive team and the engineering team.

You can enter individual email addresses or an email group in the **Recipients** field.

1. After you choose **Email** as the destination type, choose **Manage email groups**. Then choose **Add email group**, **New email group**.
1. Enter a unique name.
1. For recipient emails, enter any number of email addresses.
1. Choose **Save**.


#### Authenticate sender account

If your email provider requires SSL or TLS, you must authenticate each sender account before you can send an email. Enter these credentials in the OpenSearch keystore using the CLI. Run the following commands (in your OpenSearch directory) to enter your username and password. The `<sender_name>` is the name you entered for **Sender** earlier.

```bash
./bin/opensearch-keystore add plugins.alerting.destination.email.<sender_name>.username
./bin/opensearch-keystore add plugins.alerting.destination.email.<sender_name>.password
```

Note: Keystore settings are node-specific. You must run these commands on each node.
{: .note}

To change or update your credentials (after you've added them to the keystore on every node), call the reload API to automatically update those credentials without restarting OpenSearch:

```json
POST _nodes/reload_secure_settings
{
"secure_settings_password": "1234"
}
```

Channel | A notification channel to use in an action. See [notifications]({{site.url}}{{site.baseurl}}/notifications-plugin/index) for more information.

---

Expand Down Expand Up @@ -347,7 +279,6 @@ Variable | Data Type | Description
:--- | :--- | : ---
`ctx.trigger.actions.id` | String | The action's ID.
`ctx.trigger.actions.name` | String | The action's name.
`ctx.trigger.actions.destination_id`| String | The alert destination's ID.
`ctx.trigger.actions.message_template.source` | String | The message to send in the alert.
`ctx.trigger.actions.message_template.lang` | String | The scripting language used to define the message. Must be Mustache.
`ctx.trigger.actions.throttle_enabled` | Boolean | Whether throttling is enabled for this trigger. See [adding actions](#add-actions) for more information about throttling.
Expand Down Expand Up @@ -376,13 +307,13 @@ Variable | Data Type | Description

## Add actions

The final step in creating a monitor is to add one or more actions. Actions send notifications when trigger conditions are met and support [Slack](https://slack.com/), [Amazon Chime](https://aws.amazon.com/chime/), and webhooks.
The final step in creating a monitor is to add one or more actions. Actions send notifications when trigger conditions are met. See the [Notifications plugin]({{site.url}}{{site.baseurl}}/notifications-plugin/index) to see what communication channels are supported.

If you don't want to receive notifications for alerts, you don't have to add actions to your triggers. Instead, you can periodically check OpenSearch Dashboards.
{: .tip }

1. Specify a name for the action.
1. Choose a destination.
1. Choose a [notification channel]({{site.url}}{{site.baseurl}}/notifications-plugin/index).
keithhc2 marked this conversation as resolved.
Show resolved Hide resolved
1. Add a subject and body for the message.

You can add variables to your messages using [Mustache templates](https://mustache.github.io/mustache.5.html). You have access to `ctx.action.name`, the name of the current action, as well as all [trigger variables](#available-variables).
Expand All @@ -393,7 +324,7 @@ If you don't want to receive notifications for alerts, you don't have to add act
{% raw %}{ "text": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue. - Trigger: {{ctx.trigger.name}} - Severity: {{ctx.trigger.severity}} - Period start: {{ctx.periodStart}} - Period end: {{ctx.periodEnd}}" }{% endraw %}
```

In this case, the message content must conform to the `Content-Type` header in the [custom webhook](#create-destinations).
In this case, the message content must conform to the `Content-Type` header in the [custom webhook]({{site.url}}{{site.baseurl}}/notifcations-plugin/index).
1. If you're using a bucket-level monitor, you can choose whether the monitor should perform an action for each execution or for each alert.

1. (Optional) Use action throttling to limit the number of notifications you receive within a given span of time.
Expand All @@ -418,6 +349,24 @@ After an action sends a message, the content of that message has left the purvie
If you want to use the `ctx.results` variable in a message, use `{% raw %}{{ctx.results.0}}{% endraw %}` rather than `{% raw %}{{ctx.results[0]}}{% endraw %}`. This difference is due to how Mustache handles bracket notation.
{: .note }

### Questions about destinations

Q: What plugins do I need installed besides Alerting?

A: To continue using the notification action in the Alerting plugin, you need to install the backend plugins `notifications-core` and `notifications`. You can also install the Notifications Dashboards plugin to manage Notification channels via OpenSearch Dashboards.

Q: Can I still create destinations?
A: No, destinations have been deprecated and can no longer be created/edited.

Q: Will I need to move my destinations to the Notifications plugin?
A: No. To upgrade users, a background process will automatically move destinations to notification channels. These channels will have the same ID as the destinations, and monitor execution will choose the correct ID, so you don't have to make any changes to the monitor's definition. The migrated destinations will be deleted.

Q: What happens if any destinations fail to migrate?
A: If a destination failed to migrate, the monitor will continue using it until the monitor is migrated to a notification channel. You don't need to do anything in this case.

Q: Do I need to install the Notifications plugins if monitors can still use destinations?
A: Yes. The fallback on destination is to prevent failures in sending messages if migration fails; however, the Notification plugin is what actually sends the message. Not having the Notification plugin installed will lead to the action failing.


---

Expand Down
Loading