[DOC] Add new cluster metrics monitor supported APIs (#4525)

*Add new cluster metrics monitors APIs

---------

Signed-off-by: Melissa Vagi <[email protected]>

_observing-your-data/alerting/actions.md

@@ -0,0 +1,64 @@
+---
+layout: default
+title: Actions
+nav_order: 15
+grand_parent: Alerting
+parent: Monitors
+redirect_from:
+  - /monitoring-plugins/alerting/monitors/
+---
+
+# Actions
+
+Actions send notifications when trigger conditions are met. See [Notifications]({{site.url}}{{site.baseurl}}/notifications-plugin/index/) to learn about creating notifications. If you don't want to receive notifications, don't add actions to your triggers.
+
+## Adding actions
+
+To add an action:
+
+1. In the **Triggers** panel, select **Add action**.
+1. Enter the action details, including action name, notification channel, and notification message body, in the **Notification** section.
+
+    You can add variables to your messages using [Mustache templates](https://mustache.github.io/mustache.5.html/). You have access to `ctx.action.name`, the name of the current action, and all [actions variables](#actions-variables).
+
+    If your notification channel is a custom webhook that expects a particular data format, include JSON (or XML) directly in the message body:
+
+    ```json
+    {% raw %}{ "text": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue. - Trigger: {{ctx.trigger.name}} - Severity: {{ctx.trigger.severity}} - Period start: {{ctx.periodStart}} - Period end: {{ctx.periodEnd}}" }{% endraw %}
+    ```
+
+    In the preceding example, the message content must conform to the `Content-Type` header in the [custom webhook]({{site.url}}{{site.baseurl}}/notifications-plugin/index/).
+
+1. If you're using a bucket-level monitor, choose whether the monitor should perform an action for each execution or for each alert.
+1. (Optional) Use action throttling to limit the number of notifications you receive within a given time frame.
+
+    For example, if a monitor checks a trigger condition every minute, you could receive one notification per minute. If you set action throttling to 60 minutes, you receive no more than one notification per hour, even if the trigger condition is met dozens of times in that hour.
+
+1. Choose **Create**.
+
+After an action sends a message, the content of that message has left the purview of the [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugin. Securing access to the message (for example, access to the Slack channel) is your responsibility.
+
+#### Example message
+
+```mustache
+{% raw %}Monitor {{ctx.monitor.name}} just entered an alert state. Please investigate the issue.
+- Trigger: {{ctx.trigger.name}}
+- Severity: {{ctx.trigger.severity}}
+- Period start: {{ctx.periodStart}}
+- Period end: {{ctx.periodEnd}}{% endraw %}
+```
+
+To use the `ctx.results` variable in a message, use `{% raw %}{{ctx.results.0}}{% endraw %}` rather than `{% raw %}{{ctx.results[0]}}{% endraw %}`. This difference is due to how Mustache handles bracket notation.
+{: .note }
+
+#### Actions variables
+
+Variable | Data type | Description
+:--- | :--- | : ---
+`ctx.trigger.actions.id` | String | The action ID.
+`ctx.trigger.actions.name` | String | The action name.
+`ctx.trigger.actions.message_template.source` | String | The message to send in the alert.
+`ctx.trigger.actions.message_template.lang` | String | The scripting language used to define the message. Must be Mustache.
+`ctx.trigger.actions.throttle_enabled` | Boolean | Whether throttling is enabled for this trigger. See [adding actions](#adding-actions) for more information about throttling.
+`ctx.trigger.actions.subject_template.source` | String | The message's subject in the alert.
+`ctx.trigger.actions.subject_template.lang` | String | The scripting language used to define the subject. Must be Mustache.

_observing-your-data/alerting/api.md

@@ -11,16 +11,7 @@ redirect_from:
 
 Use the Alerting API to programmatically create, update, and manage monitors and alerts.
 
----
-
-#### Table of contents
-- TOC
-{:toc}
-
-
----
-
-## Create a query-level monitor
+## Query-level monitors
 Introduced 1.0
 {: .label .label-purple }
 
@@ -324,7 +315,7 @@ For a full list of timezone names, refer to [Wikipedia](https://en.wikipedia.org
 
 ---
 
-## Create a bucket-level monitor
+## Bucket-level monitors
 
 Bucket-level monitors categorize results into buckets separated by fields. The monitor then runs your script with each bucket's results and evaluates whether to trigger an alert. For more information about bucket-level and query-level monitors, see [Create monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/).
 
@@ -588,12 +579,11 @@ Introduced 2.0
 
 Document-level monitors check whether individual documents in an index match trigger conditions. If so, the monitor generates an alert notification. When you run a query with a document-level monitor, the results are returned for each document that matches the trigger condition. You can create trigger conditions based on query names, query IDs, or tags that combine multiple queries.
 
-To learn more about per document monitors that function similarly to the document-level monitor API, see [Monitor types]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#monitor-types).
+To learn more about per document monitors that function similarly to the document-level monitor API, see [Monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/monitors/).
 
-### Search for monitor findings
-
-You can use the Alerting search API operation to search the findings index `.opensearch-alerting-finding*` for available document findings with a GET request. By default, a GET request without path parameters returns all available findings. To learn more about monitor findings, see [Document findings]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#document-findings).
+### Search the findings index
 
+You can use the Alerting search API operation to search the findings index `.opensearch-alerting-finding*` for available document findings with a GET request. By default, a GET request without path parameters returns all available findings. 
 
 To retrieve any available findings, send a GET request without any path parameters as follows:
 
@@ -621,7 +611,6 @@ Path parameter | Description | Usage
 `startIndex` | The pagination indicator. | Default is `0`.
 `searchString` | The finding attribute you want returned in the search. | To search in a specific index, specify the index name in the request path. For example, to search findings in the `indexABC` index, use `searchString=indexABC'.
 
-
 ### Create a document-level monitor
 
 You can create a document-level monitor with a POST request that provides the monitor details in the request body.

_observing-your-data/alerting/dashboards-alerting.md

@@ -46,7 +46,7 @@ Keep in mind the following requirements when setting up or creating alerting vis
 
 ## Creating alerting monitors
 
-By default, when you begin to create the alert monitor workflow using the Dashboard interface, you are presented with a menu-driven interface. This interface provides a range of options that are displayed in full screen, in pop-ups, in pull-downs, or in dropdowns. They allow you to define the metrics that can be monitored, set thresholds, customize triggers that automate workflows, and generate actions when conditions are met. Currently, you can only create [per query monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/monitors/#monitor-types).
+By default, when you begin to create the alert monitor workflow using the Dashboard interface, you are presented with a menu-driven interface. This interface provides a range of options that are displayed in full screen, in pop-ups, in pull-downs, or in dropdowns. They allow you to define the metrics that can be monitored, set thresholds, customize triggers that automate workflows, and generate actions when conditions are met. Currently, you can only create [per query monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/monitors/).
 
 To create an alerting monitor: 
 

_observing-your-data/alerting/index.md

@@ -9,17 +9,44 @@ redirect_from:
 ---
 
 # Alerting
-OpenSearch Dashboards
-{: .label .label-yellow :}
 
-You can use the Alerting plugin in OpenSearch Dashboards to monitor your data and create alert notifications that trigger when conditions occur in one or more indexes.
+To create an alert, do the following: 
 
-You create a monitor with trigger conditions that generate various alert notifications through the message channel you select as a destination. Notifications can be sent to email, Slack, or Amazon Chime.
+- Configure a _monitor_, which is a job that runs on a defined schedule and queries OpenSearch indexes. Required.
+- Configure one or more _triggers_, which define the conditions that generate events. Optional.
+- Configure _actions_, which is what happens after an alert is triggered. Optional.
 
-The monitor you create notifies you when data from one or more OpenSearch indexes meets certain conditions. For example, you might want to notify a [Slack](https://slack.com/) channel if your application logs more than five HTTP 503 errors in one hour, or you might want to page a developer if no new documents have been indexed in the past 20 minutes.
+## Getting started
 
-To get started, choose **Alerting** in OpenSearch Dashboards.
+To get started with creating alerts:
 
-![OpenSearch Dashboards side bar with link]({{site.url}}{{site.baseurl}}/images/dashboards-nav.png)
+1. Choose **Alerting** from the OpenSearch Plugins main menu, then **Create monitor**. If alerts exist, you'll see a list of those alerts and the Create monitor button won't appear. In this case, select the **Monitors** tab, then **Create monitor**.   
+2. Create a per query, per bucket, per cluster metrics, or per document monitor. For instructions, see [Monitors]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/).
+3. Create one or more triggers. For instructions, see [Triggers[({{site.url}}{{site.baseurl}}/observing-your-data/alerting/triggers/)].
+4. For Actions, set up a notification channel for the alert. For instructions, see [Actions]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/actions/).
 
-***Figure 1: Alerting plugin in OpenSearch Dashboards***
+## Alerting terminology
+
+The following table lists alerting terminology commonly used in OpenSearch.
+
+Term | Definition
+:--- | :---
+Monitor | A job that runs on a defined schedule and queries OpenSearch indexes. The results of these queries are then used as input for one or more *triggers*.
+Trigger | A condition that, if met, generates an *alert*.
+Tag | A label that can be applied to multiple queries to combine them with the logical `OR` operation in a per document monitor. You cannot use tags with other monitor types.
+Alert | An event associated with a trigger. When an alert is created, the trigger performs *actions*, which can include sending a notification.
+Action | The information that you want the monitor to send after being triggered. Actions have a *channel*, a message subject, and a message body.
+Channel | A notification channel to use in an action. Supported channels are Amazon Chime, Slack, Amazon Simple Notification Service (Amazon SNS), email, or custom webhook. See [Notifications]({{site.url}}{{site.baseurl}}/notifications-plugin/index/) for more information.
+Finding | An entry for an individual document found by a per document monitor query that contains the document ID, index name, and timestamp. Findings are stored in the Findings index `.opensearch-alerting-finding*`.
+
+## Alert states
+
+The following table lists the alert states. 
+
+State | Description
+:--- | :---
+Active | The alert is ongoing and unacknowledged. Alerts remain in this state until you acknowledge them, delete the trigger associated with the alert, or delete the monitor entirely. Alerts also can be moved out of the active state if the trigger condition is no longer met. For example, if an index has 4,000 documents and a trigger condition is `numOfDocs > 5000`, an active alert is generated when 3,000 documents are added to the index. If the added 3,000 documents are then deleted from the index, the alert changes to the completed state because the condition is no longer triggered.
+Acknowledged | The alert is acknowledged but the root cause is not fixed.
+Completed | The alert is no longer ongoing. Alerts enter this state after the corresponding trigger evaluates to `false`.
+Error | An error occurred while executing the trigger---usually the result of a bad trigger or destination.
+Deleted | The monitor or trigger associated with this alert was deleted while the alert was ongoing.