Feature/monitorial (#13)

* updated docs added new monitorial materalisation * fixed spelling * updated execute immediate to prereq * fixed syntax * updated names * fixed spelling * fixed up condition
DataEngineersNZ · May 22, 2023 · 1fe474f · 1fe474f
1 parent 6641997
commit 1fe474f
Show file tree

Hide file tree

Showing 20 changed files with 430 additions and 216 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,8 +1,13 @@
 # dbt_dataengineers_materializations Changelog
 
+## 0.2.6
+
+* Modified alert materialisation to be just the Snowflake Alert Object
+* Added a new materialisation for Snowflake Monitorial Alerts
+
 ## 0.2.5
 
-* Updatre of Alert statement to work with Monitorial Monitors Repository so a macro can defind the Execute Immediate statement instead of in config
+* Update of Alert statement to work with Monitorial Monitors Repository so a macro can defind the Execute Immediate statement instead of in config
 
 ## 0.2.4
 

diff --git a/README.md b/README.md
@@ -7,13 +7,14 @@ This [dbt](https://github.com/dbt-labs/dbt) package contains materizations that
 Add the following to your packages.yml file
 ```
   - git: https://github.com/DataEngineersNZ/dbt-snowflake-datops-materilizations.git
-    revision: "0.2.1"
+    revision: "0.2.6"
 ```
 ----
 
 ## Contents
 Conatins the following materializations for Snowflake:
 
+* Monitorial.io Monitor
 * Alerts
 * File Format
 * Stages
@@ -24,54 +25,93 @@ Conatins the following materializations for Snowflake:
 * Materialised View
 * Generic
 
-## Alerts
+## Monitoral Alerts
 
 Usage
 ```sql
 {{ 
-    config(materialized='alert',
-    is_serverless = False,
-    warehouse_name_or_size  = 'alert_wh',
+    config(materialized='monitorial',
     schedule  = '60 minute',
-    action = 'monitorial',
-    description = 'your description of what is representing the alert',
+    diplay_message = 'your description of what is representing the alert',
     enabled_targets = ['local-dev', 'test', 'prod']
     )
 }}
 ```
-| property                   | description                                                                                      | required | default                       |
-| -------------------------- | ------------------------------------------------------------------------------------------------ | -------- | ----------------------------- |
-| `materialized`             | specifies the type of materialisation to run                                                     | yes      | `alert`                       |
-| `is_serverless`            | specifies if the warehouse should be serverless (task object) or dedicated (alert object)        | no       | `False`                       |
-| `warehouse_name_or_size`   | specifies the warehouse size if serverless otherwise the name of the warehouse to use            | no       | `alert_wh`                    |
-| `schedule`                 | specifies the schedule for periodically evaluating the condition for the alert. (CRON or minute) | yes      | `60 minute`                   |
-| `action`                   | specifies the action to run (either 'monitorial' or enter your own action)                       | no       | `monitorial`                  |
-| `description`              | specifies the description of the alert                                                           | no       | ``                            |
-| `environment`              | specifies the target environment for the alert                                                   | no       | `target.name`                 |
-| `notification_email`       | specifies an override for where the alerts should be emailed to                                  | no *     | `[email protected]` |
-| `api_key`                  | specifies the api key required to authenticate the message                                       | no *     | ``                            |
-| `execute_immediate`        | specifies the statement that needs to be run to feed into the alert                              | no *     | ``                            |
-| `notification_integration` | specifies the email intgeration that should be used                                              | no *     | `EXT_EMAIL_INTEGRATION`       |
-| `error_integration`        | specifies the error intgeration that should be used when using serverless alerts                 | no *     | `EXT_ERROR_INTEGRATION`       |
-| `enabled_targets`          | specifies if the targets which the alert should be enabled for                                   | no       | `[target.name]`               |
-
-* only required if `action` is set to `monitorial`
-* `notification_email` can be set as a global variable in the `dbt_project.yml` file using the `alert_notification_email` variable
-* `notification_integration` can be set as a global variable in the `dbt_project.yml` file using the `alert_notification_integration` variable
-* `error_integration` can be set as a global variable in the `dbt_project.yml` file using the `error_notification_integration` variable
-* `api_key` can be set as a global variable in the `dbt_project.yml` file using the `alert_notification_api_key` variable
+| property                 | description                                                                                                  | required | default                                      |
+| ------------------------ | ------------------------------------------------------------------------------------------------------------ | -------- | -------------------------------------------- |
+| `materialized`           | specifies the type of materialisation to run                                                                 | yes      | `monitorial`                                 |
+| `is_serverless`          | specifies if the warehouse should be serverless (task object) or dedicated (alert object)                    | no *     | `False`                                      |
+| `warehouse_name_or_size` | specifies the warehouse size if serverless otherwise the name of the warehouse to use                        | no *     | `pc_monitorial_wh`                           |
+| `object_type`            | specifies the type of object to be created (options are `alert` or `task`)                                   | no *     | `alert`                                      |
+| `schedule`               | specifies the schedule for periodically evaluating the condition for the alert. (CRON or minute)             | yes      | `60 minute`                                  |
+| `severity`               | specifies the severity of the alert (options are `Critial`, `Error`, `Warning`, `Info`, `Debug`, `Resolved`) | no       | `error`                                      |
+| `environment`            | specifies the target environment for the alert                                                               | no       | `target.name`                                |
+| `display_message`        | specifies the message to be sent out with the alert                                                          | yes      |                                              |
+| `prereq`                 | specifies the statement that needs to be run to feed into the alert                                          | no       | ``                                           |
+| `api_key`                | specifies the monitorial api key required for authentication                                                 | no *     |                                              |
+| `message_type`           | specifes the type of message to be sent, for example `User Login Failure`                                    | no       | `USER_ALERT`                                 |
+| `delivery_type`          | specifies the type of delivery mechanism for the alert (options are `api` or `email`)                        | no       | `api`                                        |
+| `email_integration`      | specifies the email intgeration that should be used                                                          | no *     | `EXT_EMAIL_MONITORIAL_INTEGRATION`           |
+| `notification_email`     | specifies an override for where the alerts should be emailed to                                              | no *     | `pc_monitorial_db.utils.monitorial_dispatch` |
+| `api_function`           | specifies the external function  that should be used when sending via api                                    | no *     | `EXT_ERROR_INTEGRATION`                      |
+| `error_integration`      | specifies the error intgeration that should be used when using serverless alerts                             | no *     | `EXT_ERROR_MONITORIAL_INTEGRATION`           |
+| `enabled_targets`        | specifies if the targets which the alert should be enabled for                                               | no       | `[target.name]`                              |
+
+* `is_serverless` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_serverless` variable
+* `warehouse_name_or_size` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_warehouse_name_or_size` variable
+* `object_type` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_object_type` variable
+* `api_key` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_api_key` variable
+* `delivery_type` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_delivery_type` variable
+* `email_integration` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_email_integration` variable
+* `api_function` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_api_function` variable
+* `error_integration` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_error_integration` variable
+
 
 **Example**
 ```yaml
 vars:
-  alert_notification_email: "[email protected]"
-  alert_notification_integration: "EXT_EMAIL_INTEGRATION"  
-  alert_notification_api_key: "********-****-****-****-************"
-  error_notification_integration: "EXT_ERROR_INTEGRATION"
+  ####################################################
+  ### dbt_dataengineers_materializations variables ###
+  ####################################################
+  default_monitorial_email_integration: "EXT_EMAIL_MONITORIAL_INTEGRATION"
+  default_monitorial_api_integration: "EXT_API_MONITORIAL_INTEGRATION"
+  default_monitorial_error_integration: "EXT_ERROR_MONITORIAL_INTEGRATION"
+  default_monitorial_api_function: "pc_monitorial_db.utils.monitorial_dispatch"
+  default_monitorial_serverless: false
+  default_monitorial_object_type: "alert"
+  default_monitorial_notification_email: "[email protected]"
+  default_monitorial_warehouse_name_or_size: "pc_monitorial_wh"
+  default_monitorial_api_key: "********************"
+  default_delivery_type: "api"    #options are api or email
+```
+
+For more information on Monitorial.io please visit [https://www.monitorial.io/](https://www.monitorial.io/) or contact us at [[email protected]](mailto:[email protected])
+
+
+## Alerts
+
+Usage
+```sql
+{{ 
+    config(materialized='alert',
+    is_serverless = False,
+    action='INSERT INTO yourtable (alert_id, alert_name, result) VALUES (1, ''smaple alert'', ''sample result'')',
+    warehouse_size  = 'alert_wh',
+    schedule  = '60 minute',
+    enabled_targets = ['local-dev', 'test', 'prod']
+    )
+}}
 ```
+| property          | description                                                                                      | required | default         |
+| ----------------- | ------------------------------------------------------------------------------------------------ | -------- | --------------- |
+| `materialized`    | specifies the type of materialisation to run                                                     | yes      | `alert`         |
+| `warehouse_size`  | specifies the warehouse size if serverless otherwise the name of the warehouse to use            | no       | `alert_wh`      |
+| `schedule`        | specifies the schedule for periodically evaluating the condition for the alert. (CRON or minute) | yes      | `60 minute`     |
+| `action`          | specifies the action to run after the  if exists statement                                       | no       | `monitorial`    |
+| `enabled_targets` | specifies if the targets which the alert should be enabled for                                   | no       | `[target.name]` |
 
-For more information on snowwatch please visit [https://www.dataengineers.co.nz/solutions/snowwatch/](https://www.dataengineers.co.nz/solutions/snowwatch/) or contact us at [[email protected]](mailto:[email protected])
 
+### We recommended using Monitorial Monitors in preference to custom alerts, as you can send the results to multiple channels and have more control over the message that is sent out.
 
 ## Stored Procedures
 
@@ -162,12 +202,12 @@ Usage
 
 
 * only one of `schedule` or `task_after` is required.
-* `error_integration` can be set as a global variable in the `dbt_project.yml` file using the `error_notification_integration` variable
+* `error_integration` can be set as a global variable in the `dbt_project.yml` file using the `default_monitorial_error_integration` variable
 
 **Example**
 ```yaml
 vars:
-  error_notification_integration: "EXT_ERROR_INTEGRATION"
+  default_monitorial_error_integration: "EXT_ERROR_MONITORIAL_INTEGRATION"
 ```
 
 ## Streams
@@ -281,15 +321,17 @@ example
 ```sql
 {{ config(materialized='generic') }}
 
-CREATE OR REPLACE api integration SnowWatch_Prod_API
+CREATE OR REPLACE api integration EXT_API_MONITORIAL_INTEGRATION
     api_provider = azure_api_management
     azure_tenant_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
     azure_ad_application_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
-    api_allowed_prefixes = ('https://de-snowwatch-prod-ae-apim.azure-api.net')
+    api_allowed_prefixes = ('https://api.monitorial.io')
     API_KEY = 'xxxxxxxxxxxxxxxxxxxx'
     enabled = true; 
 ```
 
+** Note: ** Integrations require `AccountAdmin` privilages which your `dbt` project should not be running under. It is recommended you adopt `Terraform` to deploy integrtaions out from
+
 ## User Defined Functions
 
 When creating a user defined function, you can use a number of different languages. The following are the supported languages:

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'dbt_dataengineers_materializations'
-version: '0.2.5'
+version: '0.2.6'
 config-version: 2
 
 require-dbt-version: [">=1.3.0", "<2.0.0"]

diff --git a/macros/alerts/enable_alerts.sql b/macros/alerts/enable_alerts.sql
@@ -2,24 +2,16 @@
     {% if flags.WHICH == 'run' %}
         {% do log("START: Locating alerts to resume", info=True) %}
         {% set alerts = [] %}
-        {% set tasks = [] %}
         {% set nodes = graph.nodes.values() if graph.nodes else [] %}
         {% for node in nodes %}
             {% if node.config.materialized == "alert" %}
-                {% if node.config.is_serverless %}
-                    {% do tasks.append(node) %}
-                {% else %}
-                    {% do alerts.append(node) %}
-                {% endif %}
+                {% do alerts.append(node) %}
             {% endif %}
         {% endfor %}
 
         {% if alerts|count > 0 %}
             {% do dbt_dataengineers_materializations.resume_alerts(alerts, false) %}
         {% endif %}
-        {% if tasks|count > 0 %}
-            {% do dbt_dataengineers_materializations.resume_alerts(tasks, true) %}
-        {% endif %}
 
     {% endif %}
 {% endmacro %}
@@ -28,13 +20,8 @@
     {% for node in alert_nodes %}
         {% if target.name in node.config.enabled_targets %}
             {% set relation = api.Relation.create(database=node.database, schema=node.schema, identifier=node.name) %}
-            {% if is_task %}
-                {% do log('Resuming ' ~ level ~ ' task - ' ~ task_relation, info=true) %}
-                {% do dbt_dataengineers_materializations.snowflake_resume_alert_task_statement(relation) %}
-            {% else %}
-                {% do log('Resuming ' ~ level ~ ' alert - ' ~ alert_relation, info=true) %}
-                {% do dbt_dataengineers_materializations.snowflake_resume_alert_statement(relation) %}
-            {% endif %}
+            {% do log('Resuming ' ~ level ~ ' alert - ' ~ alert_relation, info=true) %}
+            {% do dbt_dataengineers_materializations.snowflake_resume_alert_statement(relation) %}
         {% endif %}
     {% endfor %}
 {% endmacro %}