Merge pull request #22 from fivetran/feature/joellabes-dbt-20

Feature/joellabes dbt 20

README.md

@@ -1,3 +1,4 @@
+[![Apache License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) ![dbt Logo and Version](https://img.shields.io/static/v1?logo=dbt&label=dbt-version&message=0.20.x&color=orange)
 # Fivetran Log ([docs](https://fivetran.github.io/dbt_fivetran_log/#!/overview))
 
 This package models Fivetran Log data from [our free internal connector](https://fivetran.com/docs/logs/fivetran-log). It uses account-level data in the format described by [this ERD](https://fivetran.com/docs/logs/fivetran-log#schemainformation).
@@ -33,8 +34,18 @@ The package's main goals are to:
 
 
 ## Installation Instructions
+`dbt_fivetran_log` currently supports `dbt 0.20.x`.
+
 Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
 
+Include in your `packages.yml`
+
+```yaml
+packages:
+  - package: fivetran/fivetran_log
+    version: [">=0.4.0", "<0.5.0"]
+```
+
 ## Configuration
 By default, this package will run using your target database and the `fivetran_log` schema. If this is not where your Fivetran Log data is, add the following configuration to your `dbt_project.yml` file:
 
@@ -51,7 +62,7 @@ vars:
 ```
 
 ### Disabling Transformation Models
-If you have never created Fivetran-orchestrated [basic SQL transformations](https://fivetran.com/docs/transformations/basic-sql), your source data will not contain the `transformation` and `trigger_table` tables. Moreover, if you have only created *scheduled* basic transformations that are not triggered by table syncs, your source data will not contain the `trigger_table` table (though it will contain `transformation`).
+If you have never created Fivetran-orchestrated [basic SQL transformations](https://fivetran.com/docs/transformations/basic-sql), your source data will not contain the `transformation` and `trigger_table` tables. Moreover, if you have only created *scheduled* basic transformations that are not triggered by table syncs, your source data will not contain the `trigger_table` table (though it will contain `transformation`). 
 
 To disable the corresponding functionality in the package, you must add the following variable(s) to your `dbt_project.yml` file. By default, all variables are assumed to be `true`:
 
@@ -67,6 +78,19 @@ vars:
     fivetran_log_using_triggers: false # this will disable only trigger_table logic 
 ```
 
+### Disabling Fivetran Error and Warning Messages
+Some users may wish to exclude Fivetran error and warnings messages from the final `fivetran_log__connector_status` model due to the length of the message. To disable the `errors_since_last_completed_sync` and `warnings_since_last_completed_sync` fields from the final model you may add the following variable to you to your `dbt_project.yml` file. By default, this variable is assumed to be `true`:
+```yml
+# dbt_project.yml
+
+...
+config-version: 2
+
+vars:
+  fivetran_log:
+    fivetran_log_using_sync_alert_messages: false # this will disable only the sync alert messages within the connector status model
+```
+
 ### Changing the Build Schema
 By default this package will build the Fivetran Log staging models within a schema titled (<target_schema> + `_stg_fivetran_log`)  and the Fivetran Log final models within your <target_schema> + `_fivetran_log` in your target database. If this is not where you would like you Fivetran Log staging and final models to be written to, add the following configuration to your `dbt_project.yml` file:
 

dbt_project.yml

@@ -1,9 +1,9 @@
 config-version: 2
 
 name: 'fivetran_log'
-version: '0.3.1'
+version: '0.4.0'
 
-require-dbt-version: [">=0.18.0", "<0.20.0"]
+require-dbt-version: ">=0.20.0"
 
 models:
     fivetran_log:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'fivetran_log_integration_tests'
-version: '0.3.1'
+version: '0.4.0'
 config-version: 2
 profile: 'integration_tests'
 

integration_tests/requirements.txt

@@ -1 +1 @@
-dbt==0.19.0
+dbt~=0.20.0

models/fivetran_log__connector_status.sql

@@ -118,18 +118,19 @@ final as (
         connector_recent_logs.last_sync_started_at,
         connector_recent_logs.last_sync_completed_at,
         connector_recent_logs.set_up_at,
-        coalesce(schema_changes.number_of_schema_changes_last_month, 0) as number_of_schema_changes_last_month,
-
-        {{ fivetran_utils.string_agg("case when connector_recent_logs.event_type = 'SEVERE' then connector_recent_logs.message_data else null end", "'\\n'") }} as errors_since_last_completed_sync,
-        {{ fivetran_utils.string_agg("case when connector_recent_logs.event_type = 'WARNING' then connector_recent_logs.message_data else null end", "'\\n'") }} as warnings_since_last_completed_sync
+        coalesce(schema_changes.number_of_schema_changes_last_month, 0) as number_of_schema_changes_last_month
 
+        {% if var('fivetran_log_using_sync_alert_messages', true) %}
+        , {{ fivetran_utils.string_agg("case when connector_recent_logs.event_type = 'SEVERE' then connector_recent_logs.message_data else null end", "'\\n'") }} as errors_since_last_completed_sync
+        , {{ fivetran_utils.string_agg("case when connector_recent_logs.event_type = 'WARNING' then connector_recent_logs.message_data else null end", "'\\n'") }} as warnings_since_last_completed_sync
+        {% endif %}
 
     from connector_recent_logs
     left join schema_changes 
         on connector_recent_logs.connector_id = schema_changes.connector_id 
 
     join destination on destination.destination_id = connector_recent_logs.destination_id
-    group by 1,2,3,4,5,6,7,8,9,10
+    {{ dbt_utils.group_by(n=10) }}
 )
 
 select * from final

packages.yml

@@ -1,3 +1,3 @@
 packages:
 - package: fivetran/fivetran_utils
-  version: [">=0.1.0", "<0.2.0"]
+  version: [">=0.2.0", "<0.3.0"]