Merge pull request #51 from fivetran/bugfix/transformation-log-and-ch…

…angelog-test

Bugfix/transformation log and changelog test

CHANGELOG.md

@@ -1,3 +1,7 @@
+# dbt_fivetran_log v0.5.4
+## Fixes
+- The unique combination of columns test within the `fivetran_log__schema_changelog` model has been updated to also check the `message_data` field. This is needed as schema changelog events may now sync at the same time. ([#51](https://github.com/fivetran/dbt_fivetran_log/pull/51))
+- The `fivetran_log__connector_status` model has been adjusted to filter out all logs that contain a `transformation_id`. Transformation logs are not always synced as a JSON object and thus the package may encounter errors on Snowflake warehouses when parsing non-JSON fields. Since transformation records are not used in this end model, they have been filtered out. ([#51](https://github.com/fivetran/dbt_fivetran_log/pull/51))
 # dbt_fivetran_log v0.5.3
 ## Fixes
 - Per the [Fivetran Log December 2021 Release Notes](https://fivetran.com/docs/logs/changelog#december2021) every sync results in a final `sync_end` event. In the previous version of this package, a successful sync was identified via a `sync_end` event while anything else was a version of broken. Since all syncs result in a `sync_end` event now, the package has been updated to account for this change within the connector.

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'fivetran_log'
-version: '0.5.3'
+version: '0.5.4'
 
 require-dbt-version: [">=1.0.0", "<2.0.0"]
 

integration_tests/ci/sample.profiles.yml

@@ -47,7 +47,7 @@ integration_tests:
     spark:
       type: spark
       method: http
-      schema: fivetran_log_tests
+      schema: fivetran_log__integration_test
       host: "{{ env_var('CI_SPARK_DBT_HOST') }}"
       organization: "{{ env_var('CI_SPARK_DBT_ORGANIZATION') }}"
       token: "{{ env_var('CI_SPARK_DBT_TOKEN') }}"

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'fivetran_log_integration_tests'
-version: '0.5.3'
+version: '0.5.4'
 config-version: 2
 profile: 'integration_tests'
 

models/fivetran_log.yml

@@ -290,6 +290,7 @@ models:
           combination_of_columns: 
             - connector_id
             - destination_id
+            - message_data
             - created_at
     columns:
       - name: connector_id
@@ -310,4 +311,4 @@ models:
       - name: table_name
         description: Name of the table, if a table is being altered or created. 
       - name: schema_name
-        description: Name of the schema, if a schema or table is being created. 
+        description: Name of the schema, if a schema or table is being created. 

models/fivetran_log__connector_status.sql

@@ -1,10 +1,17 @@
-with connector_log as (
+with transformation_removal as (
 
-    select *,
-        sum( case when event_subtype in ('sync_start') then 1 else 0 end) over ( partition by connector_id 
-            order by created_at rows unbounded preceding) as sync_batch_id
+    select *
     from {{ ref('stg_fivetran_log__log') }}
+    where transformation_id is null
+
+),
 
+connector_log as (
+    select 
+        *,
+        sum( case when event_subtype in ('sync_start') then 1 else 0 end) over ( partition by connector_id 
+            order by created_at rows unbounded preceding) as sync_batch_id
+    from transformation_removal
     -- only looking at errors, warnings, and syncs here
     where event_type = 'SEVERE'
         or event_type = 'WARNING'