Bug/redshift json parse

[fivetran-catfritz]

PR Overview

This PR will address the following Issue/Feature:

• [Bug] JSON parsing error in Redshift #113
• Ticket T-621124

This PR will result in the following new package version:

• v1.5.0: I made this breaking mostly since I changed the partitioning for bigquery and cluster for snowflake. It won't cause errors, but the warehouses won't change the partition unless there's a full refresh. Also I was feeling cautious since we are changing the logic of the json_parse macro and also adding the lookback for the incremental fivetran_platform__audit_table. Though tested, these are updates that could introduce unexpected behavior.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes

• The following changes are marked as a breaking change out of caution, as a full refresh may be required if you are experiencing issues after the update.
• For Bigquery and Databricks users, updated the partition_by config to coordinate with the filter used in the incremental logic.
• For Snowflake users, added a cluster_by config for performance.

Feature Updates

• Updated incremental logic for fivetran_platform__audit_table so that it looks back 7 days to catch any late arriving records.
• Updated json parsing logic to prevent run failures when incoming json-like strings are invalid.
• Added filter to fivetran_platform__connector_status so only necessary log records will be parsed.

Under The Hood

• Added macros:
  • fivetran_log_json_parse to handle the updated json parsing.
  • fivetran_log_lookback for use in fivetran_platform__audit_table.
• Updated testing of invalid json strings.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• dbt run –full-refresh && dbt test
• dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

• The appropriate issue has been linked, tagged, and properly assigned
• All necessary documentation and version upgrades have been applied
• docs were regenerated (unless this PR does not include any code or yml updates)
• BuildKite integration tests are passing
• Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

• See validation worksheet

If you had to summarize this PR in an emoji, which would it be?

🔍

[fivetran-joemarkiewicz]

@fivetran-catfritz Thanks so much for working through this update!! I have only a few final questions to be addressed below. Coincidentally, when testing these changes I ran into a real world scenario where I had late arriving sync events from Fivetran. If we were using the old method those late events would have been missed; however, with this update they were captured successfully!

Additionally, one last request would be to doubly confirm with the original customers data that the JSON fix proposal will address this issue. I have 99% confidence that this does, but it would be great to fully validate that the compiled code works as expected with the customers data.

Let me know if you have any questions!

[fivetran-joemarkiewicz]

This may be a strange question, but what scenario would we need a default_start_date? Shouldn't there always by a max(date) on incremental runs? What scenario would there be where we need the coalesce to select the default start date?

[fivetran-catfritz]

Good point. I trialed removing this and it seems to compile and run fine. However for now I will leave this pending discussion on Tuesday.

[fivetran-joemarkiewicz]

Thanks, I just wanted to make sure there likely wouldn't be a scenario where this field would actually be null. I don't believe including it will hurt so I am comfortable leaving it in.

[fivetran-joemarkiewicz]

@fivetran-catfritz thanks again for working through these updates! This PR looks good to go!

[fivetran-avinash]

lgtm