Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix/transformation log and changelog test #51

Merged

Conversation

fivetran-joemarkiewicz
Copy link
Contributor

Are you a current Fivetran customer?

Fivetran created PR

What change(s) does this PR introduce?

This PR makes two notable changes:

  1. The unique_combination_of_columns schema test for the fivetran_log__schema_changelog model has been adjusted to also account for the message_data field. This adjustment is necessary as the Fivetran Log connector has been modified to sync multiple records at the same time. Therefore, multiple schema changes may come in at the same time. Therefore, we now want to look at the actual change for uniqueness as well.
  2. A number of customers have noted that their fivetran_log__connector_status model fails due to JSON parsing not being able to be performed on non-JSON fields (this is only occurring in Snowflake instances). It was found that these non-JSON fields are limited only to TRANSFORMATIONS event_subtypes. Since transformation logs are not needed in this model in the first place, an adjustment has been made to filter the records out before further modeling takes place.
    • Please note I broke this out into separate CTEs since we needed to first filter out all transformations and then do our normal filtering.

Did you update the CHANGELOG?

  • Yes

Does this PR introduce a breaking change?

  • Yes (please provide breaking change details below.)
  • No (please provide an explanation as to how the change is non-breaking below.)

This is a bug fix that will not impact existing customers package runs.

Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)

  • Yes

Is this PR in response to a previously created Bug or Feature Request

How did you test the PR changes?

  • CircleCi
  • Local (please provide additional testing details below)

This succeeded on my local branch as well as customers in addition to CircleCi.

Select which warehouse(s) were used to test the PR

  • BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

Provide an emoji that best describes your current mood

🧪

Feedback

We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your feedback on our existing dbt packages or what you'd like to see next.

@fivetran-joemarkiewicz fivetran-joemarkiewicz marked this pull request as ready for review April 18, 2022 19:38
Copy link
Contributor

@fivetran-jamie fivetran-jamie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@fivetran-joemarkiewicz fivetran-joemarkiewicz merged commit 4690ad5 into main Apr 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants