[Bug] Duplicated months in fivetran_log__mar_table_history

[simon-stepper]

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

The change introduced in v0.7.1 means fivetran_log__mar_table_history is now longer aggregated on a monthly basis but instead on a daily basis. The latest_mar CTE is not filtering our grouping by a month. Hence, the unique combination of columns test fails.

Relevant error log or model output

See below for a screenshot with an excerpt of the output in our dbt project.

Expected behavior

Only one row per table, month, source, destination combination as before

dbt Project configurations

Adjust source for Fivetran Log dbt package and disable transformation models as we don't use that functionality.

vars:
fivetran_log:
fivetran_log_database: raw
fivetran_log_schema: fivetran_log
fivetran_log_using_transformations: false # this will disable all transformation + trigger_table logic
fivetran_log_using_triggers: false # this will disable only trigger_table logic

Package versions

packages:

• package: fivetran/fivetran_log
  version: [">=0.7.0", "<0.8.0"]

What database are you using dbt with?

snowflake

dbt Version

Core:

• installed: 1.4.1
• latest: 1.4.1 - Up to date!

Plugins:

• snowflake: 1.4.1 - Up to date!

Additional Context

Will open a PR with a fix in a moment

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[fivetran-joemarkiewicz]

Hi @simon-stepper thank you for being so quick to upgrade and catch this issue.

This was in fact an oversight in our latest release and one we need to account for within the identified model. I will review your PR and look to fold this into the next patch release as soon as possible.

[fivetran-joemarkiewicz]

Posting an update here:

@simon-stepper was able to open a PR to propose a fix for this issue. I was able to review and made a few minor edits. For any other customers experiencing this issue, it would be great if you could give this branch a test and confirm if the test validations are accurate and passing on your end. Thanks!

packages:
  - git: https://github.com/simon-stepper/dbt_fivetran_log.git
    revision: 73-duplicated-months
    warn-unpinned: false 

[pkanter]

@fivetran-joemarkiewicz , I'm unsure of how to test the fix. Is it just a matter of adding the verbiage you provided into my packages folder?

[fivetran-joemarkiewicz]

@pkanter correct! You can comment out the official fivetran_log version and put the git version in it's place. A dbt clean and dbt deps should then pull in these changes for you to dbt run and dbt test

[fivetran-joemarkiewicz]

My local packages.yml looks like the following:

packages:
  # - package: fivetran/fivetran_log
  #   version: [">=0.7.0", "<0.8.0"]
  - git: https://github.com/simon-stepper/dbt_fivetran_log.git
    revision: 73-duplicated-months
    warn-unpinned: false 

[pkanter]

Just built my project twice, no errors here.

[fivetran-joemarkiewicz]

Brilliant! And your values are still tying out to the UI following these changes from the branch?

[pkanter]

I just did a refresh of my PBI model, compared the new table to the measures I had written myself, and everything ties out. Thank you for the fix.

[fivetran-joemarkiewicz]

Perfect! Thanks so much for the quick turnaround and check. I will kick off our internal release review process to get these changes merged and patched into the next release.

[pkanter]

Just updated to 0.7.2, ran and tested without error. Thanks for the quick-fix.

[fivetran-joemarkiewicz]

Thanks so much for the confirmation @pkanter and for the PR @simon-stepper.

Since this issue has been addressed, I will close out the bug report. Please feel free to reopen if necessary.