v0.7.2 dbt_fivetran_log

Bug Fixes

• Fixed duplicated rows in fivetran_log__mar_table_history and set the model back to a monthly granularity for each source, destination, and table. (#74)

Under the Hood

• Adjusted the uniqueness test within the fivetran_log__mar_table_history to also include the schema_name as the same table may exist in multiple schemas within a connector/destination. (#74)

Contributors

• @simon-stepper (#73)

Full Changelog: v0.7.1...v0.7.2

v0.7.1 dbt_fivetran_log

Bug Fixes

• Modified the logic within the fivetran_log__mar_table_history model to no longer filter out previous historical MAR records. Previously, these fields were filtered out as the active_volume source (since deprecated and replaced with incremental_mar) produced a cumulative daily MAR total. However, the incremental_mar source is not cumulative and will need to include all historical records. (#72)

Under the Hood

• Added coalesce statements to the paid_monthly_active_rows and free_monthly_active_rows fields within the fivetran_log__mar_table_history model to coalesce to 0. (#72)

Contributors

• @pkanter (#63)

Full Changelog: v0.7.0...v0.7.1

dbt_fivetran_log v0.7.0

🚨 Breaking Changes 🚨:

PR #68 includes the following breaking changes:

• The active_volume source (and accompanying stg_fivetran_log__active_volume model) has been deprecated from the Fivetran Log connector. In its place, the incremental_mar table (and accompanying stg_fivetran_log__incremental_mar model) has been added. This new source has been swapped within the package to reference the new source table.
  • This new source table has enriched data behind the paid and free MAR across Fivetran connectors within your destinations.
• Removed the monthly_active_rows field from the fivetran_log__mar_table_history and fivetran_log__usage_mar_destination_history models. In it's place the following fields have been added:
  • free_mothly_active_rows: Detailing the total free MAR
  • paid_mothly_active_rows: Detailing the total paid MAR
  • total_mothly_active_rows: Detailing the total free and paid MAR

PR #64 includes the following breaking changes:

• Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically {{ dbt_utils.<macro> }} have been updated to {{ dbt.<macro> }} for the below macros:
  • any_value
  • bool_or
  • cast_bool_to_text
  • concat
  • date_trunc
  • dateadd
  • datediff
  • escape_single_quotes
  • except
  • hash
  • intersect
  • last_day
  • length
  • listagg
  • position
  • replace
  • right
  • safe_cast
  • split_part
  • string_literal
  • type_bigint
  • type_float
  • type_int
  • type_numeric
  • type_string
  • type_timestamp
  • array_append
  • array_concat
  • array_construct
• For current_timestamp and current_timestamp_in_utc macros, the dispatch AND the macro names have been updated to the below, respectively:
  • dbt.current_timestamp_backcompat
  • dbt.current_timestamp_in_utc_backcompat
• dbt_utils.surrogate_key has also been updated to dbt_utils.generate_surrogate_key. Since the method for creating surrogate keys differ, we suggest all users do a full-refresh for the most accurate data. For more information, please refer to dbt-utils release notes for this update.
• packages.yml has been updated to reflect new default fivetran/fivetran_utils version, previously [">=0.3.0", "<0.4.0"] now [">=0.4.0", "<0.5.0"].

dbt_fivetran_log v0.6.4

Happy Wednesday and National Mousse Day 🍫

This release of the fivetran_log package includes the following updates:

Fixes

• Added second qualifying join clause to fivetran_log__usage_mar_destination_history in the usage cte. This join was failing this test to ensure each destination_id has a single measured_month (#67):

      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: 
            - destination_id
            - measured_month 

Under the Hood

• BuildKite testing has been added. (#70)

Contributors

• @lord-skinner (#67)

dbt_fivetran_log v0.6.3

Fixes

• Modified the argument used for the identifier in the get_relation macro used in the does_table_exist macro from name to identifier. This avoids issues on snowflake where the name of a table defined in a source yaml may be in lowercase while in snowflake it is uppercased.

Contributors

• @ggmblr (#60)

dbt_fivetran_log v0.6.2

Happy Thursday 🌻

This release of the dbt_fivetran_log package includes the following updates:

Fixes

• Extend model disablement with config: is_enabled setting in sources to avoid running source freshness when a model is disabled. (#58)

Contributors

• @epapineau (#58)

dbt_fivetran_log v0.6.1

Happy Tuesday! 🌮

This release of the dbt_fivetran_log package includes the following updates:

Fixes

• Added the option to disable the user, account_membership, and destination_membership models that may not be available depending on your Fivetran setup. In your dbt_project.yml you can now change these flags to disable parts of the package. Use the below to configure your project. (#52) and (#55)

  fivetran_log_using_account_membership: false # Disables account membership models
  fivetran_log_using_destination_membership: false # Disables account membership models
  fivetran_log_using_user: false # Disables account membership models

Contributors

• @cmcau (#52)

dbt_fivetran_log v0.6.0

Happy Monday!

This release of the dbt_fivetran_log package includes the following updates:

🚨 Breaking Changes 🚨

• This release includes updates to the fivetran_log__credit_mar_destination_history and stg_fivetran_log__credits_used models to account for the new Fivetran pricing model. These changes include: (#50)
  • stg_fivetran_log__credits_used
    • The field credits_consumed has been renamed to credits_spent
  • fivetran_log__credit_mar_destination_history
    • The model has been renamed to fivetran_log__usage_mar_destination_history
    • The field credits_per_million_mar has been renamed to credits_spent_per_million_mar
    • The field mar_per_credit has been renamed to mar_per_credit_spent

🎉 Features 🎉

• README documentation updates for easier experience leveraging the dbt package. (#53)
• Added fivetran_log_[source_table_name]_identifier variables to allow for easier flexibility of the package to refer to source tables with different names. (#53)
• This package now accounts for the new Fivetran pricing model. In particular, the new model accounts for the amount of dollars spend vs credits spent. Therefore, a new staging model stg_fivetran_log__usage_cost has been added. (#50)
  • This model relies on the usage_cost source table. If you do not have this source table it means you are not on the new pricing model yet. Please note, the dbt package will still generate this staging model. However, the model will be comprised of all null records.
• In addition to the new staging model, two new fields have been added to the fivetran_log__usage_mar_destination_history model. These fields mirror the credits spent fields, but account for the amount of dollars spent instead of credits. (#50)
  • amount_spent_per_million_mar
  • mar_per_amount_spent

Under the Hood

• Introduces a new macro does_table_exist to be leveraged in the new pricing model updates. This macro will check the sources defined and provide either true or false if the table does or does not exist in the schema. (#50)

dbt_fivetran_log v0.5.4

Happy Wednesday 🐫!

This release of the dbt_fivetran_log package includes the following updates:

Fixes

• The unique combination of columns test within the fivetran_log__schema_changelog model has been updated to also check the message_data field. This is needed as schema changelog events may now sync at the same time. (#51)
• The fivetran_log__connector_status model has been adjusted to filter out all logs that contain a transformation_id. Transformation logs are not always synced as a JSON object and thus the package may encounter errors on Snowflake warehouses when parsing non-JSON fields. Since transformation records are not used in this end model, they have been filtered out. (#51)

dbt_fivetran_log v0.5.3

Happy Wednesday 🐫

This release of the dbt_fivetran_log package includes the following updates:

Fixes

• Per the Fivetran Log December 2021 Release Notes every sync results in a final sync_end event. In the previous version of this package, a successful sync was identified via a sync_end event while anything else was a version of broken. Since all syncs result in a sync_end event now, the package has been updated to account for this change within the connector.
• To account for the above fix, a new field (last_successful_sync_completed_at) was added to the fivetran_log__connector_status model. This field captures the last successful sync for the connector.