v1.4.2 dbt_fivetran_log

PR #109 includes the following updates:

Bug Fixes

• Adjusted the stg_fivetran_platform__credits_used and stg_fivetran_platform__usage_cost models to return empty tables (via a limit 0) if the respective fivetran_platform__credits_pricing and/or fivetran_platform__usage_pricing variables are disabled. This is to avoid Postgres data type errors if those tables are null.

Under the Hood

• Included an additional test case within the integration tests where the fivetran_platform__credits_pricing variable is set to false and the fivetran_platform__usage_pricing variable is set to true in order to effectively test this scenario.
• Updated seed files to ensure downstream models properly populate into fivetran_platform__usage_mar_destination_history.

Full Changelog: v1.4.1...v1.4.2

v1.4.1 dbt_fivetran_log

PR #107 includes the following updates:

Bug Fixes

• Adjusted the fivetran_platform__audit_user_activity model to parse the message_data json field to obtain the actor_email information only if the field contains actor.
  • This ensures the JSON parsing is only happening on the fields that are relevant. This will help reduce compute and avoid potential parsing errors from malformed JSON objects.

Under the Hood

• Included auto-releaser GitHub Actions workflow to automate future releases.
• Updated the maintainer PR template to resemble the most up to date format.
• Included a quickstart.yml file to allow for automated Quickstart data model deployments.

Full Changelog: v1.4.0...v1.4.1

v1.4.0 dbt_fivetran_log

Feature Updates

• This release introduces compatibility with SQL Server 🥳 🎆 🍾 (PR #101)

Bug Fixes

• Adjusts the uniqueness test on the recently introduced fivetran_platform__audit_user_activity model to test on log_id and occurred_at (PR #102).
  • Previously, the log_id was erroneously considered the primary key of this model.

Under the Hood

• Removed order by from the final select statement in each model. This was done to reduce compute costs from the models (PR #101).
• Converted all group by's to explicitly reference the names of columns we are grouping by, instead of grouping by column number. This was necessary for SQL Server compatibility, as implicit groupings are not supported (PR #101).

Full Changelog: v1.3.0...v1.4.0

v1.3.0 dbt_fivetran_log

🚨 Breaking Changes 🚨

• Deprecated the transformation and trigger_table source tables and any downstream transforms. These tables only housed information on Fivetran Basic SQL Transformations, which were sunset last year (PR #96).
  • The entire fivetran_platform__transformation_status end model has therefore been removed.
  • As they are now obsolete, the fivetran_platform_using_transformations and fivetran_platform_using_triggers variables have been removed.

👶🏽 New Model Alert 👶🏽

• We have added a new model, fivetran_platform__audit_user_activity (PR #98):
  • Each record represents a user-triggered action in your Fivetran instance. This model is intended for audit-trail purposes, as it can be very helpful when trying to trace a user action to a log event such as a schema change, sync frequency update, manual update, broken connection, etc.
  • This model builds off of this sample query from Fivetran's docs.

🪲 Bug Fixes 🪲

• Tightened incremental logic in fivetran_platform__audit_table, which was seeing duplicates on incremental runs (PR #97).
  • If you are seeing uniqueness test failures on the unique_table_sync_key field, please run a full refresh before upgrading to this version of the package.

🛠 Under the Hood 🛠

• Added a dependency on the dbt_date package (PR #98):

- package: calogica/dbt_date
  version: [">=0.9.0", "<1.0.0"]

Full Changelog: v1.2.0...v1.3.0

v1.2.0 dbt_fivetran_log

PR #92 includes the following updates:

Bug Fixes

• The unique_table_sync_key surrogate key which is created within the fivetran_platform__audit_table has been updated to also be comprised of the schema_name in addition to the connector_id, destination_id, table_name, write_to_table_start fields. This update will also ensure the uniqueness test on this record is accurately testing the true grain of the model.
  • 🚨 Please be aware that as the fivetran_platform__audit_table model is an incremental model a --full-refresh will be needed following the package upgrade in order for this change to properly be applied to all records in the end model. 🚨

Contributors

• @JustMaris (#92)

Full Changelog: v1.1.0...v1.2.0

v1.1.0 dbt_fivetran_log

PR #87 includes the following updates:

🚨 Feature Updates (Breaking Change) 🚨

The below change was made to an incremental model. As such, a dbt run --full-refresh will be required following an upgrade to capture the new column.

• Added schema_name to the fivetran_platform__audit_table end model. This schema name field is captured from the message_data JSON within the log source table. In cases where the schema_name is not provided a coalesce was added to replicate the connector_name as the schema_name.

Note: This may change the row count of your fivetran_platform__audit_table model. However, this new row count is more correct, as it more accurately captures records from database connectors, which can write to multiple schemas.

Documentation Updates

• Fixed links in the README models section to properly redirect to the dbt hosted docs for the relevant models.

Full Changelog: v1.0.1...v1.1.0

v1.0.1 dbt_fivetran_log

PR #85 includes the following update:

Bugfix 🐞

• Update staging models CTE names to current standard used in our other packages (the base, fields, final approach) to avoid potential circular references

Full Changelog: v1.0.0...v1.0.1

v1.0.0 dbt_fivetran_log

The Fivetran Log connector has been renamed to the "Fivetran Platform" connector. To align with this name change, this package is largely being renamed from fivetran_log to fivetran_platform. This is a very breaking change! 🚨 🚨 🚨 🚨

Bottom Line: What you need to update and/or know:

• If you are setting any variables for this package in your dbt_project.yml, update the name of the prefix of the variable(s) from fivetran_log_* to fivetran_platform_*. The default values for variables have not changed.
• Similarly, any references to package models will need to be updated. The prefix of package models has been updated from fivetran_log__* to fivetran_platform__*.
• If you are overriding the fivetran_log source, you will need to update the overrides property to match the new source name (fivetran_platform).
• Run a full refresh, as we have updated the incremental strategies across warehouses.
• The default build schema suffixes have been changed from _stg_fivetran_log and _fivetran_log to _stg_fivetran_platform and _fivetran_platform respectively. We recommend dropping the old schemas.

Note: Things that are NOT changing in the package:

• The name of the Github repository will not be changed. It will remain dbt_fivetran_log
• The package's project name will remain fivetran_log. You will not need to update your packages.yml reference.
• The default source schema will remain fivetran_log. The name of the source schema variable has changed though (fivetran_log_schema -> fivetran_platform_schema).

See details below!

PR #81 introduced the following changes (some unrelated to the connector name change):

🚨 Breaking Changes 🚨

• Updated the prefixes of each model from fivetran_log_* or stg_fivetran_log_* to fivetran_platform_* and stg_fivetran_platform_*, respectively.

Original model name	New model name
fivetran_log__audit_table	fivetran_platform__audit_table
fivetran_log__connector_daily_events	fivetran_platform__connector_daily_events
fivetran_log__connector_status	fivetran_platform__connector_status
fivetran_log__mar_table_history	fivetran_platform__mar_table_history
fivetran_log__schema_changelog	fivetran_platform__schema_changelog
fivetran_log__transformation_status	fivetran_platform__transformation_status
fivetran_log__usage_mar_destination_history	fivetran_platform__usage_mar_destination_history
stg_fivetran_log__account	stg_fivetran_platform__account
stg_fivetran_log__connector	stg_fivetran_platform__connector
stg_fivetran_log__credits_used	stg_fivetran_platform__credits_used
stg_fivetran_log__destination_membership	stg_fivetran_platform__destination_membership
stg_fivetran_log__destination	stg_fivetran_platform__destination
stg_fivetran_log__incremental_mar	stg_fivetran_platform__incremental_mar
stg_fivetran_log__log	stg_fivetran_platform__log
stg_fivetran_log__transformation	stg_fivetran_platform__transformation
stg_fivetran_log__trigger_table	stg_fivetran_platform__trigger_table
stg_fivetran_log__usage_cost	stg_fivetran_platform__usage_cost
stg_fivetran_log__user	stg_fivetran_platform__user

• Updated the prefix of all package variables from fivetran_log_* to fivetran_platform_*.

Original variable name	New variable name	Default value (consistent)
fivetran_log_schema	fivetran_platform_schema	fivetran_log
fivetran_log_database	fivetran_platform_database	target.database
fivetran_log__usage_pricing	fivetran_platform__usage_pricing	Dynamically checks the source at runtime to set as either true or false. May be overridden using this variable if desired.
fivetran_log__credits_pricing	fivetran_platform__credits_pricing	Dynamically checks the source at runtime to set as either true or false. May be overridden using this variable if desired
fivetran_log_using_sync_alert_messages	fivetran_platform_using_sync_alert_messages	True
fivetran_log_using_transformations	fivetran_platform_using_transformations	True
fivetran_log_using_triggers	fivetran_platform_using_triggers	True
fivetran_log_using_destination_membership	fivetran_platform_using_destination_membership	True
fivetran_log_using_user	fivetran_platform_using_user	True
fivetran_log_[default_table_name]_identifier	fivetran_platform_[default_table_name]_identifier	Default table name (ie 'connector' for fivetran_platform_connector_identifier)

• Updated the default build schema suffixes of package models from _stg_fivetran_log and _fivetran_log to _stg_fivetran_platform and _fivetran_platform respectively.

We recommend dropping the old schemas to eradicate the stale pre-name-change models from your destintation.

• Updated the name of the package's source from fivetran_log to fivetran_platform.
• Updated the name of the packages' schema files:
  • src_fivetran_log.yml -> src_fivetran_platform.yml
  • stg_fivetran_log.yml -> stg_fivetran_platform.yml
  • fivetran_log.yml -> fivetran_platform.yml
• Updated the freshness tests on the fivetran_platform source to be less stringent and more realistic. The following source tables have had their default fresness tests removed, as they will not necessarily update frequently:
  • connector
  • account
  • destination
  • destination_membership
  • user
• Updated the incremental strategy of the audit table model for BigQuery and Databricks users from merge to the more consistent insert_overwrite method. We have also updated the file_format to parquet and added a partition on a new sync_start_day field for Databricks. This field is merely a truncated version of sync_start.
  • Run a full refresh to capture these new changes. We recommend running a full refresh every so often regardless. See README for more details.
• The account_membership source table (and any of its transformations) has been deprecated. Fivetran deprecated this table from the connector in June 2023.

Considerations

• ⚠️ If you are overriding the fivetran_log source, you will need to update the overrides property to match the new source name (fivetran_platform).

Under the Hood

• Added documentation for fields missing yml entries.
• Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job (PR #80).
• Updated the pull request templates (PR #80).

Full Changelog: v0.7.4...v1.0.0

v0.7.4 dbt_fivetran_log

PR #79 includes the following updates:

Enhancements

• The sync_id field from the source log table is added to the stg_fivetran_log__log model for ease of grouping events by the sync that they are associated with.

Under the Hood

• Added the get_log_columns macro and included the fill staging cte's within the stg_fivetran_log__log model to ensure the model succeeds regardless of a user not having all the required fields.

Documentation Updates

• The sync_id field is added to the documentation in the fivetran_log.yml file.

Contributors

• @camcyr-at-brzwy (#79)

Full Changelog: v0.7.3...v0.7.4

v0.7.3 dbt_fivetran_log

PR #77 includes the following updates:

Bug Fixes

• The logic within the does_table_exist macro would run the source_relation check across all nodes. This opened dbt compile to erroneous failures in other (non fivetran_log) sources. This macro logic has been updated to only check the source_relation for the specific source in question.
• Adjusted the enabled variable used within the stg_fivetran_log__credits_used model to the more appropriate fivetran_log__credits_pricing name as opposed to fivetran_log__usage_pricing. This ensures a user may override the respective model enablement in isolation of each other.

Documentation Updates

• Added a DECISIONLOG to support the logic behind the fivetran_log__usage_pricing and fivetran_log__credits_pricing variable behaviors within the package.

Full Changelog: v0.7.2...v0.7.3