dbt_fivetran_log 0.5.2

Happy Wednesday!

This release of the dbt_fivetran_log package includes the following updates:

Fixes

• The fivetran_log__connector_status model uses a date function off of created_at from the stg_fivetran_log__log model. This fails on certain redshift destinations as the timestamp is synced as timestamptz. Therefore, the field within the staging model is cast using dbt_utils.type_timestamp to appropriately cast the field for downstream functions. Further, to future proof, timestamps were cast within the following staging models: account, account_membership, active_volume, destination_membership, destination, log, transformation, and user. (#40)

dbt_fivetran_log 0.5.1

Features

This release just introduces Databricks compatibility! 🧱🧱

dbt_fivetran_log 0.5.0

🎉 Official dbt v1.0.0 Compatibility Release 🎉

🚨 Breaking Changes 🚨

• Adjusts the require-dbt-version to now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.
  • For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
  • For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
• Upgrades the package dependency to refer to the latest dbt_fivetran_utils. The latest dbt_fivetran_utils package also has a dependency on dbt_utils [">=0.8.0", "<0.9.0"].
  • Please note, if you are installing a version of dbt_utils in your packages.yml that is not in the range above then you will encounter a package dependency error.

Additional Features

• Materializes the fivetran_log__audit_table incrementally, and employs partitioning for BigQuery users. As a non-incremental table, this model involved high runtimes for some users (#27)
  • If you would like to apply partitioning to the underlying source tables (ie the LOG table), refer to Fivetran docs on how to do so.
• Expands compatibility to Postgres!

Thanks to @b-luu for contributing! (#30)

dbt_fivetran_log 0.5.0-b1

🎉 dbt v1.0.0 Compatibility Pre Release 🎉 An official dbt v1.0.0 compatible version of the package will be released once existing feature/bug PRs are merged.

🚨 Breaking Changes 🚨

• Adjusts the require-dbt-version to now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.
  • For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
  • For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
• Upgrades the package dependency to refer to the latest dbt_fivetran_utils. The latest dbt_fivetran_utils package also has a dependency on dbt_utils [">=0.8.0", "<0.9.0"].
  • Please note, if you are installing a version of dbt_utils in your packages.yml that is not in the range above then you will encounter a package dependency error.

Include Priority First Syncs as Connector Health Statuses

Happy Monday!

This release addresses issue #23, and incorporates logic into the fivetran_log__connector_status model to accommodate the way priority-first syncs are currently logged. Priority first sync is now included as a potential connector_health value.

More details in the CHANGELOG.

dbt 0.20.0 Compatibility

🎉 dbt 0.20.0 Compatibility 🎉

🚨 This is a breaking change! 🚨 dbt v0.20.0 or greater is required for this release. If you are not ready to upgrade, consider using a previous release of this package.

Additional considerations when upgrading to this package:

• This package utilizes the latest v0.7.0 release of the dbt-labls/dbt_utils package. If your project also utilizes a version of the fishtown-analytics/dbt_utils package then you will receive a duplicate package error. If this is the case you will need to consider upgrading your other packages to be compatible with this update or use a previous release of this package.
• Similar to the above point, all previous Fivetran dbt packages utilize the fishtown-analytics/dbt_utils package and you will need to upgrade all Fivetran dbt packages to the latest dbt 0.20.0 compatibility releases in order for your packages to run without package conflicts.
• The addition of the fivetran_log_using_sync_alert_messages variable to allow users to disable the Fivetran error and warning alerts from the fivetran_log__connector_status model as some users experienced list_agg errors if the messages were too long.

Fivetran Utils Dependency Update

🚧 Fivetran Utils Dependency Update 🚧

This release includes the following non-breaking changes:

• Update to the packages.yml to reference the recently released Fivetran Utils package. This is a necessary update to ensure this package is compatible with previous dbt releases before dbt v0.20.0 updates take place.

Connector Type ID Bug Fix

This release includes the following non breaking changes:

• Resolution to Issue #14 by adding the get_connector_columns macro and stg_fivetran_log__connector_tmp model to the package in order to reflect the connector level change that resulted in removing the connector_type field and now only contains connector_type_id.

Package Updates (Breaking Changes!)

A bunch of breaking changes ahead!! 🚨🚨🚨

This new release:

• Adds double underscores to model names (fivetran_log_connector_status is now fivetran_log__connector_status).
• Renames the daily API calls model to fivetran_log__connector_daily_events, as it now includes daily metrics regarding record modifications and schema changes.
• Assumes that Fivetran Log data is provided at the account level and removes functionality for unioning individual destination-level log connectors. This can now be configured in Fivetran's UI.
• Creates a new fivetran_log__audit_table model that reports on the rows updated, deleted, and inserted/replaced into each table during connector syncs.
• Creates a new fivetran_log__schema_changelog model that provides details surrounding each connector schema change event.
• Deprecates the data_sync_status column from the final fivetran_log__connector_status model. Refer to the table's connector_health column.
• Reverts the renaming of LOG.connector_id. The value of this column was previously erroneously reported as the connector name instead of the ID.
• Incorporates model disablement variables to handle cases where the TRANSFORMATION and TRIGGER_TABLE tables do not exist.
• Deprecates the destination_database column. Refer to destination_id and destination_name.
• Incorporates custom schemas.

minor bug fix

This release fixes a minor bug in the fivetran_log_connector_daily_api_calls model. This bug was excluding api calls made on the day that the connector was first set up. This is not a breaking change.