v0.20.0 dbt_zendesk

Upstream Under-the-Hood Updates

• (Affects Redshift only) Updates the union_zendesk_connections macro to use a limit 1 instead of limit 0 for empty tables.
  • When a table is empty, Redshift ignores explicit data casts and will materialize every column as a varchar. Redshift users may experience errors in downstream transformations as a consequence.
  • For each staging model, if the source table is not found, the package will create a empty table with 0 rows for non-Redshift warehouses and a table with 1 all-null row for Redshift destinations. The 1 row will ensure that Redshift will respect the package's datatype casts.

Documentation Update

• Moved badges at top of the README below the H1 header to be consistent with popular README formats.

Full Changelog: v0.19.3...v0.20.0

v0.19.3 dbt_zendesk

PR #185 includes the following updates:

Quickstart Update

• Removed organization from using_organization_tags configuration to maintain a one-to-one mapping between a Quickstart source table and whether its corresponding variable in the models should be enabled or disabled.

Under The Hood

• Updated the packages.yml to point to the latest release of the dbt_zendesk_source package.

Full Changelog: v0.19.2...v0.19.3

v0.19.2 dbt_zendesk

PR #181 includes the following updates:

New Features

• Introduced new config variables using_brands and using_organizations to allow customers to enable and disable brand and organization source data, resulting in potential downstream impacts:
• End models that could be impacted by these variables include:
  • zendesk__ticket_backlog: using_brands and/or using_organizations can be set to false to disable brand and/or organization fields.
  • zendesk__ticket_enriched: using_organizations can be set to false to remove organization fields from the final model.
• Intermediate models that could be impacted by these variables include:
  • int_zendesk__organization_aggregates: using_organizations can be utilized to completely disable the model.
  • int_zendesk__ticket_aggregates: using_brands can be set to false to remove brand fields from the final model.
  • int_zendesk__updater_information: using_organizations can be set to false to remove organization fields from the final model.

Under the Hood

• Updated table_variables in the quickstart.yml with the new brand and organization tables.
• Updated our Buildkite model run script to ensure we test for when using_brands and using_organizations is set to either true or false.

Documentation Updates

• Updated README with instructions on how to disable brand and organization sources.

Full Changelog: v0.19.1...v0.19.2

v0.19.1 dbt_zendesk

PR #180 includes the following update:

Bug Fixes

• In v0.19.0, Snowflake users may have seen when searching for a relation, dbt found an approximate match errors when running the stg_zendesk__group_tmp model. We have updated the adapter.get_relation() logic that was causing this error in the zendesk_source package (see source package release).

Full Changelog: v0.19.0...v0.19.1

v0.19.0 dbt_zendesk

PR #178 includes the following updates:

Feature Update: Run Models on Muliple Zendesk Sources

• This release supports running the package on multiple Zendesk sources at once! See the README for details on how to leverage this feature.

Please note: This is a Breaking Change in that we have a added a new field, source_relation, that points to the source connector from which the record originated. This field addition will require a dbt run --full-refresh, even if you are not using this new functionality.

Documentation

• Cleaned up the column-level documentation descriptions for the zendesk__ticket_enriched and zendesk__ticket_metrics models.

Under the Hood

• Relevant to package maintainers only:
  • Added a consistency data validation test for each end model.
  • Added consistency_test_exclude_fields to ignore in consistency tests. These are largely timestamp fields that can differ slightly due to different runtimes, but source_relation is also currently included due to the nature of this update.
  • Filtered out records made or updated today from consistency tests to avoid false positive failures due to different runtimes.
  • Incorporated source_relation into each validation test.

Full Changelog: v0.18.1...v0.19.0

v0.18.1 dbt_zendesk

PR #174 includes the following changes:

Bug Fix

• Addressed an issue in which some records in zendesk__sla_policies might erroneously have a null sla_policy_name due to system-generated millisecond-long gaps in timestamps. The package now compares timestamps to the nearest second when selecting valid SLA policy names in int_zendesk__sla_policy_applied.

Under the Hood

• Updated consistency_sla_policies and sla_count_match data validation tests to account for the above change.

Full Changelog: v0.18.0...v0.18.1

v0.19.0-a1 dbt_zendesk

PR #173 includes the following changes:

Breaking Changes

• Model zendesk__document is now enabled by default. To disable it, set the zendesk__unstructured_enabled variable to false in your dbt_project.yml.

Full Changelog: v0.18.0...v0.19.0-a1

v0.18.0 dbt_zendesk

PR #171 includes the following changes:

Breaking Changes (Full refresh required after upgrading)

Schedule Change Support

• Support for schedule changes has been added. This feature is disabled by default since most users do not sync the required source audit_table. To enable this feature set the variable using_schedule_histories to true in your dbt_project.yml:

vars:
  using_schedule_histories: true

• Schedule changes can now be extracted directly from the audit log, providing a view of schedule modifications over time.
• The int_zendesk__schedule_spine model is now able to incorporate these schedule changes, making it possible for downstream models to reflect the most up-to-date schedule data.
  • Note this is only in effect when using_schedule_histories is true.
• This improves granularity for Zendesk metrics related to agent availability, SLA tracking, and time-based performance analysis.

dbt_zendesk_source changes (see the Release Notes for more details)

• Introduced the stg_zendesk__audit_log table for capturing schedule changes from Zendesk's audit log.
  • This model is disabled by default, to enable it set variable using_schedule_histories to true in dbt_project.yml.

New Features

• Holiday support: Users can now choose to disable holiday tracking, while continuing to use schedules, by setting variable using_holidays to false in dbt_project.yml.
• New intermediate models have been introduced to streamline both the readability and maintainability:
  • int_zendesk__timezone_daylight: A utility model that maintains a record of daylight savings adjustments for each time zone.
    • materialization: ephemeral
  • int_zendesk__schedule_history: Captures a full history of schedule changes for each schedule_id.
    • materialization: table (if enabled)
  • int_zendesk__schedule_timezones: Merges schedule history with time zone shifts.
    • materialization: ephemeral
  • int_zendesk__schedule_holiday: Identifies and calculates holiday periods for each schedule.
    • materialization: ephemeral
• Rebuilt logic in int_zendesk__schedule_spine to consolidate updates from the new intermediate models.

dbt_zendesk_source changes (see the Release Notes for more details)

• Updated the stg_zendesk__schedule_holidays model to allow users to disable holiday processing by setting variable using_holidays to false.

Bug Fixes

• Resolved a bug in the int_zendesk__schedule_spine model where users experienced large gaps in non-holiday periods. The updated logic addresses this issue.

Decision log

• Added the following DECISIONLOG entries:
  • Entry addressing how multiple schedule changes in a single day are handled. Only the last change of the day is captured to align with day-based downstream logic.
  • Entry to clarify backfilling of schedule history. The most recent schedule is sourced from stg_zendesk__schedule, while historical changes are managed separately, allowing users to disable the history feature if needed.

Under the Hood

• Replaced instances of dbt.date_trunc with dbt_date.week_start to standardize week start dates to Sunday across all warehouses, since our schedule logic relies on consistent weeks.
• Replaced the deprecated dbt.current_timestamp_backcompat() function with dbt.current_timestamp() to ensure all timestamps are captured in UTC.
• Added seed data for audit_log to enhance integration testing capabilities.
• Introduced new helper macros, clean_data and regex_extract, to process complex text of the schedule changes extracted from audit logs.
• Updated int_zendesk__calendar_spine logic to prevent errors during compilation before the first full run, ensuring a smoother development experience.

Full Changelog: v0.17.0...v0.18.0

v0.17.0 dbt_zendesk

New model (#161)

• Addition of the zendesk__document model, designed to structure Zendesk textual data for vectorization and integration into NLP workflows. The model outputs a table with:
  • document_id: Corresponding to the ticket_id
  • chunk_index: For text segmentation
  • chunk: The text chunk itself
  • chunk_tokens_approximate: Approximate token count for each segment
• This model is currently disabled by default. You may enable it by setting the zendesk__unstructured_enabled variable as true in your dbt_project.yml.
  • This model was developed with the limit of chunk sizes to approximately 5000 tokens for use with OpenAI, however you can change this limit by setting the variable zendesk_max_tokens in your dbt_project.yml.
  • See the README section Enabling the unstructured document model for NLP for more information.

Breaking Changes (Full refresh required after upgrading)

• Incremental models running on BigQuery have had the partition_by logic adjusted to include a granularity of a month. This change only impacts BigQuery warehouses and was applied to avoid the common too many partitions error some users have experienced when partitioning by day. Therefore, adjusting the partition to a month granularity will decrease the number of partitions created and allow for more performant querying and incremental loads. This change was applied to the following models (#165):

  • int_zendesk__field_calendar_spine
  • int_zendesk__field_history_pivot
  • zendesk__ticket_field_history

• In the dbt_zendesk_source v0.12.0 release, the field _fivetran_deleted was added to the following models for use in zendesk__document model (#161):

  • stg_zendesk__ticket
  • stg_zendesk__ticket_comment
  • stg_zendesk__user
  • If you have already added _fivetran_deleted as a passthrough column via the zendesk__ticket_passthrough_columns or zendesk__user_passthrough_columns variable, you will need to remove or alias this field from the variable to avoid duplicate column errors.

Bug Fixes

• Fixed an issue in the zendesk__sla_policies model where tickets that were opened and solved outside of scheduled hours were not being reported, specifically for the metrics requester_wait_time and agent_work_time.
  • Resolved by adjusting the join logic in models int_zendesk__agent_work_time_business_hours and int_zendesk__requester_wait_time_business_hours. (#164, #156)
• Fixed an issue in the zendesk__ticket_metrics model where certain tickets had miscalculated metrics.
  • Resolved by adjusting the join logic in models int_zendesk__ticket_work_time_business, int_zendesk__ticket_first_resolution_time_business, and int_zendesk__ticket_full_resolution_time_business. (#167)

Under the hood

• Added integrity validations:
  • Test to ensure zendesk__sla_policies and zendesk__ticket_metrics models produce consistent time results. (#164)
  • Test to ensure zendesk__ticket_metrics contains all the tickets found in stg_zendesk__ticket. (#167)
• Modified the consistency_sla_policy_count validation test to group by ticket_id for more accurate testing. (#165)
• Updated casting in joins from timestamps to dates so that the whole day is considered. This produces more accurate results. (#164, #156, #167)
• Reduced the weeks looking ahead from 208 to 52 to improve performance, as tracking ticket SLAs beyond one year was unnecessary. (#156, #167)
• Updated seed files to reflect a real world ticket field history update scenario. (#165)

Full Changelog: v0.16.0...v0.17.0

v0.16.0 dbt_zendesk

🚨 Minor Upgrade 🚨

Although this update is not a breaking change, it will likely impact the output of the zendesk__sla_policies and zendesk__sla_metrics models. PR #154 includes the following changes:

Bug Fixes

• Addresses the potential issue where the first_reply_time_business_minutes metric within the zendesk__ticket_metrics model would incorrectly calculate the elapsed time when daylight savings occurred. This change involved adjusting a join to reference the difference of two dates as opposed to timestamps. This more accurately applies a cutoff event during daylight savings.
• Introduction of an additional condition within the filtered_reply_times cte of the int_zendesk__reply_time_combined model to ensure tickets replied to before any schedule begins and no business minutes have been spent is reserved for only the first day the ticket is open. Previously, this condition could be met on days other than the first. This would potentially result in duplicates of sla_event_id's further downstream in the zendesk__sla_policies model.

Under the Hood

• Addition of integrity and consistency validation tests within integration tests for the zendesk__sla_policies and zendesk__ticket_metrics models.

Full Changelog: v0.15.0...v0.16.0