bugfix/databricks-sql-path-adjustment

[fivetran-joemarkiewicz]

PR Overview

This PR will address the following Issue/Feature: Issue #120

This PR will result in the following new package version: v1.7.2

This will be non breaking as it will only adjust the regex search in the Datarbricks SQL Warehouse to remove a leading backslash. There will be no breaking changes resulting from this release.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Bug Fixes

• Removal of the leading / from the target.http_path regex search within the is_databricks_sql_warehouse() macro to accurately identify SQL Warehouse Databricks destinations in Quickstart.
  • The macro above initially worked as expected in dbt core environments; however, in Quickstart implementations this data model was not working. This was due to Quickstart removing the leading / from the target.http_path. Thus resulting in the regex search to always fail.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• dbt run –full-refresh && dbt test
• dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

• The appropriate issue has been linked, tagged, and properly assigned
• All necessary documentation and version upgrades have been applied
• docs were regenerated (unless this PR does not include any code or yml updates)
• BuildKite integration tests are passing
• Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

In order to test this I did the following:

• Understood why the v1.7.1 release is not working as expected. It is due to code on the Quickstart side which removes the leading / from the target.http_path. Please see internal messages for link to code. Therefore, removing the leading / from the regex search in the is_databricks_sql_warehouse() macro should do the trick with working on both dbt core and Quickstart.
  • I received confirmation from engineering that this should work as expected.
• After applying the update to the macro I tested to ensure the logic still works as expected on a Databricks SQL Warehouse destination. You can see in the below screenshot that after running dbt run -t databricks-sql (after a full refresh to simulate if there was any incremental strategy) there was no incremental logic included in the model. Further, no incremental models were captured in the end complete statement.
  • 
• To ensure the logic is still working as expected for the Databricks All Purpose Cluster, I performed the same steps to ensure the incremental strategy was being applied. I was able to confirm this is the case!
  • 
• Lastly, I wanted to ensure that there is no change for a non Databricks destination. While no changes were made that should impact any other destination, I did still want to check at least two. I checked Snowflake and BigQuery. Both looked to be in order and utilizing the incremental strategy on subsequent loads!
• BigQuery
  • 
• Snowflake
  • 

If you had to summarize this PR in an emoji, which would it be?

◀️

[fivetran-reneeli]

Got it, so there was a leading / in the SQL Warehouse regex causing it to never match to Databricks because that in the backend, they removed that leading / in http_path. And the other warehouses were not impacted 👍 Looks good!

[fivetran-avinash]

@fivetran-joemarkiewicz good to merge!