-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature/databricks-sql-warehouse-compatibility #121
Changes from 19 commits
137ed6c
382dbe2
99d7f83
12ea509
c575861
b0ac2fc
bd2175b
d41ebf8
839c093
4e87cfa
cf257e5
37e5c52
91ffac9
aa43436
bd30997
2cc5a70
8aad434
9df06fb
064955d
ce3abc9
ba8ef14
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
name: 'fivetran_log_integration_tests' | ||
version: '1.7.0' | ||
version: '1.7.1' | ||
|
||
config-version: 2 | ||
profile: 'integration_tests' | ||
|
@@ -10,7 +10,7 @@ dispatch: | |
|
||
vars: | ||
fivetran_log: | ||
fivetran_platform_schema: fivetran_platform_integration_tests | ||
fivetran_platform_schema: "fivetran_platform_integration_tests" | ||
fivetran_platform_account_identifier: "account" | ||
fivetran_platform_incremental_mar_identifier: "incremental_mar" | ||
fivetran_platform_connector_identifier: "connector" | ||
|
@@ -24,7 +24,7 @@ vars: | |
|
||
models: | ||
fivetran_log: | ||
+schema: fivetran_platform | ||
+schema: "{{ 'sqlw_tests' if target.name == 'databricks-sql' else 'fivetran_platform' }}" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is an artifact of needing to use two different schemas for the Databricks jobs |
||
|
||
seeds: | ||
fivetran_log_integration_tests: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
{% macro is_databricks_sql_warehouse(target) %} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we may want to add this to fivetran_utils in the future since Databricks SQL Warehouse folks using other packages with incremental models will have the same issue There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree. This is something we will migrate there in the future. |
||
{% if target.type in ('databricks') %} | ||
{% set re = modules.re %} | ||
{% set path_match = target.http_path %} | ||
{% set regex_pattern = "/sql/.+/warehouses/" %} | ||
{% set match_result = re.search(regex_pattern, path_match) %} | ||
{% if match_result %} | ||
{{ return(True) }} | ||
{% else %} | ||
{{ return(False) }} | ||
{% endif %} | ||
{% else %} | ||
{{ return(False) }} | ||
{% endif %} | ||
{% endmacro %} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is all necessary as the SQL Warehouse still uses the same
hive_metastore
and comes into conflict with the All Purpose Cluster when they both run in parallel with the same schema.Therefore, we need to explicitly use a different schema for the two.