Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Microbatch] Optimizations: use view for temp relation + remove using clause during delete statement #1192

Merged
merged 6 commits into from
Nov 6, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/unreleased/Fixes-20241104-104610.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Fixes
body: 'Performance fixes for snowflake microbatch strategy: use temp view instead
of table, remove unnecessary ''using'' clause'
time: 2024-11-04T10:46:10.005317-05:00
custom:
Author: michelleark
Issue: "1228"
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

The append strategy can use a view because it will run a single INSERT statement.

When unique_key is none, the delete+insert strategy can use a view beacuse a
When unique_key is none, the delete+insert and microbatch strategies can use a view beacuse a
single INSERT statement is run with no DELETES as part of the statement.
Otherwise, play it safe by using a temporary table.
#} */
Expand All @@ -32,10 +32,10 @@
) %}
{% endif %}

{% if strategy == "delete+insert" and tmp_relation_type is not none and tmp_relation_type != "table" and unique_key is not none %}
{% if strategy in ["delete+insert", "microbatch"] and tmp_relation_type is not none and tmp_relation_type != "table" and unique_key is not none %}
{% do exceptions.raise_compiler_error(
"In order to maintain consistent results when `unique_key` is not none,
the `delete+insert` strategy only supports `table` for `tmp_relation_type` but "
the `" ~ strategy ~ "` strategy only supports `table` for `tmp_relation_type` but "
~ tmp_relation_type ~ " was specified."
)
%}
Expand All @@ -49,7 +49,7 @@
{{ return("view") }}
{% elif strategy in ("default", "merge", "append") %}
{{ return("view") }}
{% elif strategy == "delete+insert" and unique_key is none %}
{% elif strategy in ["delete+insert", "microbatch"] and unique_key is none %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 This is a greeeeaaaat catch! Nice!

{{ return("view") }}
{% else %}
{{ return("table") }}
Expand Down
1 change: 0 additions & 1 deletion dbt/include/snowflake/macros/materializations/merge.sql
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@
{% do arg_dict.update({'incremental_predicates': incremental_predicates}) %}

delete from {{ target }} DBT_INTERNAL_TARGET
using {{ source }}
where (
{% for predicate in incremental_predicates %}
{%- if not loop.first %}and {% endif -%} {{ predicate }}
Expand Down
Loading