Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Update list_relations_without_caching to support Snowflake bundle 2024_03 #1049

Merged
merged 7 commits into from
May 21, 2024

Conversation

mikealfare
Copy link
Contributor

@mikealfare mikealfare commented May 16, 2024

resolves #1016

Problem

Snowflake introduced an optional bundle 2024_03 which will shortly become mandatory. This bundle changes a few things:

  • dynamic tables used to show up with a type of dynamic_table in show terse objects and show objects, they now show up as tables
  • there is a new field in show objects (but not show terse objects) called is_dynamic that can be used to differentiate tables and dynamic tables

Solution

  • when getting the results back from the list_relations_without_caching macro, check to see if there is an is_dynamic field
  • handle both cases by supplying a "false-y" value when it does not exist
  • since this logic is now a bit heftier, move it into its own method
  • at a later date, pull the optional piece as we will be able to depend on the existence of is_dynamic
  • update existing tests to incorporate dynamic tables
  • add a new test that explicitly checks that we're correctly identifying dynamic tables
  • run the new test with the bundle enabled and disabled

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@mikealfare mikealfare self-assigned this May 16, 2024
@cla-bot cla-bot bot added the cla:yes label May 16, 2024
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-snowflake contributing guide.

1 similar comment
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-snowflake contributing guide.

@mikealfare
Copy link
Contributor Author

I will need to manually run this against Snowflake with the bundle on/off (depending on whether it's currently enabled). I will attach that run after getting feedback (so I don't need to do it multiple times).

@mikealfare mikealfare marked this pull request as ready for review May 20, 2024 20:48
@mikealfare mikealfare requested a review from a team as a code owner May 20, 2024 20:48
@mikealfare
Copy link
Contributor Author

The original run (Latest#1) on this PR tested against an enabled bundle. The second run of integration tests (Latest#2) tested against a disabled bundle. You can see both runs here.

Copy link
Contributor

@VersusFacit VersusFacit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure about some of the conditional logic. Elaboration of cases should articulate if this order is correct or not.

… returning a large number of tuples and encapsulate the result to relation intention of this method
Copy link
Contributor

@VersusFacit VersusFacit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your thorough explanations and the added comments. I concur with your approach, although I personally find no qualms with moving the quote policy into the main parse function. If nothing else, that field could be just an argument and part of the signature. So for me the finished outcome is great!

@mikealfare mikealfare merged commit 5da6fdb into main May 21, 2024
18 checks passed
@mikealfare mikealfare deleted the bug/1016/bundle-2024_03 branch May 21, 2024 05:17
@mikealfare mikealfare added backport 1.6.latest backport 1.7.latest Tag for PR to be backported to the 1.7.latest branch backport 1.8.latest labels May 21, 2024
Copy link
Contributor

The backport to 1.6.latest failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.6.latest 1.6.latest
# Navigate to the new working tree
cd .worktrees/backport-1.6.latest
# Create a new branch
git switch --create backport-1049-to-1.6.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5da6fdbfd0895cf86c49dd2b95d89d3d2a57aa70
# Push it to GitHub
git push --set-upstream origin backport-1049-to-1.6.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.6.latest

Then, create a pull request where the base branch is 1.6.latest and the compare/head branch is backport-1049-to-1.6.latest.

Copy link
Contributor

The backport to 1.7.latest failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.7.latest 1.7.latest
# Navigate to the new working tree
cd .worktrees/backport-1.7.latest
# Create a new branch
git switch --create backport-1049-to-1.7.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5da6fdbfd0895cf86c49dd2b95d89d3d2a57aa70
# Push it to GitHub
git push --set-upstream origin backport-1049-to-1.7.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.7.latest

Then, create a pull request where the base branch is 1.7.latest and the compare/head branch is backport-1049-to-1.7.latest.

Copy link
Contributor

The backport to 1.8.latest failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.8.latest 1.8.latest
# Navigate to the new working tree
cd .worktrees/backport-1.8.latest
# Create a new branch
git switch --create backport-1049-to-1.8.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5da6fdbfd0895cf86c49dd2b95d89d3d2a57aa70
# Push it to GitHub
git push --set-upstream origin backport-1049-to-1.8.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.8.latest

Then, create a pull request where the base branch is 1.8.latest and the compare/head branch is backport-1049-to-1.8.latest.

mikealfare added a commit that referenced this pull request May 21, 2024
mikealfare added a commit that referenced this pull request May 21, 2024
mikealfare added a commit that referenced this pull request May 21, 2024
mikealfare added a commit that referenced this pull request May 21, 2024
mikealfare added a commit that referenced this pull request May 21, 2024
mikealfare added a commit that referenced this pull request May 21, 2024
…4_03 results in dynamic table to issue (#1057)

* backport #1049
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.7.latest Tag for PR to be backported to the 1.7.latest branch backport 1.8.latest cla:yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Dynamic tables with Snowflake change bundle 2024_03 results in dynamic table to issue
2 participants