Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Does not work with Snowflake and quoting true #360

Closed
ghost opened this issue Jun 20, 2023 · 2 comments · Fixed by #381
Closed

[Bug]: Does not work with Snowflake and quoting true #360

ghost opened this issue Jun 20, 2023 · 2 comments · Fixed by #381
Labels
bug Something isn't working

Comments

@ghost
Copy link

ghost commented Jun 20, 2023

Overview

When used in a project with snowflake as target and quoting:identifier:true it fails to load the models. Setting quoting:identifier:false on the model dbt_artifacts in dbt_project.yml does not work.

How to reproduce

Create a project with target on Snowflake and set the quoting to true on identifiers. Add the dbt_artifacts package and run it.

Expected behaviour

That it would work with quoting:true but it seems that there is inconsistent usa of case in the package code.

Environment

Core:

  • installed: 1.5.0
    Plugins:
  • snowflake: 1.5.0
# Add the results here

Please paste the contents of your packages.yml file here:

packages:
  - git: "https://github.com/dbt-labs/dbt-utils.git" # git URL
    revision: 1.1.0 # tag or branch name
  - package: brooklyn-data/dbt_artifacts
    version: 2.4.2
@ghost ghost added the bug Something isn't working label Jun 20, 2023
@QuentinCoviaux
Copy link

Hey, we are also facing the same issue due to the quoting being enabled in our project.

We found a temporary workaround though, until we can have proper support for this.

  1. In dbt_project.yml, temporarily change quoting.identifier to false (to make the deploy easier, we did this in a local dev environment).
  2. Create the dbt_artifacts tables/views in Production with the following command: dbt run -m dbt_artifacts --target prod . This will create the models without casing enforced.
  3. In Production, add the upload_results() to the on-run-end and make sure the quoting.identifier is set to true.
    3.1. To note that upload_sources throws an error if you have some freshness test filters. I believe this used to be a problem and was fixed in Source freshness filters with string quotations break upload_sources macro in Snowflake (v1.0.0 beta prelease) #141 for non-quoted projects.
    In such cases, the output from the macro will be of sort: '{"warn_after": {"count": null, "period": null}, "error_after": {"count": 2, "period": "hour"}, "filter": "datediff(\'day\', \"EventLoadedAt\", current_timestamp()) < 2"}', which is creating a parsing error: 100069 (22P02): Error parsing JSON: missing comma, pos 124.
    To bypass this error, create a custom upload_results macro in your project that will overwrite the package macro and remove the upload_sources call - maybe one could find a more elegant solution but we didn't really need the sources for now since we're mostly interested in execution times.

@glsdown glsdown mentioned this issue Sep 14, 2023
13 tasks
@glsdown
Copy link
Contributor

glsdown commented Sep 14, 2023

Hi @nc-ckb Thanks for raising this and sorry it's taken a while.

As noted in the dbt docs, due to the way snowflake handles the capitalisation of tables without quoting it was causing errors with the DML we were using. However I have a PR open #381 that should resolve this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants