Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix insertion macro for Databricks Unity Catalog #252

Merged

Conversation

TannerHopkins
Copy link
Contributor

This PR fixes an issue for Databricks projects with Unity Catalog enabled.
Currently, the insert_into_metadata_table macro only specifies the schema_name.table_name, but with catalogs (databases) introduced in Unity Catalog, the database_name is also required or the insertion fails.
This should be backwards compatible with non-UC implementations of databricks since they can now also be fully qualified with a database_name of hive_metastore - assuming this value gets populated automatically, of course.

If there are any issues with non-UC implementations of databricks, or spark since it also shares this macro, an alternate solution would be to first check that the database_name has a value, and if so use it, otherwise just keep schema_name.table_name

@TannerHopkins TannerHopkins temporarily deployed to Approve Integration Tests December 5, 2022 16:41 Inactive
@TannerHopkins TannerHopkins temporarily deployed to Approve Integration Tests December 5, 2022 16:41 Inactive
@TannerHopkins TannerHopkins temporarily deployed to Approve Integration Tests December 5, 2022 16:41 Inactive
@TannerHopkins TannerHopkins had a problem deploying to Approve Integration Tests December 5, 2022 16:41 Failure
@TannerHopkins TannerHopkins temporarily deployed to Approve Integration Tests December 5, 2022 19:13 Inactive
@TannerHopkins TannerHopkins temporarily deployed to Approve Integration Tests December 5, 2022 19:13 Inactive
@TannerHopkins TannerHopkins temporarily deployed to Approve Integration Tests December 5, 2022 19:13 Inactive
@TannerHopkins TannerHopkins temporarily deployed to Approve Integration Tests December 5, 2022 19:13 Inactive
@TannerHopkins
Copy link
Contributor Author

Looks like this first attempt failed (UC not enabled on this cluster error).
Going to try it with an if statement - looks like this is actually what @NiallRees originally did to make dbt_artifacts compatible with unity catalog.
I think this macro was just missed in that original PR

@NiallRees NiallRees merged commit 235618c into brooklyn-data:main Dec 8, 2022
@VDFaller
Copy link

VDFaller commented Mar 9, 2023

Is this in version 2.2.2? If not, would it be the reason I'm getting

23:32:37  Database error while running on-run-end
23:32:37  
23:32:37  Finished running 2 table models in 0 hours 0 minutes and 36.15 seconds (36.15s).
23:32:37  
23:32:37  Completed with 1 error and 0 warnings:
23:32:37
23:32:37  on-run-end failed, error:
23:32:37   Table not found: dev_vince.dbt_artifacts.model_executions; line 4 pos 16

My relevant dbt_project.yml looks like

models:
  dbt_artifacts:
    +database: "{{target.catalog}}"
    +schema: dbt_artifacts

and my relevant profile looks like

databricks:
  target: dev
  outputs:
    dev:
      catalog: dev_vince
      schema: literally_anything_else
      type: databricks

@jaypeedevlin
Copy link
Contributor

@VDFaller Have you done a full dbt run or dbt run -s dbt_artifacts? If not the tables wouldn't have been created for the on-run-end to insert the data into...

@VDFaller
Copy link

Nope. much better. Thank you much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants