Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove leading period from info logging #74

Closed
jtcohen6 opened this issue Apr 8, 2020 · 3 comments
Closed

Remove leading period from info logging #74

jtcohen6 opened this issue Apr 8, 2020 · 3 comments
Milestone

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Apr 8, 2020

This is low-priority cosmetic issue.

There appears an extra leading period in dbt run:

11:19:56 | 1 of 15 START table model .dbt_jcohen.materialized................... [RUN]
11:20:02 | 1 of 15 OK created table model .dbt_jcohen.materialized.............. [OK in 6.84s]
11:20:02 | 2 of 15 START incremental model .dbt_jcohen.incremental.............. [RUN]
11:20:10 | 2 of 15 ERROR creating incremental model .dbt_jcohen.incremental..... [ERROR in 7.55s]
...

Interestingly, the period appears only in the first line of dbt seed:

11:19:12 | 1 of 3 START seed file .dbt_jcohen.other_seed........................ [RUN]
11:19:20 | 1 of 3 OK loaded seed file dbt_jcohen.other_seed..................... [CREATE 100 in 8.06s]
11:19:20 | 2 of 3 START seed file .dbt_jcohen.other_seed_update................. [RUN]
11:19:27 | 2 of 3 OK loaded seed file dbt_jcohen.other_seed_update.............. [CREATE 200 in 6.72s]
11:19:27 | 3 of 3 START seed file .dbt_jcohen.seed.............................. [RUN]
11:19:33 | 3 of 3 OK loaded seed file dbt_jcohen.seed........................... [CREATE 5 in 5.74s]
  • This issue is new in 0.16.0. When I run 0.15.3, it doesn't appear.
  • I imagine it may be related to Spark's idiosyncratic behavior around databases/schemas.
  • Does it have something to do with the addition of a generate_database_name macro in dbt-core?
@jtcohen6
Copy link
Contributor Author

jtcohen6 commented Apr 15, 2020

I believe the culprit is get_node_representation.

Spark's profile.yml config requires the user to specify schema. A user can optionally also specify database, which is meaningless, because schema == database in Spark. I confirmed that I see this issue when I specify only schema in my profile, and the leading period disappears when I also specify database with the same value as schema. This is a bit surprising, since we set database = schema as a _post_init on the credentials object.

I'm not sure if there's a way to override get_node_representation for Spark, since it's a class method on ModelRunner, not BaseAdapter.

This leads me to think there could be more serious functional issues related to the database model configs that I haven't had a chance to test yet.

@jtcohen6 jtcohen6 added this to the 0.16.1 milestone Apr 21, 2020
@jtcohen6
Copy link
Contributor Author

After more testing, I confirmed this is not an issue with the profile configs, but with relations themselves. Given a profile that sets schema: my_schema_name, this error disappears if I set the following in dbt_project.yml:

models:
  my_project:
    database: my_schema_name

At the same time, if I set it to be something else entirely:

models:
  my_project:
    database: not_the_same

dbt will print out

1 of 15 START table model not_the_same.my_schema_name.table_model..... [RUN]

But it will actually just attempt to create a table located at the namespace my_schema_name.table_model.

I'm going to open a separate issue laying out how I think we should reconcile custom schemas and databases, likely for 0.17.0.

@jtcohen6
Copy link
Contributor Author

jtcohen6 commented May 22, 2020

Fixed by dbt#2450. We should enjoy nicer-looking output in 0.17.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant