-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate more complete compilation output #588
Comments
add comments to each statement explaining what it is |
Would it be possible to output all the compiled statements during Alternatively something like a --dry-run flag would be useful to run the parsing stage only, this might be preferable as compiling all the materialization macros, checking for existing tables etc, is possibly going to trigger a lot more DB calls than just a regular compile! |
hey @lewish! Yeah - I really want to do this. We prioritized it and deprioritized it for 0.9.1 because it proved to be a bigger chunk to bite off than we could manage at the time. This is scheduled for our next minor release, 0.11.0. dbt previously had a |
@lewish thinking about this further, it's not exactly clear how dbt should handle introspective adapter functions during compilation. If you have a model like:
I think our only real option here is to run the query. Ideally, we'd be able to cache this query so that it's only executed once during the whole parsing stage. Just wanted to point out that there's a bit of a challenge here, and there's sort of a tradeoff between speed and correctness. |
Example branch: https://github.com/fishtown-analytics/dbt/compare/investigate/compiled-output?expand=1 This is very doable, but will require non-trivial changes to the materialization code. Closing this as the "investigation" is complete. We should prioritize an issue in the future to actually implement the functionality described here. |
Presently, compiled output in the
target/
directory only shows themain
statement in a dbt resource. This is undesirable for mulit-step resources like archives, incremental models, or table models with flags applied (eg.--non-destructive
). Instead, dbt should show every executed SQL statement involved in creating a resource, as well as pre- and post- hooks.dbt writes rendered sql to the
target/
directory twice: once during parsing and then again during model running.Both "parsing" and "running" invoke the statement block. In this same
Statement
block, there's a line that saysif name == 'main'
, thenwrite
the compiled output to a file. Instead, this block shouldwrite
the compiled output to a file for every statement. The "main" statement should still be used to determine the overall "status" for the resource.The write function exists in the dbt context, and is responsible for writing compiled sql out to a file. This method should be changed (or a new method should be created) which writes the output of multiple statements into the same file. This could either be implemented by 1) writing statement sql to a file using
append
mode or 2) buffering the entire compiled sql output for a resource, then writing the whole thing to a file when the resource has completed running.The text was updated successfully, but these errors were encountered: