You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the rpc server triggers a re-compilation of a dbt project, all of the nodes in the project are re-compiled. This recompilation process can be slow (especially when models need to hit the database), and yet the compiled node results are largely unused.
This compilation process is currently used to inject ephemeral models as CTEs into queries submitted via run_sql and compile_sql. Can we extricate ephemeral model injection from the rest of compilation? Or, are we otherwise able to limit the set of models that are compiled in a given re-compilation of the entire project?
Let's try to pursue an approach where the rpc server maintains a parsed, not compiled, representation of the manifest in memory. When the run_sql or compile_sql methods are invoked, only the ephemeral ancestors of the "just-in-time" rpc node should be compiled. Any "concrete" models will not need to be compiled, as their parsed representations provide all the information needed for effective compilation of the rpc node.
The text was updated successfully, but these errors were encountered:
Describe the feature
When the rpc server triggers a re-compilation of a dbt project, all of the nodes in the project are re-compiled. This recompilation process can be slow (especially when models need to hit the database), and yet the compiled node results are largely unused.
This compilation process is currently used to inject ephemeral models as CTEs into queries submitted via
run_sql
andcompile_sql
. Can we extricate ephemeral model injection from the rest of compilation? Or, are we otherwise able to limit the set of models that are compiled in a given re-compilation of the entire project?Let's try to pursue an approach where the rpc server maintains a parsed, not compiled, representation of the manifest in memory. When the
run_sql
orcompile_sql
methods are invoked, only the ephemeral ancestors of the "just-in-time" rpc node should be compiled. Any "concrete" models will not need to be compiled, as their parsed representations provide all the information needed for effective compilation of the rpc node.The text was updated successfully, but these errors were encountered: