-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BLOCKING] Define what values end up in the rendering context, and where #1255
Comments
I'm on board, but we need to document how |
Yeah! A great example is: should vars defined or overridden in dbt_project.yml's I would prefer to only support cli-supplied |
@beckjake strong agree that we should document these contexts more clearly. I'm all for documenting the current state of things (even if it's dumb, unintuitive) but I don't think we should actually mutate any contexts right now. If anything, I want to bias in favor of restricting the available context as much as possible (for now). We ran into this with When we do document this, I want to document only the set of context variables that we're sure we can support well going forwards. That might mean things like not documenting I'm very on board with documenting the craziness, but let's produce two sorts of descriptions:
The differences between the two will point out where things are inconsistent/poorly defined. We should then queue up work to resolve these inconsistencies. @beckjake if you can do the first part, I can do the second part :) |
Without checking, pretty sure that if you've defined But I agree, that's why I think we should only support cli-defined I'll do my best to document the status quo over the next few days. As you indicate, var scoping is arcane at best! |
I just kicked this into Wilt Chamberlain. We should do it, but I just need to find some time to write everything down (and figure out a sensible format, since it's a lot of data!) Would it be reasonable to unit test the context? I'm thinking we could write a contract, then use that contract to generate (even manually!) the docs. Otherwise, it's kind of a mess to keep everything straight everywhere. This doesn't really handle things like the difference in |
Yes, I think that's a fantastic idea! |
Just a quick thought:
I think this would be a really useful jumping off point for consistently giving at least a minimal scope to these edge cases you've mentioned, but is there any reason why anything cli-defined should not be included in this minimal scope? A real case where this could be useful is sources that are dependent on target (which may be redundant to mirror as a var to ensure schema.yml has this context). Taking the sources example a step farther, it could be that a target or var(s) (specified at the command line) determine some logic / several other vars that in turn influence how sources should be handled. At present, doing something like this would require some pretty ugly Jinja embedded in schema.yml and / or more CLI variables whose logic you have to enforce elsewhere. It would be extra nice if there was a way to package logic like this (macros?) that would depend only on what is cli-defined, and be in the scope of some of these edge cases. I've heard mentioned that there may be some work towards "pure" macros which don't have dependency on .yml, and could then be used in .yml. Is there an open issue / any update for something like this? Also curious to hear any update on this issue / any thoughts on broadening it to include anything CLI defined. |
Yeah! I think a lot about this,
"pure macros" are an idea I bring up periodically during planning meetings, usually in the context of this issue and stuff like it... There's no issue for them because it's kind of a half-baked idea with a lot of tiny annoying aspects to consider. Off the top of my head:
So yeah, there's my partial brain-dump on why this is so complicated! If you wrote up an issue for "pure macros" and specify what you think the answers to some of these questions should be, I'd love it! I can't guarantee we'd implement it, but user feedback would definitely help us make progress on this front. |
Feature
Currently, rendering in dbt is pretty ad-hoc. We should define what is rendered, where, and with what context very explicitly. It might end up being complex or layered, but at least having it defined and somewhat consistent should be possible.
Feature description
The status quo:
description
fields inschema.yml
are rendered in a context that only supportsdoc
dbt_project.yml
rendering supportsvar
andenv_var
, but does not render any hooks or vars fields (they're used elsewhere)sources
section ofschema.yml
are rendered withvar
andenv_var
available.models
section ofschema.yml
are generally not rendered at all, exceptdescription
s as described above.We should define some rules for these things and formally document them. I think a good start would be:
var
andenv_var
are availble in all rendering contextsdoc
is additionally available in description field renderingmodels
gets treated the same assources
inschema.yml
target
is available inschema.yml
- indbt_project.yml
it's kind of a dependency ordering issue, since we potentially use information in there to pick the target!@drewbanin any feedback on other variables you'd expect to be available in
schema.yml
? It would be nice if we had some sort of user-facing documentation on what you can use and where. I've implemented a lot of this stuff and I find it pretty confusing myself...Who will this benefit?
developers, end users, anyone who wants to understand what dbt is doing with their config.
The text was updated successfully, but these errors were encountered: