-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Standardize dbt-generated archival column names #251
Comments
@drewbanin agree? |
@jthandy yeah, totally agree. We do things like this in other places too. Mainly, in generating temp tables during the run process (eg. It would be really good to differentiate between dbt-land and user-land in a consistent way |
@jthandy we can change this pretty easily, but we'd have to think through the migration process for existing dbt archive users |
This is great, definitely makes sense to namespace the dbt-added columns. I don't think it'd be too much to ask users to migrate to the new format, although I can see how this could scare some folks since this might be the only source of this archived data for them. Is there any way to gracefully use the old schema and column names if someone hasn't upgraded yet? E.g. if you don't detect |
@clausherther we're changing a whole lot around archival in the Wilt Chamberlain release, so I'm inclined to make it a clean break i think. Really good docs + migration instructions coupled with some intelligence in dbt will set us up for success here long-term I think! This is currently undocumented, but we added a command called
We can be smart about making a backup table, using a transaction, etc. I'd like to implement this in a way that will fail immediately if you try to use the new archive logic on an unmigrated archive table. All told, this scheme should help users leverage the new and compelling functionality coming to archival |
…e-flow Update release flow
Enhancement
When looking at the output schema from
dbt archive
, it's difficult to tell dbt-generated fields from other fields. Most tools solve this by appending a suffix to table/fields names. I think we should do the same:Archive column mapping:
When we make this change, we should also unquote these identifiers (see #1167)
Migration
Hoo boy. Unlike models, we don't have the luxury of asking users to drop and recreate their archives. Here's a general strategy for migration:
I don't believe that this migration code is sensible to bake into the materialization itself. Instead, we may be able to leverage
run-operation
s to implement database-aware migration code.The text was updated successfully, but these errors were encountered: