MVP of dbt-based data publishing framework #1505
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
atvaccaro
changed the title
1 similar comment
atvaccaro
changed the title
atvaccaro
requested review from
charlie-costanzo,
holly-g and
evansiroky
as code owners
lauriemerrell
suggested changes
Jun 2, 2022
There was a problem hiding this comment.
- thank you for writing all these docs, I know it is not your favorite 🙏
- this is cool and I am excited!
- mostly nit picky comments, I just want the docs to be as detailed as possible in case this sits for a bit and then we have to come back to it later..... I think my implicit standard is -- could someone who's not Andrew and can't talk to Andrew successfully publish based on the documentation here alone? (if you don't think that level is possible yet, I guess perhaps that will need to wait)
- ETA: can you update description? seems like linked Google doc is obsolete, so can probably remove. and info on any testing would be good. also -- I don't think this PR does close all those tickets? we still need to actually implement, right? this gives the framework / infra to publish but does not actually publish
|
Another request per meeting with Eric: can we update to give an option (maybe a modification of |
done! the bucket is a CLI option still, but there's separate dry-run and deploy concepts now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR implements a minimum-viable data publishing framework based on dbt exposures. Initially, our goal is to primarily support uploads to CKAN; that process is documented in this PR as well.
Once deployed, #284 and #427 can be closed
Type of change
How has this been tested?
I've tested metadata generation and GCS writing locally. However, we can't test publishing to CKAN until we actually go through the process, which we will be doing once this PR is merged.
Screenshots (optional)