-
-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert epacems_to_parquet command to run dagster asset #2300
Conversation
…ger errors for missing tables
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know we're not merging until the tests are in, but figured I would approve this now so we can just merge when the time comes.
) | ||
if df.empty: | ||
raise AssertionError( | ||
f"The {table_name} table is empty. Materialize " |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah! I found the message you were talking about @bendnorman ! How helpful :)
Codecov ReportBase: 85.7% // Head: 85.7% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## dagster-asset-etl #2300 +/- ##
=================================================
Coverage 85.7% 85.7%
=================================================
Files 79 79
Lines 8906 8906
=================================================
+ Hits 7633 7637 +4
+ Misses 1273 1269 -4
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
PR Overview
The
epacems_to_parquet
command callspudl.etl.etl_epacems
which is no longer a thing in thedagster-asset-etl
branch. This PR replaces the call topudl.etl.etl_epacems
with the execution of a dagster job that just runs thehourly_emissions_epacems
asset. This command still expects theepacamd_eia
andplants_entity_eia
tables to be inpudl.sqlite
. Thepudl_sqlite_io_manager
will throw errors if the tables don't exist in the database.I'd like to hold on on merge this into
dagster-asset-etl
until we can get the CI passing in #2299 and merged into this branch.PR Checklist
dev
).