Harvest generator operating dates when they're within a year of one another #3419
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
e-belfer
added
admin
zaneselvans
approved these changes
Feb 26, 2024
katie-lamb
pushed a commit
that referenced
this pull request
Mar 5, 2024
…nother (#3419) * Stash debugging process * Add _gen_operating_date method * Restore harvesting notebook * Actually implement special col case fixes * Clean up logs * Add to release notes * Fix some docstring cut-and-paste issues. * String / docstring cleanup. * Update EIA860m temporal coverage in README * Clean up docs and generalize, assert static col --------- Co-authored-by: Zane Selvans <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 6, 2024
* take out test set * add in model * [pre-commit.ci] auto fixes from pre-commit.com hooks For more information, see https://pre-commit.ci * debugging model * Update conda environment to include new splink. * [pre-commit.ci] auto fixes from pre-commit.com hooks For more information, see https://pre-commit.ci * working splink model * update splink version * add devtools notebook and fix fuel type filling * updates to devtools notebook * updates to notebook * Update conda lockfiles after merging main. * [pre-commit.ci] auto fixes from pre-commit.com hooks For more information, see https://pre-commit.ci * experiment with other blocking rules * add initial experiment tracking * add more metrics to experiment tracking * update blocking rules * add experiment tracking to old model to better compare * change blocking rules * change blocking rules * Harvest generator operating dates when they're within a year of one another (#3419) * Stash debugging process * Add _gen_operating_date method * Restore harvesting notebook * Actually implement special col case fixes * Clean up logs * Add to release notes * Fix some docstring cut-and-paste issues. * String / docstring cleanup. * Update EIA860m temporal coverage in README * Clean up docs and generalize, assert static col --------- Co-authored-by: Zane Selvans <[email protected]> * Add RMI beta access to parquet.catalyst.coop (#3434) * Add RMI beta access to builds.catalyst.coop * s/builds/parquet * Add new citations of Catalyst / PUDL (#3435) * Add new citations of Catalyst / PUDL. * Add issue/PR to harvesting bugfix release notes * Fix some capitalization in BibTex inputs. * Add book references, fix bad DOI formatting and capitalization. * Fix minor citation formatting issues. * Fix minor citation formatting issues. * dynamically generate fuel type list * remove old model * debug ferc to ferc * fix cleaning pipeline * take out test set * add in model * debugging model * [pre-commit.ci] auto fixes from pre-commit.com hooks For more information, see https://pre-commit.ci * working splink model * [pre-commit.ci] auto fixes from pre-commit.com hooks For more information, see https://pre-commit.ci * update splink version * add devtools notebook and fix fuel type filling * updates to devtools notebook * updates to notebook * experiment with other blocking rules * Update conda lockfiles after merging main. * add initial experiment tracking * add more metrics to experiment tracking * update blocking rules * add experiment tracking to old model to better compare * change blocking rules * change blocking rules * dynamically generate fuel type list * remove old model * debug ferc to ferc * fix cleaning pipeline * update release notes and add accuracy * update devtools notebook * update accuracy sig figs * clean up after rebase * update pyproject * add in accuracy metric * take out todo comments * update notebook with correct paths --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Zane Selvans <[email protected]> Co-authored-by: Katie Lamb <[email protected]> Co-authored-by: E. Belfer <[email protected]> Co-authored-by: Dazhong Xia <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Closes #3340.
What problem does this address?
When generator operating dates are within a year of each other, keep the last observed date in the year. Then try to harvest these dates again. This fixes 38 of the 59 generators reported without an operational date in #3340. The others still have inconsistent datetimes that should not be harvested.
What did you change?
Added
_gen_operating_date()method that mirrors_lat_long()method. Also actually fix the harvesting to apply changes made to special columns to the main harvesting process. This can be applied to other static entity columns.Testing
How did you make sure this worked? How can a reviewer verify this?
Run
debug_harvesting.ipynb. Look at the generators highlighted in the issue and confirm that those with dates within a year of one another have been harvested with the last date kept.To-do list