Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema cleanup #3529

Merged
merged 3 commits into from
Apr 2, 2024
Merged

Schema cleanup #3529

merged 3 commits into from
Apr 2, 2024

Conversation

zaneselvans
Copy link
Member

@zaneselvans zaneselvans commented Apr 2, 2024

Overview

Some miscellaneous column mapping cleanup, mostly for the entity-specific columns which get turned into operating_date values. Things I came across while looking into a better fix for #423.

  • Map the operating_year and operating_month columns in the EIA860M retired generators consistently -- these were getting silently lost because the other EIA860 generator operating date columns are name generator_operating_date.
  • Ensure all *_(operating|retired)_(year|month) columns have specific names in column mapping metadata CSVs, and remove the manual renames that were happening elsewhere to ensure that there aren't a bunch of different operating_date and retirement_date columns floating around in the transforms which actually apply to different entities. This change affected the operating month/year/date columns for cooling_system, emission_control, fgd, and fgp.
  • Put the convert_to_date() and month_year_to_date() helper functions next to each other in their module... they really need to be combined!
  • Rename obstacle_num_faa to obstacle_id_faa and update field description to provide more context on what those IDs refer to.
  • Rename core_eia860__yearly_generators_wind to core_eia860__scd_generators_wind, as it's not a data table, but a generator attribute table.
  • Add a new alembic migration to deal with the wind table and FAA column renames.

Testing

  • I ran the full ETL locally and didn't get any failures due to bad / missing column or table names.
  • I ran make pytest-validate on the fully populated database and it passed.
  • I ran make pytest-integration-full on the fully populated database and it passed.

To-do list

Preview Give feedback

@zaneselvans zaneselvans added the eia860 Anything having to do with EIA Form 860 label Apr 2, 2024
@zaneselvans zaneselvans requested a review from cmgosnell April 2, 2024 02:44
@zaneselvans zaneselvans self-assigned this Apr 2, 2024
@zaneselvans zaneselvans marked this pull request as ready for review April 2, 2024 02:44
Copy link
Member

@cmgosnell cmgosnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📜 🧼 👍🏻

@zaneselvans zaneselvans added this pull request to the merge queue Apr 2, 2024
Merged via the queue into main with commit 379feef Apr 2, 2024
12 checks passed
@zaneselvans zaneselvans deleted the schema-cleanup branch April 2, 2024 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
eia860 Anything having to do with EIA Form 860
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants