Fix bugs in allocate_gen_fuel

[grgmiller]

Overview

This PR fixes several issues that we identified in the analysis.allocate_gen_fuel module and had fixed in our fork of pudl for OGE. We are now trying to get rid of our dependency on the pudl code, so want to migrate all of our changes over to pudl so that we can directly use the output table from this module.

This is part of work we are tracking in OGE here: singularity-energy/open-grid-emissions#369

First, this addresses an issue where some retiring generators were incorrectly identified and being dropped: singularity-energy#1

This PR fixes a bug where during the generation and fuel allocation process, data for report months after the reported retirement date was getting dropped for generators that retire mid-year. For example, the retirement date of both generators at plant 50937 was "2022-09-01", and the previous behavior was to drop all data after september, even though this plant continued to report fuel consumption after september. This fix keeps all report dates through the end of the current year to avoid dropping this data.

Second, this addresses issues with duplicate generators, as described in this PR: singularity-energy#3

When running the pudl.analysis.allocate_gen_fuel pipeline for 2016 and 2017, we were getting a TypeError at group_duplicate_keys(), because this function was trying to groupby().sum() non-numeric columns like generator_retirement_date.
The group_duplicate_keys() will only work if we drop any datetime and boolean columns before using this function, and considering carefully whether we want to sum any of the frac columns or not. This PR, however, does not touch this function, but rather fixes the issue upstream.
We were only running into this issue with group_duplicate_keys() because there were duplicate keys in the dataframe, so this PR addresses the root cause of where duplicate keys were getting introduced in the first place.
It turns out that when creating the gen_assoc table with associate_generator_tables(), one of the steps is remove_inactive_generators(), which removes certain generators by creating six different dataframes with different generators based on their operating status: existing, retiring_generators, retired_plants, proposed_generators, proposed_plants, and unassociated_plants. These six dataframes are then concat'ed together. Previously our assumption was that these six dataframes should be non-overlapping. However, it turns out that this is not always the case.
For example, in 2016, plant 56846 generator GTG1 ended up in both proposed_generators and proposed_plants, which was causing it to be duplicated.
We fix this by simply adding .drop_duplicates() after these six dataframes are concat'ed together. This fixes the issue that we were experiencing in 2016 and 2017.
For now, we will leave group_duplicate_keys() alone even though it does not work. It effectively acts as an error if there are ever any duplicate keys since it will raise a typeerror like we saw for 2016 and 2017.

Testing

We have successfully run this after importing pudl and running it in the OGE pipeline. However, we had previously been testing this with an older release of pudl (v2023.12.01).

I have a Windows machine so there are not great instructions on getting the pudl dev environment set up on Windows. This is a pretty small amount of code change, so I'm hoping that someone with the dev environment already set up may be able to help test this.

To-do list

Preview Give feedback

1. If updating analyses or data processing functions: make sure to update or write data validation tests (e.g., test_minmax_rows())
2. Update the release notes: reference the PR and related issues.
3. Ensure docs build, unit & integration tests, and test coverage pass locally with make pytest-coverage (otherwise the merge queue may reject your PR)
4. Review the PR yourself and call out any questions or issues you have
5. For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
6. For significant ETL, data coverage or analysis changes, once make pytest-coverage passes, ensure the full ETL runs locally and run data validation tests using make pytest-validate (a ~10 hour run). If you can't run this locally, run the build-deploy-pudl GitHub Action (or ask someone with permissions to). Then, check the logs on the #pudl-deployments Slack channel or gs://builds.catalyst.coop.

[grgmiller]

@cmgosnell not sure who would be best to review this so added you for now.

[cmgosnell]

hey @grgmiller !! I'm so sorry that it's taken me absolutely forever to get to this PR!! I will commit to making sure we get this merged in quickly now that my attention is turned to it.

two intentions are:

1. make sure there are never duplicated generators
2. for retiring generators, make sure that there are additional months included to make sure any net gen or fuel reporting post-retirement is included in the allocation

For the first one, the simple drop dupes make sense np.

The desire for the second fix makes a lot of sense. But what I interpret from what you have done here doesn't exactly make sense to me. It looks like you are trying to include any records ever from a generator that says its retired but reported producing mwh. Which I believe means if a generator retired in January 2001 but reported MWh in February, all subsequent months of that generator would be included. That seems incorrect to me. I'd propose two different alternative solutions:

• add non-null fuel into the mask finding the retiring generator/months. Right now it is only selecting for non-full MWh.
• change the selection to make it include retiring records for the full year (or for some set number of future months) instead of in perpetuity.

If either of these options sounds okay to you I'm happy to try to implement and use your example plant 50937 as a test case to make sure we achieve your desired behavior.

[grgmiller]

Thanks @cmgosnell. I think what you proposed makes sense, and I think aligns with what I was thinking, although maybe I didn't explain clearly.

In my original example, I said:

For example, the retirement date of both generators at plant 50937 was "2022-09-01", and the previous behavior was to drop all data after september, even though this plant continued to report fuel consumption after september. This fix keeps all report dates through the end of the current year to avoid dropping this data.

In this case, by "current year" I meant the year in which it retired (2022), not the actual current year today (2024). So for this plant, if it retired in 2022-09-01, we'd only keep records through 2022-12-01 (which I think is consistent with what you were proposing?).

I think some of the confusion might be because in OGE we only run the pipeline a single year at a time, rather than for all years, so the code as I proposed, while compatible with our use case, might not work as intended when running all years.

I also think that adding non-null fuel (in addition to MWh) also makes sense.

[cmgosnell]

hey @grgmiller ! Thanks for your response! Yea i was wondering/fuzzily remembering that in OGE you are working with only one year. So yes I think I understand your need here and will tweak your pr a bit to enable retiring generator (status=retired but either burning fuel or making MWh) records to show up for every month in the report year which they are retiring.

[aesharpe]

Non-blocking, could update this to specifically call out the inclusion of mid-year retirement cases.