Get last non-null value instead of latest XBRL filing. #3545

jdangerx · 2024-04-05T19:45:49Z

Overview

Closes #3309.

What problem does this address?

A bunch of utilities sent diff-only reports to update their 2021 data, which led to some important XBRL tables being mostly-null.

What did you change?

After much debate and investigation, we decided to change the XBRL deduplication to take the latest non-null value (across multiple filings) instead of taking the value from the latest single filing.

Testing

How did you make sure this worked? How can a reviewer verify this?

@cmgosnell I'd love if you could take a look and see if this affects rate base table in the expected way.

To-do list

Give feedback

If updating analyses or data processing functions: make sure to update or write data validation tests (e.g., test_minmax_rows())
Update the release notes: reference the PR and related issues.
Ensure docs build, unit & integration tests, and test coverage pass locally with make pytest-coverage (otherwise the merge queue may reject your PR)
Review the PR yourself and call out any questions or issues you have
For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
For significant ETL, data coverage or analysis changes, once make pytest-coverage passes, ensure the full ETL runs locally and run data validation tests using make pytest-validate (a ~10 hour run). If you can't run this locally, run the build-deploy-pudl GitHub Action (or ask someone with permissions to). Then, check the logs on the #pudl-deployments Slack channel or gs://builds.catalyst.coop.
Options

cmgosnell · 2024-04-10T16:13:39Z

@jdangerx the few logs and/or metrics make this change seem pretty beneficial!

before your change

Log from corrections from core_ferc1__yearly_depreciation_by_function_sched219:

Correcting 95.02% of all non-null reported values (439/462) out of a total of 143284 original records.

out_ferc1__yearly_rate_base.row_type_xbrl.value_counts()
row_type_xbrl
reported_value             442993
calculated_value            10204
subdimension_correction       205
correction                     75

calculation_metrics = defs.load_asset_value("_core_ferc1__calculation_metric_checks")
errors = calculation_metrics[
    calculation_metrics.filter(like="is_error").any(axis=1)
]
len(errors)
> 36

post change

Log from corrections from core_ferc1__yearly_depreciation_by_function_sched219:

Correcting 32.94% of all non-null reported values (28/85) out of a total of 143284 original records.

out_ferc1__yearly_rate_base.row_type_xbrl.value_counts()
row_type_xbrl
reported_value             442993
calculated_value            10204
correction                    178
subdimension_correction        15

calculation_metrics = defs.load_asset_value("_core_ferc1__calculation_metric_checks")
errors = calculation_metrics[
    calculation_metrics.filter(like="is_error").any(axis=1)
]
len(errors)
> 24

some comparison

it looks like it fully cleaned up a bunch of utilities calced values in the core_ferc1__yearly_utility_plant_summary_sched200 table

newly_good_idx = (
    calculation_metrics[calculation_metrics.error_frequency != 0]
    .index.difference(calculation_metrics_new[calculation_metrics_new.error_frequency != 0].index)
)
calculation_metrics.loc[newly_good_idx].reset_index(level="group_value").index.value_counts()

> group             table_name                                       
> utility_id_ferc1  core_ferc1__yearly_utility_plant_summary_sched200    157

cmgosnell

i love how simple this change is. i have two very minor suggestions.

cmgosnell · 2024-04-10T17:36:06Z

src/pudl/io_managers.py

+        dupe_mask = original.duplicated(subset=xbrl_context_cols, keep=False)
+        duped_groups = original.loc[dupe_mask].groupby(
+            xbrl_context_cols, as_index=False
+        )
+        never_duped = original.loc[~dupe_mask]
+        return pd.concat([never_duped, duped_groups.last()], ignore_index=True)


omigosh this is so nice and simple!

Can you add dropna=True into the groupby (even though it is set that was as a default) just to make is more clear how that last is getting the non-null values?

cmgosnell · 2024-04-10T17:37:43Z

src/pudl/io_managers.py

+        This means that if a utility reports a non-null value, then later
+        either reports a null value for it or simply omits it from the report,
+        we keep the old non-null value, which may be erroneous. This appears to
+        be fairly rare, affecting < 0.005% of reported values.


can we add a little defensive check in the here to make sure that expected % of values stays semi-constant? I assume this rate is ~ table dependent but i assume the worst offenders % would be a good high bar mark to test against

Hm, this is a little tricky - we'd have to calculate both the best snapshot and the apply diffs methodologies, stack them all up, then compare the two - which we can do, but seems like a lot of complexity to add for this check.

Yea i didn't realize how complicated this request was going to be. I don't think what you have here is too too complicated but it is a lot more than was here before. I'd be inclined to keep it in but if you feel otherwise I think it would be fine to revert to not checking this.

jdangerx · 2024-04-10T22:14:11Z

src/pudl/io_managers.py

+        ):
+            """Compare deduplication methodologies.
+
+            By cross-referencing these we can make sure that the apply-diff


😓 this turned out to be surprisingly annoying

hm yeah it is surprisingly annoying!! I'm now semi regretting suggesting it but i think this looks okay. i like that it makes it clear that there are a few options and

This suggestion is probably oos for this pr but i wonder if this type of check that assesses methodological choices we've made in validation tests when it gets a lil too complicated. I've personally struggled with this distinction and the tension around testing with a lot of extra code that is not necessary for the actually thing you want to consistently produce. but i don't think this is tooo much!

jdangerx · 2024-04-15T18:54:57Z

cc @e-belfer

jdangerx · 2024-04-15T22:11:10Z

Oy, the integration tests failed with a bunch of error thresholds being broken:


ERROR test/integration/console_scripts_test.py::test_pudl_service_territories[inprocess-pudl_service_territories --entity-type balancing_authority -y 2022 --limit-by-state --no-dissolve -o -balancing_authority_geometry_limited.parquet-expected_cols0] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/console_scripts_test.py::test_pudl_service_territories[inprocess-pudl_service_territories --entity-type balancing_authority -y 2021 -y 2022 --no-dissolve -o -balancing_authority_geometry.parquet-expected_cols1] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/console_scripts_test.py::test_pudl_service_territories[inprocess-pudl_service_territories --entity-type utility -y 2022 --dissolve -o -utility_geometry_dissolved.parquet-expected_cols2] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/datasette_metadata_test.py::test_database_metadata - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_annual_eia_outputs[gen_fuel_by_generator_energy_source_owner_eia923] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_null_rows[mcoe_generators-0.9] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_outputs_by_table_suffix[eia861] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_outputs_by_table_suffix[ferc714] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_ferc714_outputs[out_ferc714__summarized_demand] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_ferc714_outputs[out_ferc714__respondents_with_fips] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_service_territory_outputs[out_eia861__yearly_balancing_authority_service_territory] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_service_territory_outputs[out_eia861__yearly_utility_service_territory] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_state_demand_outputs[out_ferc714__hourly_estimated_state_demand] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value                                                                                                                                                                                                                                                                                                                                                                                                            
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False

What seems a little strange is that the same dang dataframe is getting reported in every failing test - that seems surprising to me. I don't know quite what to make of it but I'll see if I can get this breaking locally for now. @zaneselvans this seems like something you might have some insight into?

zaneselvans · 2024-04-15T22:14:35Z

I think the reason the same dataframe is coming up repeatedly is that it's not a test failure, it's a defensive check which is erroring out in the ETL phase, and so you're getting identical "errors" raised in every test which depends on the fixture that's generating the errors.

jdangerx · 2024-04-15T22:15:36Z

Ahhh that makes sense. OK! I'll go look at where those checks are (presumably) hard coded and see if I can make heads or tails of it.

zaneselvans · 2024-04-15T22:17:51Z

I'd probably try to materialize the exploded tables locally, which I think would exercise this check, and then you could fiddle with the parameters that set the acceptable error levels. However, this could reveal real issues. @cmgosnell will be the most familiar with what the error levels are supposed to look like and why the new data might be setting off these alarm bells.

jdangerx · 2024-04-15T23:05:36Z

OK, so core_ferc1__yearly_depreciation_changes_sched219 is newly failing in the fast ETL, but not the full.

I compared the calculation reconciliation metrics across this branch & the main branch for the fast ETL:

metric	new value (apply-diffs)	old value (main)	threshold
error frequency	0.059571	0.05792	0.058
relative error magnitude	NaN	NaN	0.045
null calc value freq	0.014635	0.015244	0.7
null reported value freq	0.539281	0.560459	1.0

So the error frequency is newly a little higher, and the null calcs/reported values are both less frequent.

I'm not sure how to convince myself that this is OK vs. continuing to dig deeper - @cmgosnell maybe let's grab some time in the morning tomorrow to chat about this?

zaneselvans · 2024-04-15T23:32:47Z

If that's the only thing tripping it up, and it's only happening in the fast and not the full ETL, then upping the allowed error frequency to 0.06 seems fine. Those thresholds are arbitrary, and really intended to detect changes so they can be evaluated for significance / tracked down if they're wacked.

But I'm worried there might be a long series of little error tolerance tweaks that need to be made after this one is fixed.

… are never *that* far off from each other.

zaneselvans · 2024-04-16T02:09:05Z

I ran the fast ETL through the Dagster UI locally and thankfully core_ferc1__yearly_depreciation_changes_sched219 was the only asset that failed. I bumped the error frequency to 0.06 and re-ran it, and it passed. So I think we should just make that tiny adjustment to get this merged in.

@cmgosnell do you think we should check the other error metrics for the other FERC 1 assets, and ratchet them down if we have significantly lower error rates with the new diff-based extractions? Do we expect this to have broad impacts across the board? I can't remember if we had an easy way of generating all the error tolerance outputs.

@jdangerx I also noticed 2 failing asset checks in the fast ETL, which was a little surprising:

_core_eia860__fgd_equipment_fgd_equipment_null_check
_core_eia860__cooling_equipment_cooling_equipment_null_cols

If they're failing, should they be causing the fast ETL to fail as well?

cmgosnell · 2024-04-16T13:08:08Z

I'm glad y'all figured out how to tweak the error tolerances! ( i don't super love how it ended up, structurally/ergonomically )

It makes sense to me that more non-null values in data would result in at least slightly worse error metircs. It is very standard for the fast ETL to require higher error tolerances and that is not a big change. Believe it or not the XBRL data is less calculable with its own calculations as the old data. So it makes sense that adding more XBRL values -> more errors.

I was hoping to spend a lil time after this gets merged in to see if null frequency tolerance can be turned down a bit on any of the tables.

We do have a way to check all of the metrics all at once. that's this asset: _core_ferc1__calculation_metric_checks. That runs all of the calc checks on all of the tables that have non-null params in reconcile_table_calculations (i.e. are we checking their calcs). It outputs a table of all of the error metrics with standard tolerances in the table (it was a lil too complicated/deemed not worth it to use table-specific tolerances in this asset).

jdangerx · 2024-04-16T14:35:29Z

If they're failing, should they be causing the fast ETL to fail as well?

Yes... we have them set as "blocking" and also had a bunch of extra logic in there to stop the fast ETL from failing. Strange...

The change in how we select XBRL records to extract in PR #3545 also changed how many records appear in a few tables, but only by 1 or 2 records. If anything it's surprising how few records were added/lost, so I've just adopted the new row counts in these tables: * plant in service * small plants * all FERC 1 plants

jdangerx requested a review from cmgosnell April 5, 2024 19:45

jdangerx marked this pull request as ready for review April 9, 2024 14:21

jdangerx force-pushed the 3309-change-xbrl-dedup branch from fe95010 to d4807c4 Compare April 9, 2024 14:28

zaneselvans added ferc1 Anything having to do with FERC Form 1 xbrl Related to the FERC XBRL transition labels Apr 10, 2024

cmgosnell requested changes Apr 10, 2024

View reviewed changes

jdangerx force-pushed the 3309-change-xbrl-dedup branch from d4807c4 to 7f32384 Compare April 10, 2024 22:13

jdangerx commented Apr 10, 2024

View reviewed changes

jdangerx requested a review from cmgosnell April 10, 2024 22:14

cmgosnell approved these changes Apr 15, 2024

View reviewed changes

jdangerx force-pushed the 3309-change-xbrl-dedup branch from 7f32384 to 0f9e95b Compare April 15, 2024 20:58

jdangerx added this pull request to the merge queue Apr 15, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 15, 2024

jdangerx force-pushed the 3309-change-xbrl-dedup branch from 0f9e95b to dccd775 Compare April 15, 2024 23:14

jdangerx added 4 commits April 15, 2024 19:15

Get last non-null value instead of latest XBRL filing.

392b549

Actually remember to update release_notes.

f0d589e

Add a defensive check to make sure that best-snapshot and apply-diffs…

9913dd7

… are never *that* far off from each other.

Fix unit test failure

fd71f57

Adjust error tolerances slightly to accommodate new XBRL data.

f8f2d2b

zaneselvans force-pushed the 3309-change-xbrl-dedup branch from dccd775 to f8f2d2b Compare April 16, 2024 03:22

zaneselvans enabled auto-merge April 16, 2024 03:23

zaneselvans added this pull request to the merge queue Apr 16, 2024

Merged via the queue into main with commit b082be6 Apr 16, 2024
12 checks passed

zaneselvans deleted the 3309-change-xbrl-dedup branch April 16, 2024 04:43

zaneselvans assigned jdangerx Apr 16, 2024

This was referenced Apr 16, 2024

Nightly Build Failure 2024-04-16 #3573

Closed

Update expected row counts for FERC 1 tables #3574

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get last non-null value instead of latest XBRL filing. #3545

Get last non-null value instead of latest XBRL filing. #3545

jdangerx commented Apr 5, 2024

To-do list

cmgosnell commented Apr 10, 2024 •

edited

Loading

cmgosnell left a comment

cmgosnell Apr 10, 2024

cmgosnell Apr 10, 2024 •

edited

Loading

jdangerx Apr 10, 2024

cmgosnell Apr 15, 2024

jdangerx Apr 10, 2024 •

edited

Loading

cmgosnell Apr 15, 2024

jdangerx commented Apr 15, 2024

jdangerx commented Apr 15, 2024 •

edited

Loading

zaneselvans commented Apr 15, 2024

jdangerx commented Apr 15, 2024

zaneselvans commented Apr 15, 2024

jdangerx commented Apr 15, 2024 •

edited

Loading

zaneselvans commented Apr 15, 2024 •

edited

Loading

zaneselvans commented Apr 16, 2024 •

edited

Loading

cmgosnell commented Apr 16, 2024

jdangerx commented Apr 16, 2024

Get last non-null value instead of latest XBRL filing. #3545

Get last non-null value instead of latest XBRL filing. #3545

Conversation

jdangerx commented Apr 5, 2024

Overview

Testing

To-do list

cmgosnell commented Apr 10, 2024 • edited Loading

before your change

post change

some comparison

cmgosnell left a comment

Choose a reason for hiding this comment

cmgosnell Apr 10, 2024

Choose a reason for hiding this comment

cmgosnell Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

jdangerx Apr 10, 2024

Choose a reason for hiding this comment

cmgosnell Apr 15, 2024

Choose a reason for hiding this comment

jdangerx Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

cmgosnell Apr 15, 2024

Choose a reason for hiding this comment

jdangerx commented Apr 15, 2024

jdangerx commented Apr 15, 2024 • edited Loading

zaneselvans commented Apr 15, 2024

jdangerx commented Apr 15, 2024

zaneselvans commented Apr 15, 2024

jdangerx commented Apr 15, 2024 • edited Loading

zaneselvans commented Apr 15, 2024 • edited Loading

zaneselvans commented Apr 16, 2024 • edited Loading

cmgosnell commented Apr 16, 2024

jdangerx commented Apr 16, 2024

cmgosnell commented Apr 10, 2024 •

edited

Loading

cmgosnell Apr 10, 2024 •

edited

Loading

jdangerx Apr 10, 2024 •

edited

Loading

jdangerx commented Apr 15, 2024 •

edited

Loading

jdangerx commented Apr 15, 2024 •

edited

Loading

zaneselvans commented Apr 15, 2024 •

edited

Loading

zaneselvans commented Apr 16, 2024 •

edited

Loading