Limit EIA-861 years in the fast ETL

[zaneselvans]

Overview

For a long time we have processed all years of the EIA-861 data in both the full and fast ETL. Initially this was a hack to get around the fact that some of the tables / columns are discontinued and so result in entirely null columns if you just process the last year or two of data, and our old data validation tests couldn't accommodate that expectation. But with #4105 / #4382 that's no longer a problem.

This PR switches to processing just a couple of years of data for the EIA-861 in the Fast ETL like we do with all of the other datasets. This will speed up the fast ETL a bit, and reduce the amount of data required for the CI.

Closes #2628

Documentation

Make sure to update relevant aspects of the documentation:

• Update the release notes: reference the PR and related issues.

Testing

• Rematerialized all my EIA-861 assets locally in both full and fast tests.
• Ran dbt build --select "source:pudl.core_eia861*" --target etl-fast w/ fast outputs

To-do list

• Update the EIA-861 transforms to allow only a few years to be processed.
• Update the dbt expect_columns_not_all_null test parameters so they don't fail on just a few years of data.
• Refactor BA association table repair in the FERC-714 outputs to not rely on always having all years of data.
• Review the PR yourself and call out any questions or issues you have.

[zaneselvans]

@e-belfer The part of this I was already familiar with was a quick fix, but now in the FERC-714 Outputs I see that the BA association fixes depend on always having all years of EIA-861 data available, and they feel a little inscrutable to me. I don't know if any of that context is still rattling around inside your brain but if so, do you have any thoughts on how this might be refactored to work cleanly when there's only a subset of the EIA-861 years available?

[e-belfer]

@e-belfer The part of this I was already familiar with was a quick fix, but now in the FERC-714 Outputs I see that the BA association fixes depend on always having all years of EIA-861 data available, and they feel a little inscrutable to me. I don't know if any of that context is still rattling around inside your brain but if so, do you have any thoughts on how this might be refactored to work cleanly when there's only a subset of the EIA-861 years available?

I've never looked at this part of the code before, so I am equally as lost as you are! We could probably add more conditionals in there but it'd make it even jankier, so maybe that section of the code is wanting a more thorough refactor? I don't however have an immediate insight into the best way forward.

[zaneselvans]

@e-belfer Oh weird. Git was saying that all of this code came from you. But looking at the actual PR #2550 it looks like it was a mass cut-and-paste in the general Dagster re-organization.

The ASSOCIATIONS defining the fixes were created by Ethan, so maybe this is actually his code. So I guess we just have to figure it out. But it does seem pretty brittle / complex as it is.

[krivard]

Mostly minor nudges. The ferc714 bits are a real trip.

It's not clear to me where data_maturity should get added to empty data frames, but we probably shouldn't try to do it twice?

[krivard]

nit, because this function is already here, so we might as well use it: putting an extraction step in the transform module is a weird architectural choice

[zaneselvans]

I poked at trying to add data_maturity in the extract where it's being set for non-empty dataframes, and didn't see an ideal place -- since the code that does that work doesn't get run at all if the dataframe is going to end up empty. Instead I moved it into the _pre_process() function in this module, which means it doesn't need to be anywhere else in here. The only other changes now are to accommodate the possibility of having zero rows in division.

[krivard]

Doesn't _post_process handle this?

[zaneselvans]

Unfortunately, there's an explicit reference to the data_maturity field within the transform function, so it needs to happen before we get to _post_process.

I suspect the "right" place to ensure that data maturity is added (even in the case of empty dataframes) is in the EIA Excel extractor... Maybe I should look into that more deeply.

[krivard]

Consider adding an early exit here if eia861_years is empty

[krivard]

Consider defining a local version of ASSOCIATIONS to use both here and at line 209

[krivard]

would it make sense to skip iterations here when year is not in eia861_years, instead of filtering at 216?

[zaneselvans]

Anybody else looking at this -- the FERC-714 stuff doesn't actually work right now. I tried to make some little tweaks, but eventually came to understand that a bigger refactor of the spot-fix application (which deeply depends on having all years of data available) is necessary to make this work (and probably desirable just in general since it's so complex as it is)

[e-belfer]

The ASSOCIATIONS method originates from #618, and was created in #881. I'm planning on working backwards and converting these to manual spot-fixes targeting the values we want to include, rather than back or forward filling based on different years of the tables.