Skip to content

Add oldest data (1991-1995) back to FERC Form 2 archiver - #1284

Open
e-belfer wants to merge 2 commits into
mainfrom
update-ferc2-data
Open

Add oldest data (1991-1995) back to FERC Form 2 archiver#1284
e-belfer wants to merge 2 commits into
mainfrom
update-ferc2-data

Conversation

@e-belfer

@e-belfer e-belfer commented Aug 19, 2026

Copy link
Copy Markdown
Member

Overview

Closes #1259

What problem does this address?

  • New FERC Online archive doesn't contain oldest FERC Form 2 data.

What did you change in this PR?

  • Add back the old archiving method, just for the oldest years of data.

Question for reviewer

  • Do we want to add the parts = all partition back for newer DBF and XBRL data?

Testing

How did you make sure this worked? How can a reviewer verify this?
Run the archiver and verify the oldest data is not deleted: https://github.com/catalyst-cooperative/pudl-archiver/actions/runs/32282792533

To-do list

  • Update relevant documentation - like comments, docstrings, README, release notes, etc.
  • Review the PR yourself and call out any questions or issues you have

@e-belfer e-belfer self-assigned this Aug 19, 2026
@e-belfer e-belfer added bug Something isn't working ferc2 FERC Form 2 labels Aug 19, 2026
@codecov-commenter

codecov-commenter commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 40.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.9%. Comparing base (700e1b3) to head (209fb55).

Files with missing lines Patch % Lines
src/pudl_archiver/archivers/ferc/ferc2.py 40.0% 12 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main   #1284     +/-   ##
=======================================
- Coverage   60.0%   59.9%   -0.2%     
=======================================
  Files         67      67             
  Lines       3411    3430     +19     
=======================================
+ Hits        2049    2056      +7     
- Misses      1362    1374     +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@e-belfer
e-belfer requested a review from zschira August 19, 2026 18:15
@e-belfer e-belfer mentioned this pull request Aug 19, 2026
2 tasks
@cmgosnell cmgosnell moved this from New to In review in Catalyst Megaproject Aug 26, 2026
@cmgosnell cmgosnell mentioned this pull request Aug 28, 2026
6 tasks

@zschira zschira left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I had just been meaning to do a test run locally before approving, which I've done now and seems to be working fine

@e-belfer

e-belfer commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

@zschira any opinions about this?

Do we want to add the parts = all partition back for newer DBF and XBRL data?

@zschira

zschira commented Sep 1, 2026

Copy link
Copy Markdown
Member

Do we want to add the parts = all partition back for newer DBF and XBRL data?

I think it would make sense to add in for the DBF data since that is a meaningful distinction there, but since XBRL data isn't split into multiple partitions like this, it's probably fine to leave out there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ferc2 FERC Form 2

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

FERC 2 1991-1995 no longer available

4 participants