Backfill historical lobbying data for trend analysis and revolving door tracking

[ryparker]

Summary

Currently we only collect lobbying data for the current Congress (2025–2026). The Senate LDA API has data going back to ~1999. Backfilling historical filing data would unlock several high-value features.

What this enables

• 15+ year spending trends per industry, client, and firm
• Historical revolving door tracking — officials who left office years ago and became lobbyists
• "Is this unusual?" comparisons — e.g. "Defense industry lobbying surged 45% this quarter vs. 5-year average"
• Year-over-year analysis on the map (time-lapse slider showing how lobbying patterns shift geographically)
• Deeper client/firm relationship mapping — long-term lobbying campaigns become visible
• Better contribution matching — historical officials available as matching candidates

Current state

Metric	Current	With backfill
Filing years	2025–2026	1999–2026
Filings	~110K	~500K+ estimated
Trend depth	None	25+ years
Revolving door candidates	~60 (unmatched)	Hundreds more

Scope

• Expand LDA API collection date range in the Lobbying collector task
• Backfill LD-1, LD-2, and LD-203 filings for prior Congresses
• Re-run contribution payee matching and revolving door matching against expanded dataset
• Refresh materialized views and industry aggregates
• Add time-range filtering to frontend queries

Considerations

• Storage and processing time for ~400K+ additional filings
• Incremental backfill strategy (newest first, working backwards)
• Some older data may have quality issues (incomplete fields, different formats)
• Frontend queries need to handle larger datasets efficiently (pagination, aggregation)