[v2.1] WeatherBench2 verification: eval CLI, scorecard plots & test suite

[jsschreck]

Summary

Adds full WeatherBench2-style deterministic verification tooling directly into CREDIT. This is a proper integration — the metrics module lives in credit/verification/ and the CLI tools are first-class applications/ scripts, not one-off analysis notebooks.

New CLI commands:

# Fast path — aggregate existing per-init CREDIT metrics CSVs
python applications/eval_weatherbench.py --csv /path/to/metrics/ --out scores.csv

# Full path — compute from forecast netCDFs vs ERA5 with true anomaly ACC
python applications/eval_weatherbench.py   --netcdf /path/to/forecasts/   --era5 /glade/campaign/cisl/aiml/ksha/CREDIT_data/ERA5_plevel_1deg/all_in_one/ERA5_plevel_1deg_6h_2022_conserve.zarr   --clim /glade/campaign/cisl/aiml/akn7/CREDIT_CESM/VERIF/ERA5_clim/ERA5_clim_1990_2019_6h_cesm_interp.nc   --out scores.csv   --plot figures/   --label "WXFormer v2"

# Plots only (from existing scores CSV)
python applications/plot_weatherbench.py --scores scores.csv --out figures/ --label "WXFormer v2"

What's included

File	Description
credit/verification/deterministic.py	Latitude-weighted RMSE, bias, true anomaly ACC with correct WB2 normalization
credit/verification/wb2_references.py	Published baseline scores: IFS HRES, Pangu-Weather, GraphCast
applications/eval_weatherbench.py	Evaluation CLI — CSV fast path + netCDF full path
applications/plot_weatherbench.py	Scorecard plots — RMSE, ACC, bias, heatmap, regional breakdown
tests/test_weatherbench.py	38 unit + integration tests, all passing

Figures generated

• RMSE vs lead time per variable with IFS HRES / Pangu-Weather / GraphCast reference lines
• ACC vs lead time — true anomaly ACC (requires climatology) not Pearson correlation
• Bias vs lead time — global mean bias (forecast − ERA5)
• Scorecard heatmap — skill score vs IFS HRES at days 1/2/3/5/7/10 (green = better than IFS)
• Regional RMSE breakdown — global / tropics / N. extratropics / S. extratropics

Notes

• Ensemble netCDFs are supported: ensemble mean is taken automatically before scoring
• acc_* columns from legacy per-init CSVs are Pearson r, renamed to pearson_r_* to avoid confusion with true WB2 ACC
• Reference baselines sourced from: Rasp et al. 2024 (WB2/JAMES), Bi et al. 2023 (Pangu/Nature), Lam et al. 2023 (GraphCast/Science)
• ERA5 pressure-level data at /glade/campaign/cisl/aiml/ksha/CREDIT_data/ERA5_plevel_1deg/ covers 2020–2022

Test plan

• python -m pytest tests/test_weatherbench.py — 38/38 pass
• Run full 2022 eval against scheduler netCDFs on casper — verified over the weekend
• Verify true ACC curves look sensible vs IFS HRES reference — verified over the weekend

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 60.97561% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.45%. Comparing base (fd24d56) to head (f7b888b).

Files with missing lines	Patch %	Lines
credit/verification/wb2_references.py	0.00%	28 Missing ⚠️
credit/verification/deterministic.py	78.49%	17 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #310      +/-   ##
==========================================
+ Coverage   16.13%   16.45%   +0.32%     
==========================================
  Files         122      125       +3     
  Lines       19604    19727     +123     
  Branches     3308     3326      +18     
==========================================
+ Hits         3164     3247      +83     
- Misses      16148    16185      +37     
- Partials      292      295       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.