CONUS Leg B: species-free serializer (63) + loader SF-path tests#82
Closed
holoros wants to merge 45 commits into
Closed
CONUS Leg B: species-free serializer (63) + loader SF-path tests#82holoros wants to merge 45 commits into
holoros wants to merge 45 commits into
Conversation
Adds Greg Johnson CONUS gompit survival to the perseus projection path with a four-arm runner (default/calibrated/posthoc/iterative), closed-cohort toggle, and FVS TreeId-tracked TPA. Fixes project_mortality.spp_group to return an int group (was a str -> KeyError against cch_organon; the cycle_survival path had never run end to end). Validation (NE, 6 stands, 100yr) rejects naive substitution: disabling FVS mortality (MORTMULT 0) to let gompit own mortality makes FVS growth unrealistic (no competition mortality -> runaway density/biomass; NOAUTOES is a no-op for NE). See GOMPIT_INTEGRATION_FINDINGS.md for the diagnosis and the four design options. Do not launch a CONUS gompit campaign on this wiring as-is.
Model-level prediction check of Greg Johnson gompit survival vs per-species base rate on 7.6M FIA remeasurement records (133 species). gompit AUC 0.740 vs 0.673, log loss 0.420 vs 0.443, 131/133 species improved. Headline: gompit tracks observed survival down into the most crowded crown-closure quartile (0.779 pred vs 0.777 obs) where the base rate stays flat at 0.835 and misses crowding mortality. No FVS engine -> no density-feedback confound; this is the defensible mortality-model result, distinct from the not-yet-valid projection wiring. Adds R figure script, PNG, summary md, and the source comparison CSVs.
Substitutes Greg Johnson gompit survival for FVS native mortality INSIDE the growth loop (TREGRO->MORTS), so growth, density, and the substituted mortality interact each cycle. Resolves the density-feedback failure of the post-hoc TPA overlay. New: base/gompmort.f90 (GOMPLOAD coefficient reader + species resolve via FIAJSP; GOMPCCH ORGANON crown-closure-at-tip port, affine-mapped; GOMPSURV period survival) and common/GOMPMC.f90 shared state. Hooked into vls/morts.f90 (shared by NE/CS/LS): per-tree gompit override for fitted species, native background for unfit, VARMRT density redistribution bypassed. Env-gated (FVS_GOMPIT, FVS_GOMPIT_COEF) so one binary does native or gompit. NE validation (8 stands, 100yr): gompit 147 t/ac vs native 173 at yr100 -- bounded, realistic, no runaway (cf. post-hoc 923, iterative crash). Gompit kills less in open stands, more in crowded stands: the crown-closure signal, dynamic.
CS and LS share vls/morts.f90, so adding base/gompmort.f90 to their source lists was sufficient -- both build and validate with no further code changes (gompit module is variant-agnostic). Mean AGB at yr100 (native -> gompit): NE 176->161 (-8%), CS 166->124 (-26%), LS 156->94 (-40%). All bounded and realistic, no runaway; gompit trims late-rotation stocking as crown closure rises, variant-specific in magnitude. Adds 3-panel figure + per-variant CSVs.
Proves the gompit hook generalises beyond the shared vls family: SN uses its own sn/morts.f90 and got the identical pattern (include, GOMPCCH, per-tree override, VARMRT bypass, GOMPLOAD in MORCON), builds and runs bounded. Scaled A/B (NE/CS/LS ~200 stands each, SN 10), mean AGB yr100 native->gompit: NE 63->49 (-21%), CS 107->84 (-21%), LS 95->56 (-41%), SN 56->12 (-78%). All bounded, no runaway/crash. 4-panel figure updated. SN -78% flagged: small sample + the coarse FIA->ORGANON group proxy fitting southern pines/oaks poorly. Documents two items needing Aaron (not autopiloted): the GOMPMORT keyword (shared keyword-table build-break risk) and group-map refinement (would require re-fitting the affine cch map, 35d_validate_cch.R).
…alidated Western variants share the Dixon/VARMRT mortality structure; a patcher applied the identical hook to cr/ws/ec/ca morts.f90. All built and validated, bounded, no runaway. yr100 native->gompit: EC -5%, CA -15%, CR -35%, WS -44%. EC (-5%, PNW conifer) is the best ORGANON-proxy fit and nearly matches native; SN (-78%, southern species) the worst -- the effect size tracks how well the coarse FIA->ORGANON group map suits each variant, direct evidence for the group-map refinement (needs affine re-validation, Aaron's loop). Coverage now 8 variants across 4 mortality-routine families (NE/CS/LS shared vls; SN own; CR/WS/EC/CA own western). Adds 2-panel all-variant figure (trajectories + yr100 percent-change summary).
WC and PN share vwc/morts.f90, an ORGANON-based logistic mortality structurally different from Dixon/VARMRT (annual RIP from per-species logistic, own SDI-max feedback, no VARMRT). Adapted hook: for fitted species replace the annual RIP with the gompit annual rate 1-exp(-H) (GOMPSURV T=1); the routine's existing period conversion then reproduces gompit period survival exactly. Validated, bounded: PN 328->190 (-42%), WC 204->89 (-56%). Dense wet Westside conifer (high crown closure) -> large gompit effect, vs dry open EC (-5%); effect size tracks both proxy fit and stand density, exactly what cch captures. Coverage now 10 variants across 5 mortality-routine families. All-variant figure updated (10 panels + sorted yr100 summary).
Universal patchers applied the gompit hook to all remaining variants whose morts.f90 has the standard rate line (acd,bm,nc,oc,op,so,ut,em,tt,on) or the WK2(I)=WKI assignment (ak,ci,ie,kt). All morts compile; 23/25 executables build (on fails an unrelated pre-existing fire-module link error, fmcrow_; its hook compiles fine). bc deferred -- different control flow broke insertion, needs a manual hook. 18 variants validated native-vs-gompit, all bounded, no runaway/crash. yr100 change sorted: SN -78, CI -75, WC -56, WS -44, EM -43, PN -42, LS -41, AK -38, CR -35, IE -24, NE -21, CS -21, CA -15, BM -9, EC -5, SO -0, NC +0. NC/SO/EC (western conifer) near zero = best ORGANON-proxy fit; SN/CI largest. 17-variant figure updated (UT excluded, degenerate 0-biomass sample).
Single reference: the model, the code (GOMPMC.f90, gompmort.f90, the 5-edit morts hook), env activation, coverage table (24/25 hooked across all 5 mortality-routine families, 23 built, 18 validated with results), reproduction steps, the two owner decisions (group-map refinement needs affine re-fit; GOMPMORT keyword deferred for build-break risk), next steps, and the artifact index.
…-builders Validated TT (native 3.6 -> gompit 0.9, sparse subalpine, bounded). 19 variants now validated. Confirmed BC and ON fail to build even unmodified (BC: dbsrd1_/dminit_; ON: fmcrow_), so gompit covers all 23 buildable variants; BC kept pristine (nonblock-DO control flow hostile to the hook AND unbuildable). OC/OP/KT/ACD remain built+hooked but have no stress stands (their FIA plots are labelled under other variants). Handoff + all-variant figure updated.
Activation now also via a keyfile keyword (reproducible), not just FVS_GOMPIT env. Implemented at the only safe insertion point in the legacy dispatcher: keywds.f90 TABLE slot 148 (previously blank, no index shift), a new GOTO target (14800) + handler in vbase/initre.f90 that calls GOMPON, which flags LGOMPKW. GOMPLOAD (MORCON, after the keyword reader) then activates on (env OR keyword). BLOCK DATA GOMPBD initialises LGOMP/LGOMPKW/GHAVE (.AND. is not short-circuit in the morts hook). Coeff path still from FVS_GOMPIT_COEF (deployment detail). Validated on NE (5 stands, yr100 AGB): native 211.1, env-gompit 175.5, keyword-gompit 175.5 -- keyword is byte-identical to the env path and both differ from native. NE builds and links cleanly with the shared base edits.
…cally best) Built a genus/crown-form crosswalk over all 18 ORGANON SWO groups and re-fit the affine cch map on the held validation sample (113k trees, 4000 plots). Both the full genus crosswalk (Spearman 0.853) and a conifer-only refinement (0.875) DEGRADE the cch reproduction vs the coarse softwood/hardwood proxy (0.925). The PNW-specific ORGANON crown equations add species variance that does not match how the stored CCH1 was generated; uniformity wins for a rank-order proxy. Conclusion: keep the validated coarse proxy unchanged. The variant spread in gompit effect is a real cch-sensitivity property, not a crosswalk artifact -- which justifies the simple proxy. Resolves the flagged group-map item on evidence.
Broad robustness test of the validated in-engine gompit system: 19 variants across all 5 mortality-routine families x ~60 stands x native/gompit (~45,000 projection-years). ZERO gompit rows NaN/inf/negative/>2000 t/ac; no crashes, no runaway. Max gompit AGB 1122 t/ac (NC dense redwood, physical). Effects NC -4% to UT -82%, productive variants -13% to -27%. Updates handoff: both flagged items resolved (group-map refinement tested and rejected on evidence -- coarse proxy empirically best; GOMPMORT keyword implemented and validated). The gompit Fortran integration is complete, robust, keyword-activatable, and documented.
Skip a (variant,batch) task if its output csv+ledger already exist, so the full 381-task FIADB array can be resubmitted to rerun only timed-out/failed tasks without redoing completed batches. Launched the full default-vs-calibrated FIADB campaign (job array 0-380, 4h wall) over ~1.8M stands.
Grounded scope of the next phase: three FVS engines (default/calibrated/gompit) into the perseus_api_v1 state-series model; disturbance+harvest scenarios from the p_disturbance rasters -> statewide estimates; and TreeMap2022 spatially-explicit. Key feasibility result: TreeMap stores a donor FIA-plot ID per 30m pixel (tm_id_plain.tif), so spatially-explicit projection reuses the running FIADB per-plot compute almost for free via a raster join -- not a billion-run campaign. Cost-ordered sequence + the harvest/disturbance/climate/ TreeMap-scope decisions that need Aaron.
…l v4) Deep Cardinal search found the harvest model fully built: conus_hcs Harvest Choice System, CV-validated CONUS v4, conditioned on price/ownership/region/rd, delivered as 240m annual-probability + intensity rasters (p_partial, p_clearcut, p_stand_replacement, intensity, expected_ba_removed, value_at_risk, svi) + a stumpage panel. Harvest becomes a coupling task, not a modelling task. Decisions resolved: harvest = couple conus_hcs; climate static; TreeMap ME pilot first.
…eline Recon found the PERSEUS engine is the empirical yield-curve engine (ycx) with owner-specific harvest scenarios, TreeMap spatial expansion, and the FIADB-vs-TreeMap comparison ALREADY built. FVS (default/calibrated/gompit) plugs in as a drop-in growth engine: per-plot AGB trajectories replace the Chapman-Richards curves, then reuse the same expansion/scenario/TreeMap/merge. Confirmed join TreeMap TM_ID -> PLT_CN = campaign STAND_CN. Both tracks are integration, not greenfield; gating input is the running FVS campaign.
Produces ycx_<ST>_state_series.csv (metric,mgmt,year,value,value_lo,value_hi, native per-ha density) from the CONUS campaign csvs, the exact schema ycx_merge_perseus.py reads. Engines fvs_default/calibrated/gompit; agb_dry + agc_live_total (x0.47); calendar = 2025+PROJ_YEAR; reserve(no-harvest) bucket; 10/90 percentile bands. Validated on partial campaign output (18 states, 83k plots, 2025-2125; OR agb_dry 30->382 Mg/ha). Full coverage + merge + ME pilot follow as the campaign fills in.
Clips TreeMap2022_CONUS to Maine, strips the RAT to raw TM_ID, joins the VAT (TM_ID->PLT_CN) -> me_treemap_donors.csv (TM_ID, n_pix, PLT_CN, area_ha). Result: 3578 unique donor FIA plots cover 6.78M forested ha of ME (75.3M 30m pixels). Decouples the ME pilot from the CONUS campaign: only these 3578 plots need FVS.
…mpare Stage B (me_donor_fvs.py): runs FVS default+calibrated on the 3578 Maine donor plots (100% matched in the CONUS standinit) -> me_donor_trajectories.csv. Stage C (me_treemap_stageC_compare.py): joins trajectories to donor pixel areas and aggregates Maine AGB/AGC two ways -- TreeMap spatial (sum density x pixel area) vs FIADB uniform (mean density x total area) -- isolating the area expansion choice with FVS as the growth engine; outputs comparison CSV + figure. Decoupled from the CONUS campaign; Stage B running now.
Stage C now reads comma-separated trajectory CSVs so the Maine comparison shows all three engines (default, calibrated, gompit) x (TreeMap spatial vs FIADB uniform). Gompit arm launched on the same 3578 ME donor plots via the gompit lib + FVS_GOMPIT (default growth, gompit mortality), relabeled CONFIG=gompit.
The donor plots drew empty tree lists from FIA_fresh/treeinit (which does not cover the TreeMap-2022 donor CNs -> AGB ~0). Switch to TreeMap's own tree table (TreeMap2022_CONUS_Tree_Table.csv) -- the imputed donor tree list, the correct self-consistent source for a TreeMap projection. me_donor_extract.py filters trees+standinit to the ME donors; me_donor_fvs.py builds treeinit from the TreeMap trees (SPCD/DIA/HT/CR/TPA_UNADJ).
Fixed Tg unit (short ton=9.07e-7 Tg, was 1000x off; ratios were always correct). First spatially-explicit FVS result over 3578 ME donor plots (TreeMap tree lists), 100yr no-harvest: - TreeMap-spatial ME AGB 17.5% above FIADB-uniform at 2025 (894 vs 761 Tg; 132 vs 112 Mg/ha); ratio converges to ~1.0 by 2125 as stands mature. - Engines diverge: default over-accumulates (4117 Tg), calibrated moderates (3665), gompit plateaus then declines (peak ~2771 ~2090 -> 2667) -- the national mortality gives the most realistic late-succession ceiling. Pipeline is a raster join over per-plot FVS compute; scales to CONUS. Adds results md + figure + csv.
…ines Mirrors ycx_merge_perseus.py. Reads per-state aggregated densities (perseus_series_<config>) + n_plot counts, expands to Tg via a fixed FIA-anchored area model, injects three national models under cls FVS (reserve no-harvest), with incremental, non-destructive states/meta bookkeeping. Default+calibrated merged; gompit added once its array lands.
FVS_TREEINIT_PLOT TREE_COUNT under-expands overstory ~6.5x (median 1.0 vs FIA TPA_UNADJ ~5.85), concentrated in eastern variants, making campaign biomass ~6x light and unpublishable. treeinit_fix_tpa.py rejoins each row via TREE_CN to the raw FIA TREE table and restores TPA_UNADJ; engine unchanged. CONUS mean BA 48.8->91.3 ft2/ac. Campaign relaunched against treeinit_fixed (out_fvs_v2 / out_gompit_v2).
FVS reports the inventory treelist before imputing missing tree heights; NSBE biomass needs height, so the 2025 point is systematically understated (50-90%, by missing-height fraction) and jumps by 2030. Anchor + report the national FVS series from 2030 (first complete-height year). All FIA-anchored states now reconcile exactly to fia.json carbon at 2030; trajectories bounded and engine-ordered (gompit<=calibrated<=default in late succession).
Per-state annual removal rates sampled at FIA plot locations from the conus_hcs expected-BA-removed rasters + annualized p_disturbance (build_state_harvest_rates.R). fvs_managed_scenario.py applies them to the reserve trajectories; merge injects a managed (harvest) bucket + harvest_c_yr for all three engines. Live on dashboard.
Scales the ME pilot to CONUS via the TreeMap VAT (pixel Count per PLT_CN -> per-plot area, 241.9 Mha, no raster scan). FVS 2030 carbon on TreeMap area (9284 TgC) cross-validates to within 9% of TreeMap's own imputed carbon. Area-expansion choice (TreeMap-spatial vs FIADB-uniform) is negligible at CONUS (~2%) but diverges at finer scales (state 0.36-1.39, forest type 0.23-1.96): the scale-dependence Aaron asked about.
… steps treeinit_fix_v2.py fills tree heights (FIA TREE.HT + per-species H-D impute) so the 2025 inventory biomass is whole; campaign rerun launched (out_*_v3). fvs_strata_trends.py aggregates carbon by owner/ecoregion/state across reserve+managed with bootstrap CI. Adds refinements/next-steps doc.
…esign Height rerun (treeinit_h, 100% heights) is a negative result: the 2025 PY0 understatement is FVS reporting cycle-0 before its H-D model runs, not input heights (v3 ~3-4% above v2 but the jump persists), so the 2030 anchor stands. Adds the concrete custom-config draw-propagation design for the posterior CI.
…ed on ME Propagates posterior draws through FVS via UncertaintyEngine.generate_keywords_ for_draw, monkeypatching the calibrated keyword path (no production change). Single-draw SLURM-array design. ME (ne, 30 draws x 60 plots): parameter CI is 1-5% wide vs 30-60% structural engine spread -> the mortality model choice, not the calibrated parameters, dominates projection uncertainty.
…live Intensive harvest confined to FIADB plantation plots (STDORGCD=1, 10.2% of CONUS forest); extensive on natural stands; calibrated so the plantation- weighted blend reproduces the sampled conus_hcs rate. Merge injects four scenario buckets per engine. On dashboard.
…pipeline fvs_dashboard_stress.py validates all invariants across 49 states x 3 engines x 4 scenarios. Found+fixed the NV degenerate band (clamp to contain value); confirmed gompit late-succession decline is intended and the calibrated engine differs from default everywhere (13-32% lower). run_fvs_perseus_pipeline.sh folds the treeinit TPA+height fix in at stage 1 for reproducible campaigns.
Single reference: what is live, the raw-FIA->dashboard pipeline, key findings (treeinit bug, gompit capping, plantation scenarios, uncertainty hierarchy, FIADB-vs-TreeMap scales), tooling inventory, Cardinal locations, reproduce steps, and remaining roadmap. Dashboard now on height-corrected v3 canonical.
Published the CONUS FVS PERSEUS projection dataset (v1.0.0, 10.5281/zenodo.20555667; concept 20555666), 21 files, CC-BY-4.0, cross-linked to the fvs-modern software DOI.
…onservation) The FVS-only managed (extensive) bucket was orphaned vs the dashboard meta.buckets taxonomy. Emit it as managed (conservation) so the FVS curves are selectable alongside other engines. Live; stress test + API integrity pass.
…neration FVS engines are direct-injected to public/api/series (not from the master DB pipeline); they survive regenerations (additive) but a full rebuild needs a re-run of pipeline stage 7. Both gates (fvs_dashboard_stress + check_api_ integrity) must pass. Parameter CI now covers 19 calibrated states.
fvs_strata_trends.py now breaks carbon trends by owner/ecoregion/state/ forest_type x scenario with bootstrap CI. Forest-type totals (default reserve 2125): Oak/hickory ~7790, Maple/beech/birch ~3820, Oak/pine ~2040, Spruce/fir ~1880 Tg C. strata_trends_v3_default.csv committed.
Shareable capstone: exec summary, live dashboard, key findings, pipeline, deliverables, DB-pipeline coexistence, roadmap. Generated via docx-js.
…mparison hub, 2026-06-26)
63_conus_sf_to_variant_json.R lands CONUS species-free (trait-driven) component fits into categories_conus_sf per variant, in the exact shape config_loader.get_conus_sf_runtime_block decodes. Recomputes trait standardization from species_traits.rds to match the fitting script and stores scale_mean/scale_sd so out-of-bundle species standardize at runtime. Builds the Leg A/Leg B hybrid source map from existing categories_conus coverage. test_conus_sf_loader.py verifies the loader SF path end to end against a fixture shaped like the serializer output: block decode, in/out-of-bundle trait effect, species-free linear predictor, and conus_hybrid resolution. 4/4 pass.
holoros
marked this pull request as draft
July 2, 2026 17:06
Collaborator
Author
|
Converting to draft. Recon of Cardinal (sf_integration_dev/) shows the species-free Leg B pipeline is already built and fit for all six components, with an existing 62b serializer and a completed benchmark, plus an explicit hold in README_sf_integration_dev.md (do not land production configs or merge until reviewed). This branch's 63 serializer + 5 species-free Stan models overlap that work; holding for reconciliation with 62b rather than merging in parallel. |
Collaborator
Author
|
Closing as superseded. The canonical species-free pipeline is 62b_speciesfree_to_variant_json.R (in sf_integration_dev), which produced the landed configs in PR #83. This branch's 63 serializer + 5 drop-the-RE Stan models duplicate that work. Recommendation captured in the session notes: promote 62b (and its test_sf_loader.py) from sf_integration_dev into calibration/R so the canonical serializer is version-controlled in the repo proper. No need to carry two parallel serializers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the species-independent (trait-driven) component fits into categories_conus_sf per variant, matching the shape config_loader.get_conus_sf_runtime_block decodes. Adds 63_conus_sf_to_variant_json.R (extract+serialize in one pass, recomputes trait standardization from species_traits.rds, builds the Leg A/Leg B hybrid source map) and config/tests/test_conus_sf_loader.py (4/4 passing). Completes the Option B schema: Leg A was already populated across 25 variants; this fills the Leg B gap the loader already supported. DG lands now (pilot fit exists); HG/HtDBH/HCB/mortality/crown-recession land as those species-free fits complete.