Skip to content

Latest commit

 

History

History
141 lines (111 loc) · 17.9 KB

File metadata and controls

141 lines (111 loc) · 17.9 KB

Research & Engineering Decision Log — P1–P4 Continuation Session

Period: 2026-08-22 → 2026-08-23 · Final state: master (see git log for per-change hashes) Purpose: record for scientific review every material decision, encountered issue, diagnosis, and solution from the session that (a) incorporated the Proposal-1 Part-1 high-fidelity dataset, (b) executed the Proposal-2 spatial damage-field MVP iteration, (c) unblocked Proposal-3's mode-identity feasibility study, and (d) restored and completed Proposal-4's training pipeline.


1. Architecture and process decisions

# Decision Rationale Consequence / risk accepted
D1 Baseline commit + feature branches instead of a separate git worktree Identical restore semantics with one working tree shared by subagents, LSP and test runs; a second worktree would duplicate .venv and multi-GB datasets Working tree is the shared mutation surface; mitigated by file-ownership partitioning between parallel agents
D2 P1 dataset: five small CSVs committed under data/p1_part1/ via .gitignore exception block; ~2 GB bulk mode-shape corpora excluded Bulk corpora are regenerable; pre-commit hook blocks binary-ish artifacts; small CSVs anchor reproducibility Consumers need the bulk corpora on disk for shape-level work; documented in data/p1_part1/MANIFEST.md
D3 Recorded Part-1 CSVs/grids are the ground truth of record; the repo FSDT solver is a cross-check path only Solver numerics changed across sessions; recorded data must not silently drift with solver edits hf_dataset.solve_fsdt_modes docstring forbids use as a data source
D4 Canonical flat dataset layout (loaders reject missing keys; no Downloads-style numbered subdirectories) Machine-independent; explicit schema validation One-time copy step from the download tree, documented in the MANIFEST
D5 P1 learning stack: upgrade src/mechanics/p1_multifidelity/ in place rather than create the roadmap's hypothetical src/multifidelity/ The roadmap's path had zero references in code or tests; the existing package already owned the import surface pinned by tests Roadmap file-structure section is now aspirational; deviation noted in reports
D6 NPZ/CSV persistence instead of zarr (both P1 and P2) No existing zarr dependency; data loads fine file-at-a-time; repo norms favor npz+JSON Recorded as a deliberate roadmap deviation in P2_MVP_REPORT.md and the roadmap amendment
D7 Split-by-run (design-level) enforcement everywhere data touches learning Charter rule preventing pixel-level leakage between train/val/test Enforced in real_pipeline._split_run_indices, field_pipeline.load_field_dataset (overlapping splits rejected), and P4's design-level splits
D8 First-pass field encoder kept as PCA/MLP (no conv autoencoder rewrite) Measure before building; the roadmap gates architecture upgrades on observed reconstruction failure Lever table in P2_MVP_REPORT.md shows where a conv decoder would/would not help
D9 When spatial calibration failed, ship the scalar frequency-error GP as the P1 deliverable instead of forcing the field model Charter stop condition: "MVP + one upgrade attempt"; the scalar result (LOO RMSE 5.35% vs 9.45% baseline) is a real, verified deliverable Field-level calibration documented as open, with precise failure geometry
D10 CVAE retained over the roadmap's cINN; SP4 redefined as sample-based SBC + coverage cINN was never implemented; the substitution predates this session; amending the roadmap is honest, silently diverging is not Recorded as "Amendment 2026-08-22" in roadmaps/proposal2_roadmap.md
D11 Solver spatial damage: uniform 8×8 ABDAs retention scaling + per-cell Gauss-Legendre quadrature, opt-in Analytic _expanded integrals cannot express spatial variation; quadrature path is exact at ones-field (8.7e-15) v1 simplification: retention scales all stiffness blocks uniformly (no per-block physics); documented in the assembly docstring
D12 Mode-shape canonicalization at emission (sign-fix + max-norm) rather than at consumption Eigensolver sign/scale ambiguity must not leak into every downstream consumer; N2's ridge probe proved canonicalized shapes carry the localization signal (0.0248 MSE < 0.030 bar) Amplitude information is discarded on the full-shapes path; the summaries channel retains [RMS, max
D13 Noise gate: modes with transverse peak < 1e-4 of the sample's strongest mode stored as zero channels Audit measurement: 6th stored mode was pure eigensolver noise in 203/300 samples; max-norming amplified dust to O(1) CNN inputs Fixed channel count preserved for batching; test updated to pin zeros-or-ones semantics
D14 Phase-1/Phase-2 asymmetry: AE pre-training feeds real measurement-derived c to the decoder even when cond_decoder=False; de-conditioning applies only in Phase-2 ELBO With de-conditioning in Phase 1 the CNN encoder receives no gradient at all (linear probe R² = −0.26); Phase-2 de-conditioning is what forces information through z Documented asymmetry; inference consumers match Phase-2 exactly (verified by audit)
D15 SBC calibration via leave-one-out conformal variance multipliers fitted on the validation split only Pixel-independent σ understated the spatially-correlated joint spread ~16× (rank U-shape); LOO keeps every rank honest w.r.t. its own residual Multipliers act as one near-global scale on the same val rows; independent verifier ran a split-half control (median p = 0.30 vs 0.47 null) confirming generalization; raw uncalibrated p retained as sbc_pvalue_uncalibrated
D16 SBC gate requires both χ² p > 0.05 AND error < 0.10 The error statistic alone passed a p = 2.6e-18 histogram Three earlier "passing" arms reclassified as fails
D17 Post-hoc conformal recalibration accepted as the deployed calibration mechanism Conformal prediction is a principled, standard methodology; the alternative (making the raw σ head self-calibrate) requires epistemic-uncertainty machinery that is a new arc Disclosed via sbc_pvalue_uncalibrated; verifier notes the SP4 pass "rides on" the recalibration
D18 Verification methodology: parallel dual-lens review (code + physics/ML) before every merge, independent replication of headline numbers, physics probes with hand-computable known answers Caught three confounds in my own headline result that single-lens review would have missed (see I7–I9) Reviewer lanes were infra-fragile; two lanes occasionally duplicated work — accepted as the price of redundancy
D20 HeteroscedasticFieldDecoder.trunk_norm affine parameters explicitly frozen (requires_grad_(False) at construction) Robust-lane audit found a live I3-class instance: the LayerNorm sits on both the deployed forward path and the sigma-calibration path but was in no optimizer list — its γ/β silently never trained. Freeze keeps numerics identical to all reported results while making the disposition explicit; per-feature scale adaptation would be a new measured lever If adaptation is ever needed it must be introduced deliberately, not by silently adding params to an optimizer

I19. Gradient-flow regression gate

The I3 class (module on the forward path, absent from the optimizer) is structural and bit the project twice (fc head; trunk_norm). Fixed by eager _ensure_fc + explicit requires_grad_(False) freeze; pinned by tests/test_optimizer_coverage.py (7 tests: fc-movement pin, trunk_norm freeze pin, per-trainer optimizer-membership + all-trainable-move assertions via optimizer spy).

I20. Pre-registered held-out audit: PASS

I20. Pre-registered held-out audit: PASS

The flagship P2 claim was validated on the untouched 45-design test split with 5 fresh seeds (100–104, never used in development), exact MC multinomial SBC p-values (finite-sample-valid at n=45/51 bins), and a recorded dataset hash. All gates pass on every seed: coverage 0.892–0.905 (mean 0.897 vs nominal 0.90), SBC error 0.015–0.018, SBC p 0.147–0.885. Results: data/p2_heldout_audit/heldout_audit_results.json; harness: scripts_p2_heldout_audit.py (pre-registered thresholds in docstring).

I21. 1000-sample scaling experiment: MSE improves, SBC uniformity FAILS

Scaling the P2 field dataset from 300 to 1000 designs improved MSE (0.0306→0.0229, -25%) and coverage (0.887→0.928). However, SBC uniformity FAILS on all 5 pre-registered audit seeds (exact MC p = 0.003–0.049; Fisher combined p ≈ 5e-6), and this was initially reported as PASS before the adversarial review caught the discrepancy. The conformal multipliers also consume test-split ground truths (val-only discipline violated). Dataset: data/p2/fields_1000.npz. See data/p2_heldout_audit/heldout_audit_1000.json for the full audit.


2. Issues encountered → diagnosis → solution

I1. Solver numerics: penalty-spring conditioning collapse (P4 blocker)

Symptom: dataset generation produced 0/8 clips; find_flutter_boundary returned None everywhere. Diagnosis chain: design_sampler passed penalty_factor=1e6_base_matrices scaled springs to ~1e6·diag(K) → linearized pencil conditioning exploded → dense-eig backward errors O(0.1–1) → spectral_abscissa's validity gate rejected all 360 eigenvalues. Solution: absolute k_stiffness springs (min backward error ≤ 4e-7), plus dynamic QEP scaling (γ = √(‖K‖/‖M‖), s = γ·ŝ) inside spectral_abscissa so validity is meaningful on any stiffness regime. Validity went 2/360 → 360/360 across the flutter scan.

I2. Junk-mode poisoning of the spectral abscissa (P3/P4 blocker)

Symptom: after I2's fix, flutter scans still returned None: α ≈ +7e-4 at every speed. Diagnosis: least-stable pairs were 19.7–22.3 MHz penalty-constraint artifacts — accurate as pencil modes (they pass any residual gate) but physically meaningless; η_w (transverse participation) ≈ 1e-12. Solution: optional eta_w_min / omega_max gates on spectral_abscissa (defaults preserve behavior); flutter path passes eta_w_min=1e-3. Independent audit confirmed gated critical modes carry η_w = 0.023–0.19 (25–190× margin) across every repo config class.

I3. Frozen CNN readout (P2, found by review round)

Symptom: lever-I coverage 0.996 was inflated; probe showed encoder.fc bit-identical to init after training. Diagnosis: ModeShapeCNNEncoder.fc is created lazily on first forward; the optimizer parameter snapshot ran before any forward → FC head received gradients but was never in the optimizer. Solution attempt 1 (defective): guard keyed on an _fc_built attribute that is never set → no-op; caught by the follow-up review round. Solution final: unconditional _ensure_fc (self-guarded/idempotent) before the snapshot in both trainers, dummy tensor built from x_train channels. Lesson: verify fixes by probing the fixed behavior, not by trusting the diff.

I4. Sigma train/deploy feature skew (P2)

Symptom: over-coverage (0.9958 at nominal 0.90) persisted after I3. Diagnosis: Phase-2b sigma calibration consumed raw trunk features; deployment goes through trunk_norm (active per-sample standardization, γ/β never optimized). Solution: sigma sub-phase uses trunk_norm(decoder.trunk(x)). Post-fix coverage moved to 0.889 (near-nominal).

I5. Noise-channel amplification (P2)

Diagnosis: 6th stored mode had raw transverse peak < 1e-6 in 203/300 samples (membrane/shear modes ranking into the fixed window); max-norm rescaled dust to unit peak. Solution: peak-ratio gate (< 1e-4 of sample max → stored as exact zeros); test updated to zeros-or-ones semantics.

I6. Eigenmode sign/amplitude ambiguity

Solution: _canonicalize_mode_shape: sign-fix via max-|w| element with tie-robust sum-sign fallback (relative margin < 1e-3), max-norm 1. Applied at emission.

I7. Posterior collapse (P2 central scientific issue)

Characterization: ELBO pinned exactly at the free-bits floor (8 nats = 16 dims × 0.5); lever-G sweep proved the geometry bistable — init std ≤ 0.15 re-collapses (clamp kills KL gradient), std ≥ 0.2 escapes into O(10⁴)-nat noise encoding (MSE +42–65%, SBC p = 0). No continuous path via init scale. Partial remedies tested: de-conditioning (reconstruction 3.3× better — measurements flow through z when forced), KL annealing (provably inert: floor clamp kills the gradient it modulates), summaries conditioning (negative — near-redundant inputs). Deployed resolution: the heteroscedastic head evaluates at the posterior mean (μ(c) receives reconstruction gradients and carries the shape signal — lever I), while interval honesty is restored by D15's conformal layer. Raw-σ limitation disclosed.

I8. CRPS multi-observation averaging defect (found by physics audit)

Diagnosis: spread term summed over the observation axis (−2/3 instead of +1/3 for two identical ensembles); unit tests masked it with single-observation cases. Solution: per-observation spread then mean; regression test pins the exact two-identical-obs case.

I9. train_gru unconditional crash

Diagnosis: _grouped_3way_split requires test_split > 0; the train/val-only baseline passed 0.0 → ValueError whenever any clip was valid. Solution: local design-grouped validation split (purity + index alignment verified; single-group input now raises honestly).

I10–I12. Generator/trainer format and logic drift (P4)

  • np.random.SeedSequence(...) positional call → entropy= keyword (numpy ≥ 1.25).
  • Clip keys omitted excitation_idx while N_EXCITATIONS=2 guaranteed duplicates → key extended through arrays/metadata.
  • Median-variant evaluation crashed on (906, 3) vs (906,) → evaluator selects the middle column for multi-output heads.

I13. Bootstrap records crash

float.abs() on a plain float at the final bootstrap step → abs(). (Crashed a 3 h run at the last step; fixed before the final successful rerun.)

I14. Stale pytest cache misleading failure triage

lastfailed contained test names that no longer exist (from a pre-session run). Lesson: verify cache provenance against the current tree before trusting failure lists; use --cache-clear for authoritative runs.

I15. Chimera files from agent mid-write deaths

Two modules (feasibility.py, and briefly field_pipeline.py) were left interleaving fragments of two designs. Solution: programmatic reassembly from intact fragments (locate stable markers, splice), followed by compile + import + targeted tests. Repeated lane deaths made this the reliable pattern vs repeated revives.

I16. Provider instability

Repeated socket/TLS failures killed ≥ 8 agent lanes mid-task, often right after "now implementing". Mitigations: revive-once policy with delivery-first instructions; fresh dispatch when a lane went revive-immune; orchestrator takeover of nearly-finished slices; delivery-first instructions ("findings first, minimal prose").

I17. Premature commit on a masked red suite

A pytest | tail pipeline masked the exit code and a commit landed while a sibling agent was still editing. Resolution: sibling confirmed final state matched the commit; clean rerun verified 95/95. Process fix: never pipe pytest through anything that masks its exit status when gating a commit.

I18. Pre-existing-failure triage

22 failing tests in the first full run predate the session (piston-theory message drift, p3 mode-tracking, flutter-boundary semantics). Established via: (a) failure-content forensics vs current code, (b) zero-import-overlap proof, (c) targeted baseline A/B runs. Several were subsequently fixed properly by the continuation arcs rather than suppressed.


3. Verified results summary (all reproducible from committed code)

Proposal Headline (independently verified)
P1 Scalar calibration GP: LOO RMSE 5.35% vs 9.45% mean-baseline, wins all 10 modes; field pipeline characterized; charter stop condition reached
P2 All SP gates pass: SP3 coverage 0.889 (near-nominal), SP4 SBC err 0.0177 / p 0.42 (seeds 0.60/0.77/0.26), field MSE 0.0306 < 0.0419 constant-field; six-lever negative-result ledger documented
P3 Mode tracking fixed; Phase-2 feasibility gate PASSES (median cross-design MAC 0.9956, 50/50 points → Phase-5 proceeds); Phase-5 ModeDecompositionGP implemented (8 tests); Phase-5 experiment: mode-GP RMSE 4.1× better than scalar GP overall, near-crossing SKIP confirmed (no branch crossings in accessible λ range — documented with probe evidence); sweep honesty (stiffness corr 0/15→15/15) + GP ML-II
P4 Pipeline restored; 6122-clip dataset regenerated; full training exit-0: quantile coverage 0.871–0.902, huber MAE 0.0797, near-flutter MAE 0.0666, bootstrap CI [0.0814, 0.0895]; expanded report generated

4. Known open items (honestly not done)

  1. P2 SBC rank uniformity passes with the conformal layer; raw σ remains aleatoric-only (epistemic machinery — ensembles with genuine diversity, or a likelihood-aware objective — is a new arc).
  2. P2 cross-validation against real damage patterns is blocked: no COMSOL dataset with actual defects exists (P1's HF data is undamaged).
  3. P3 Phase-2 gate decision RESOLVED: gate PASSES (MAC 0.9956 ≥ 0.70), Phase-5 proceeds. Phase-5 mode-GP implemented and tested.
  4. P4 σ-width refinement toward exact nominal coverage (0.889 vs 0.90) — conformal layer is the deployed mechanism; raw-σ self-calibration is optional polish.
  5. P1 GP under-dispersed (22-36 pct coverage vs 95 nominal)
  6. P2 SBC uniformity fails at 1000 samples
  7. P1 COMSOL shape corpus VOID (2026-09-07): all 1000 Simulation_ModeShapes/ files max|w| ≤ 4.4e-8 (export degeneracy, not physics); archived correction_fields/ = −FSDT exactly (R²=1.0000), zero HF content. Field skill numbers (4.19 %) trained on fast-solver outputs, not HF data — void pending COMSOL re-export. Scalar calibration (5.35 %) unaffected. Loader now enforces 1e-6 magnitude floor.