docs(evidence): 5g.2 LIVE re-dispatch surfaces H1 eval-batch divergence (PMAT-CODE-PRETRAIN-EVAL-METHODOLOGY-001)

[noahgift]

Summary

Records the post-fix LIVE 500-step re-dispatch on RTX 4090 with PR #1579's populate-coverage fix applied. The data empirically confirms H1 (eval_batch degenerate) as the dominant remaining defect — H2 (populate gap) was a real fix but was NOT the root cause of the val_loss anomaly.

The smoking gun

At epoch 0 (after 100 training steps), the model has:

Metric	Value	Plausibility
train_loss	1.20 (perplexity 3.33)	✅ PLAUSIBLE for Qwen 0.5B fine-tuning on Python
val_loss	0.00081 (perplexity 1.0008)	❌ Physically IMPOSSIBLE for non-degenerate LM

1500× train/eval discrepancy at the same model state. Same kernel (fused_cross_entropy_cuda), same scaling (1.0/seq_len), same forward path. Different batches, both Python code from the same shards.

H2 was REAL but NOT the dominant cause

Run	train_loss	val_loss	Interpretation
2026-05-08 pre-fix (PR #1578)	0.0019	0.0008	H2 + H1 compounding
2026-05-09 1-step post-fix	2.24	0.628	H2 fixed; H1 still skews val_loss
2026-05-09 500-step post-fix	1.20	0.00081	H2 fixed; H1 dominant

The PR #1579 fix moved train_loss from 0.0019 (degenerate) to 1.20 (plausible) — a 1000× shift confirming structural completeness. But val_loss did NOT shift correspondingly: 0.0008 → 0.00075. Eval pipeline is independent of the populate gap.

Three H1 sub-hypotheses (each its own falsifier-discharge cascade)

• A) logits_buf state contamination — train_batch writes gradients in-place (KAIZEN-052); eval_batch's gpu_forward may not fully overwrite, leaving stale gradients that cross_entropy reads as "logits."
• B) Stream synchronization — host reads loss_partials before kernel finishes; stream.synchronize() should prevent this but a silent kernel failure could leave the buffer at zero.
• C) Held-out batch label corruption — pathological structure where get_target returns same tokens as get_input. Hard to hit by accident on real Python; least likely.

Why ship the evidence + contract bump but not the fix?

PR atomicity (feedback_falsifier_first_cascade_pattern.md). Each H1 sub-hypothesis is its own falsifier-discharge cascade. Shipping the audit trail NOW preserves the discovery for the next session and unblocks the operator from re-deriving it.

Contract bump

contracts/apr-pretrain-init-finetune-v1.yaml v1.0.0 → v1.1.0:

• status: DRAFT → DRAFT_PARTIAL_DISCHARGE
• 5/6 falsifiers DISCHARGED, 1/6 NUMERICALLY-PASSED-METHODOLOGY-SUSPECT
• Promotion to ACTIVE_RUNTIME requires H1 resolved AND re-dispatch producing val_loss in 1.5-2.5 plausible range

SHIP-TWO impact

• MODEL-1 ship %: unchanged at 91% (this is MODEL-2 work)
• MODEL-2 ship %: unchanged at 57% (still gated on honest 5g.3 verdict; this evidence is the audit trail showing why the prior numerical pass was not honest)
• §50.4 cascade: COMPLETE per feat: §50.4 step 5f.5 CUDA --init wireup (PMAT-CODE-PRETRAIN-INIT-CUDA-WIREUP-001) #1577
• 5g.2 dispatch: OPERATOR-RUNNABLE end-to-end (PR feat: §50.4 step 5f.5 CUDA --init wireup (PMAT-CODE-PRETRAIN-INIT-CUDA-WIREUP-001) #1577) with structurally-complete model (PR feat(aprender-train): respect config.use_bias in attention constructor (PMAT-CODE-PRETRAIN-INIT-POPULATE-COVERAGE-001) #1579) but HONEST 5g.3 verdict remains gated on H1 resolution

Test plan

• pv validate contracts/apr-pretrain-init-finetune-v1.yaml — 0 errors
• Documentation-only change (no Rust code, no falsifier semantics flip)
• Evidence pinned at dispatch.txt (.log gitignored)

Files

• contracts/apr-pretrain-init-finetune-v1.yaml (v1.0.0 → v1.1.0)
• evidence/section-60-5g-2-redispatch-2026-05-09/
  • dispatch.txt
  • epoch-{000,001,002}.metadata.json
  • README.md — H1/H2 hypothesis decomposition + audit
• .pv/lint-previous.json (refresh)

Next steps (out-of-scope follow-ups)

PMAT-CODE-PRETRAIN-EVAL-METHODOLOGY-001 sub-tasks:

• Author CudaTransformerTrainer::eval_batch sanity-bound test (assert loss > 0.5 on random-init + synthetic batch)
• Bisect H1 sub-hypotheses A/B/C with targeted instrumentation
• Fix root cause; re-dispatch 5g.2 for honest 5g.3 verdict

🤖 Generated with Claude Code