feat(aprender-train): H4 init-load diagnostic — finds BF16 dtype mislabel root cause #1 (PMAT-CODE-PRETRAIN-INIT-LOAD-003)

[noahgift]

TL;DR

H4 cascade bisect: root cause #1 FOUND and fixed (BF16 dtype mislabel in old Qwen APR), root cause #2 STILL OPEN (val_loss=18.55 still > ln(vocab)=17.21 with fresh APR).

This PR ships the diagnostic falsifier infrastructure that pinned root cause #1 via element-by-element cross-check against the HF safetensors source.

H4 Root Cause #1: BF16 dtype mislabel

The OLD qwen2.5-coder-0.5b-instruct-fp16.apr (May-4 import) tags tensors as dtype=F16 in the APR v2 header. The SOURCE HF safetensors uses dtype=BF16. The loader sees F16 and dequantizes via f16_to_f32, producing distorted values.

Element-0 cross-check on model.norm.weight (n=896):

Source	Values [0..6]
Safetensors (BF16-decoded)	7.5625, 8.0, 7.21875, 7.3125, 7.46875, 7.375
Old APR (loaded as F16)	7.0625, 7.125, 7.0, 7.0625, 6.75, 6.875 (wrong)
Fresh APR (loaded as BF16)	7.5625, 8.0, 7.21875, 7.3125, 7.46875, 7.375 ✓ matches

Fix: re-import via current apr import. The current StreamingWriter::add_raw_f16_tensor correctly preserves BF16 (line 100-104). The old APR was created with a buggy import path that mis-tagged BF16 as F16.

H4 Root Cause #2: STILL OPEN

Even with correct BF16 weights (fresh APR), val_loss at step 1 is 18.55 — still above ln(vocab) = 17.21 (uniform-over-vocab). The dtype fix moved the dial slightly (was 19.80 with old APR) but didn't resolve the sub-random predictions.

Remaining hypotheses for the residual gap:

• H4B layout: tensor row/col-major mismatch between Qwen export and aprender expectations
• H4D forward path: cuBLAS / CudaBlock may produce wrong logits despite correct weights
• Other: tied embedding fall-through (lm_head: None → embed_tokens reuse) may have sign/scale issue

What this PR ships

falsify_h4_init_stats_qwen_embed_norm_sensible — a host-gated diagnostic test that:

• Loads Qwen 0.5B init APR (prefers fresh, falls back to legacy)
• Reports tensor stats (mean, std, min, max) for key tensors
• Dumps element-0 values for cross-comparison with safetensors source
• Asserts sensible bounds (embed std ∈ [0.005, 0.5], norm ∈ [0.01, 100])

Industrial validation example:

embed_tokens.weight: mean=0.00014, std=0.0152, range[-0.196, 0.128] — HF LLaMA init scale ✓
model.norm.weight:   mean=7.46,    std=0.84,   range[-2.28, 17.38]  — Qwen-typical ✓
q_proj.bias L0:      mean=0.03,    std=7.88,   range[-65.5, 128]    — Qwen-typical ✓

Five-Whys

1. Why was the OLD Qwen APR tagged F16? Created by a buggy import path that didn't pass is_bf16 flag through to the writer. Fixed in current apr-cli; old artifact preserved on disk.
2. Why does the fresh APR not fully fix val_loss? The dtype fix makes loaded values match safetensors, but val_loss=18.55 still exceeds ln(vocab)=17.21 — forward path or some other tensor is still producing sub-random predictions.
3. Why didn't existing falsifiers catch the dtype mislabel? No falsifier asserted "loaded values match safetensors source element-by-element". PMAT-187 NaN/Inf/explosive-mean check passes because BF16-as-F16 distortion produces values that are neither NaN nor unusually large.
4. Why ship the diagnostic before the full H4 fix? The diagnostic itself proves root cause Feature Request: Decision Tree & Random Forest for Classification Tasks #1 AND provides the bisection foundation for Feature Request: Cross-Validation Utilities #2. Per feedback_falsifier_first_cascade_pattern.md, 1 PR ≈ 1 falsifier discharge.
5. Why does the operator need to know? They have an old Qwen APR that mis-decodes silently. With this diagnostic they can verify before training; without it, the silent error wastes ~17 hours of GPU time per cycle.

Test plan

• cargo test -p aprender-train --lib falsify_h4_init_stats: PASS
• cargo test -p aprender-train --lib: 7585+ tests PASS
• cargo clippy -p aprender-train --lib -- -D warnings: clean (lib-only; --tests has 4 PRE-EXISTING errors on main, not introduced by this PR)
• rustfmt --check: clean
• LIVE diagnostic on RTX 4090: confirmed dtype mismatch + element-0 divergence

SHIP-TWO impact

• MODEL-1 ship %: unchanged at 91%
• MODEL-2 ship %: unchanged at 57% — H4 root cause Feature Request: Decision Tree & Random Forest for Classification Tasks #1 found and fix available (use fresh APR), but val_loss still > ln(vocab). The next-cycle bisection (H4B or H4D) is now well-targeted.
• §60 H1C cascade: FULLY CLOSED per fix(apr-cli): upfront vocab-format detection unblocks Qwen encoding (PMAT-CODE-TOKENIZE-BPE-UPSTREAM-001) #1598
• §61 evidence: 5g.1-v2 corpus is 7.42 bits entropy / 0% unk

Out-of-scope follow-ups

PMAT-CODE-PRETRAIN-INIT-LOAD-004 (H4 residual cascade):

• Bisect H4B (layout): forward-pass element-wise compare against HF Qwen2 reference at each layer
• Bisect H4D (forward path): instrument cuBLAS GEMM outputs against a CPU reference matmul
• Fix root cause; flip MODEL-2 ship % 57% → ≥58%

Files

• crates/aprender-train/src/train/pretrain_real.rs (+189/-77, falsify_h4_init_stats_qwen_embed_norm_sensible test)

🤖 Generated with Claude Code