docs(spec): SHIP-TWO-001 §65 — SHIP-005 NOT-DISCHARGE (gx10 164-run pass@1 = 34.15%, 50pp below floor)

[noahgift]

Summary

Records the empirical RED outcome from the gx10 164-problem HumanEval run. SHIP-005 does NOT LIVE-discharge.

Verdict

passed = 56/164
pass@1   = 0.3415  ← FAIL (50.65pp below 0.848 effective floor)
pass@10  = 0.9868
pass@100 = 1.0000

Critical signal — model IS capable

• pass@1 = 34.15% (FAIL)
• pass@10 = 98.68%
• pass@100 = 100.00%

The bug is in greedy-temperature-0 sampling/decoding, not in model knowledge. The model can solve every problem given enough samples.

Three falsifiable hypotheses

H1 (priority 50%): gx10 teacher sha256 differs from lambda-vector

• gx10: 0a854098d05b...
• lambda-vector: a394dd2867...
• Sync canonical artifact + rerun → resolves in ~5h compute

H2 (priority 30%): align_continuation_indent (PR #1617) too aggressive

• May dedent completions that are already correctly indented

H3 (priority 20%): BPE tokenization artifacts on complex prompts

Methodology Lesson #12 (NEW)

A directional empirical sample can lie about full-distribution performance. The 10-problem lambda-vector sample (80% pass@1) was within 95% CI [44%, 97%] of the 86% nominal floor — appeared a strong directional signal. The 164-run revealed 34.15% — well outside that CI. The first 10 problems happen to be the easiest; harder problems concentrate later (HumanEval/100+).

Ship-% movement

• MODEL-1 ship %: stays at 94% (ceiling without further investigation). SHIP-005 remains PARTIAL pending H1/H2/H3.
• MODEL-2 ship %: unchanged at 57%.

Changes

• docs/specifications/aprender-train/ship-two-models-spec.md:
  • Atomic next action: v3.10.0 → v3.11.0
  • New §65 section ABOVE §63 (newest-first)
• evidence/section-65-ship-005-not-discharge-2026-05-11/ (NEW):
  • humaneval-164-gx10.json (raw apr eval --json, 1174 lines)
  • per-problem-summary.json (passed/failed task IDs)
  • findings.json (structured H1/H2/H3 analysis + verdict)

🤖 Generated with Claude Code

[noahgift]

Closing as superseded — the §65→§71 cascade narrative is complete on main via PRs #1629/#1631/#1633/#1634/#1636/#1642 (and the in-tree §67/§68/§69/§70/§71 sections). SHIP-005 LIVE-DISCHARGED at 86.59% pass@1 (§71); see contracts/apr-eval-humaneval-harness-invariant-v1.yaml v1.1.0 for the empirical evidence and root cause.