Commit 76f3156
feat(apr-cli): extend APR_EVAL_DEBUG diagnostic to MBPP harness (PMAT-CODE-MBPP-DIAG-001)
The §69 diagnostic surface (PR #1634) and §70 RC3 fix (PR #1635) closed
the harness-bug class for HumanEval. MBPP's path (run_mbpp_inference
+ run_mbpp_inference_cuda) was not yet instrumented. This PR extends
APR_EVAL_DEBUG to MBPP so future investigation of MBPP failures has
ground-truth diagnostics on the same surface.
What changes:
- run_mbpp_inference (CPU path) now calls
execute_python_test_with_diagnostics and emits
/tmp/apr_eval_debug_MBPP_<task>.json when APR_EVAL_DEBUG=1 is set.
- run_mbpp_inference_cuda (CUDA path) gets the same treatment.
What does NOT change:
- run_mbpp_inference still uses the legacy
AprTransformer::forward_with_cache + AprKVCache path. PMAT-CODE-
SHIP-005-FIX (PR #1616) replaced this for HumanEval with realizar::
run_inference + OwnedQuantizedModel::from_apr. MBPP needs the same
routing fix — but that's a separate multi-PR cascade scope (also
includes H4 ChatML wrap + R1+R2 extraction equivalents for MBPP).
Out of scope for this PR.
- MBPP prompts are natural language (not Python signatures), so the
§70 RC3 import-stripping bug does NOT apply to MBPP.
Why ship this now:
- Pure diagnostic — zero behaviour change for non-APR_EVAL_DEBUG callers
- Lets us run a 1-problem MBPP smoke under APR_EVAL_DEBUG=1 to verify
the legacy path's failure mode (currently undiagnosed)
- Mirrors the pattern that successfully diagnosed §69 RC3 in 5 minutes
on gx10
Test plan:
- [x] cargo check -p apr-cli --features inference → clean
- [x] cargo check -p apr-cli --features "inference,cuda,training" → clean
- [x] cargo fmt --all → clean
- [ ] gx10 single-MBPP-problem APR_EVAL_DEBUG=1 smoke (next slice;
will document MBPP failure mode in a §72-class amendment)
Refs:
- crates/apr-cli/src/commands/eval/inference.rs::write_apr_eval_debug
- contracts/apr-eval-humaneval-harness-invariant-v1.yaml v1.1.0
- PR #1634 (HumanEval diagnostic surface)
- PR #1635 (HumanEval RC3 fix; cascade base for this branch)
Closes task #53 (MBPP harness diagnostic extension; renamed from
"RC3 prompt-preamble fix" since RC3 does not apply to MBPP's NL
prompts — that decision recorded in commit body).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>1 parent b7e69bf commit 76f3156
1 file changed
Lines changed: 26 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1596 | 1596 | | |
1597 | 1597 | | |
1598 | 1598 | | |
1599 | | - | |
| 1599 | + | |
| 1600 | + | |
| 1601 | + | |
| 1602 | + | |
| 1603 | + | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
| 1607 | + | |
| 1608 | + | |
| 1609 | + | |
| 1610 | + | |
| 1611 | + | |
1600 | 1612 | | |
1601 | 1613 | | |
1602 | 1614 | | |
| |||
1718 | 1730 | | |
1719 | 1731 | | |
1720 | 1732 | | |
1721 | | - | |
| 1733 | + | |
| 1734 | + | |
| 1735 | + | |
| 1736 | + | |
| 1737 | + | |
| 1738 | + | |
| 1739 | + | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
1722 | 1746 | | |
1723 | 1747 | | |
1724 | 1748 | | |
| |||
0 commit comments