Skip to content

Commit 76f3156

Browse files
noahgiftclaude
andcommitted
feat(apr-cli): extend APR_EVAL_DEBUG diagnostic to MBPP harness (PMAT-CODE-MBPP-DIAG-001)
The §69 diagnostic surface (PR #1634) and §70 RC3 fix (PR #1635) closed the harness-bug class for HumanEval. MBPP's path (run_mbpp_inference + run_mbpp_inference_cuda) was not yet instrumented. This PR extends APR_EVAL_DEBUG to MBPP so future investigation of MBPP failures has ground-truth diagnostics on the same surface. What changes: - run_mbpp_inference (CPU path) now calls execute_python_test_with_diagnostics and emits /tmp/apr_eval_debug_MBPP_<task>.json when APR_EVAL_DEBUG=1 is set. - run_mbpp_inference_cuda (CUDA path) gets the same treatment. What does NOT change: - run_mbpp_inference still uses the legacy AprTransformer::forward_with_cache + AprKVCache path. PMAT-CODE- SHIP-005-FIX (PR #1616) replaced this for HumanEval with realizar:: run_inference + OwnedQuantizedModel::from_apr. MBPP needs the same routing fix — but that's a separate multi-PR cascade scope (also includes H4 ChatML wrap + R1+R2 extraction equivalents for MBPP). Out of scope for this PR. - MBPP prompts are natural language (not Python signatures), so the §70 RC3 import-stripping bug does NOT apply to MBPP. Why ship this now: - Pure diagnostic — zero behaviour change for non-APR_EVAL_DEBUG callers - Lets us run a 1-problem MBPP smoke under APR_EVAL_DEBUG=1 to verify the legacy path's failure mode (currently undiagnosed) - Mirrors the pattern that successfully diagnosed §69 RC3 in 5 minutes on gx10 Test plan: - [x] cargo check -p apr-cli --features inference → clean - [x] cargo check -p apr-cli --features "inference,cuda,training" → clean - [x] cargo fmt --all → clean - [ ] gx10 single-MBPP-problem APR_EVAL_DEBUG=1 smoke (next slice; will document MBPP failure mode in a §72-class amendment) Refs: - crates/apr-cli/src/commands/eval/inference.rs::write_apr_eval_debug - contracts/apr-eval-humaneval-harness-invariant-v1.yaml v1.1.0 - PR #1634 (HumanEval diagnostic surface) - PR #1635 (HumanEval RC3 fix; cascade base for this branch) Closes task #53 (MBPP harness diagnostic extension; renamed from "RC3 prompt-preamble fix" since RC3 does not apply to MBPP's NL prompts — that decision recorded in commit body). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent b7e69bf commit 76f3156

1 file changed

Lines changed: 26 additions & 2 deletions

File tree

crates/apr-cli/src/commands/eval/inference.rs

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1596,7 +1596,19 @@ fn run_mbpp_inference(
15961596
format!("{completion}\n{setup}\n{tests}\n")
15971597
};
15981598

1599-
let ok = execute_python_test(&full_program, 10);
1599+
let exec_result = execute_python_test_with_diagnostics(&full_program, 10);
1600+
let ok = exec_result.success;
1601+
1602+
if std::env::var("APR_EVAL_DEBUG").is_ok() {
1603+
write_apr_eval_debug(
1604+
&task_id,
1605+
&prompt,
1606+
&tokenizer.decode(&tokens),
1607+
completion,
1608+
&full_program,
1609+
&exec_result,
1610+
);
1611+
}
16001612

16011613
if ok {
16021614
passed += 1;
@@ -1718,7 +1730,19 @@ fn run_mbpp_inference_cuda(
17181730
format!("{completion}\n{setup}\n{tests}\n")
17191731
};
17201732

1721-
let ok = execute_python_test(&full_program, 10);
1733+
let exec_result = execute_python_test_with_diagnostics(&full_program, 10);
1734+
let ok = exec_result.success;
1735+
1736+
if std::env::var("APR_EVAL_DEBUG").is_ok() {
1737+
write_apr_eval_debug(
1738+
&task_id,
1739+
&prompt,
1740+
&tokenizer.decode(&tokens),
1741+
completion,
1742+
&full_program,
1743+
&exec_result,
1744+
);
1745+
}
17221746

17231747
if ok {
17241748
passed += 1;

0 commit comments

Comments
 (0)