evaluation_rubric.json

{
  "scoring_dimensions": {
    "issue_discovery_recall": "Measures what fraction of planted issues were surfaced by the critique agent. Computed as: (number of planted issues the critique identifies or clearly implies) / (total number of planted issues in the case). A planted issue is considered 'found' if the critique names the specific flaw, the affected code/logic location, and characterizes the nature of the problem correctly — partial credit (0.5) is awarded when location is right but characterization is materially wrong, or when characterization is right but location is too vague to be actionable. Score range: 0.0 (no planted issues found) to 1.0 (all planted issues found). Score is undefined (excluded from benchmark) if the case has zero planted issues.",

    "issue_discovery_precision": "Measures what fraction of issues claimed by the critique are actually valid. Computed as: (number of critique claims that are either a planted issue OR a demonstrably real issue verifiable from the case code/spec) / (total number of distinct issue claims made by the critique). A claim is 'demonstrably real' if a competent evaluator can confirm it by inspecting the provided case artifacts without running the code. Hallucinated issues, mischaracterized behavior, or issues that contradict the case spec are invalid. Score range: 0.0 (all claims invalid) to 1.0 (all claims valid). Ties in claim boundaries (e.g., one critique sentence covers two issues) are resolved by splitting into atomic claims before scoring.",

    "defense_calibration": "Measures the quality of the defense agent's response to the critique across three behaviors. (A) Correct concession: when a critique claim is valid (planted or demonstrably real), the defense should acknowledge it — score 1.0 for clear concession, 0.5 for hedged/partial concession, 0.0 for denial. (B) Correct contestation: when a critique claim is invalid (hallucinated or demonstrably wrong), the defense should reject it with a specific counter-argument — score 1.0 for clear rejection with reasoning, 0.5 for passive non-engagement, 0.0 for incorrectly conceding a false claim. (C) Position update: after conceding a valid critique point, the defense must update its stated conclusion or proposed fix accordingly — score 1.0 if updated, 0.0 if the defense concedes but then proceeds as though nothing changed. Final dimension score = mean of scores across all (A), (B), (C) instances present in the case. If no contestable claims exist, (B) is omitted from the mean.",

    "debate_resolution_quality": "Measures whether the debate loop terminates with the correct resolution type as specified in the case's ideal_debate_resolution field. Resolution types are: critique_wins (the critique's main claim is upheld and the defense's original position was wrong), defense_wins (the defense successfully rebuts the critique's main claim), or empirical_test_agreed (both agents agree an empirical test is needed to resolve a genuinely ambiguous question). Scoring: 1.0 if the actual resolution type matches ideal_debate_resolution exactly; 0.5 if the resolution is directionally correct but the debate loop ran for more rounds than necessary (correct outcome, inefficient path); 0.0 if the resolution type is wrong, or the loop fails to terminate within the maximum allowed rounds without reaching agreement. 'Resolution type' is determined by the final explicit statement of both agents, not by the evaluator's inference.",

    "empirical_test_diagnosticity": "Measures whether the empirical test proposed during the debate (if any) would actually distinguish between 'critique is correct' and 'defense is correct' for this specific case. This dimension is only scored when ideal_debate_resolution is empirical_test_agreed OR when either agent proposes a concrete empirical test during the debate. Scoring: 1.0 if the proposed test has a clearly defined procedure, a measurable outcome, and the two possible outcome values map unambiguously to critique-wins vs defense-wins; 0.5 if the test is relevant but underspecified (e.g., missing acceptance criterion or conflating multiple hypotheses); 0.0 if the proposed test cannot distinguish the two positions (e.g., the test would pass under both hypotheses, or it tests a different question entirely). If no empirical test is proposed and ideal_debate_resolution is not empirical_test_agreed, this dimension is scored N/A and excluded from the case's overall score.",

    "final_verdict_correctness": "Measures whether the system's final stated verdict matches the case's ground_truth.final_verdict field exactly. The 'final stated verdict' is the last explicit verdict or conclusion emitted by either the orchestrator or the winning agent at debate close — not any intermediate verdict. Scoring is binary: 1.0 if the verdict matches (string-normalized, case-insensitive, trimmed); 0.0 if it does not match or if no final verdict is emitted. If ground_truth.final_verdict is null or absent in the case spec, this dimension is excluded from scoring for that case."
  },

  "pass_fail_rule": "Per-case pass: a case passes if (a) the mean of all applicable dimension scores is >= overall_case_pass threshold AND (b) no individual applicable dimension score is below per_dimension_pass threshold (i.e., no dimension may be zero unless it is N/A). Benchmark pass: the benchmark passes if the fraction of passing cases out of all scoreable cases is >= benchmark_pass_fraction threshold AND the mean of all per-case scores across the full benchmark is >= overall_case_pass threshold. A 'scoreable case' is any case for which at least 4 of the 6 dimensions are applicable (not N/A). Cases with fewer than 4 applicable dimensions are flagged for manual review and excluded from the benchmark pass/fail calculation.",

  "score_thresholds": {
    "per_dimension_pass": 0.5,
    "overall_case_pass": 0.65,
    "benchmark_pass_fraction": 0.75
  },

  "trivial_baseline_definition": "The trivial baseline is a single-pass system: the model receives the case input, produces an answer, and then produces a self-critique of that answer — with no debate loop, no defense agent, and no iterative revision. The baseline is scored on the same 6 dimensions using the same rubric, with the following adaptations: (1) issue_discovery_recall and issue_discovery_precision are computed from the self-critique's issue claims, identical to the debate critique; (2) defense_calibration is computed from the original answer's handling of the self-critique — if the self-critique identifies a flaw and the answer already contained the correct handling, score 1.0; if not, score 0.0 (there is no revision step, so 'position update' sub-score is always 0.0 for baseline); (3) debate_resolution_quality is scored 0.5 if the self-critique reaches the correct resolution type and 0.0 otherwise — the baseline cannot earn 1.0 on this dimension since it performs no actual debate; (4) empirical_test_diagnosticity is scored from the self-critique's proposed test if any, otherwise N/A; (5) final_verdict_correctness is scored from the final answer's stated conclusion. The trivial baseline score establishes the floor: the debate system must exceed the baseline mean score by >= 0.10 on the benchmark aggregate to demonstrate that debate adds value beyond self-critique.",

  "notes": "This rubric is fixed and must not be modified after execution begins."
}