Skip to content

BENCHMARK.md

Latest commit

 

History

History
134 lines (110 loc) · 6.59 KB

BENCHMARK.md

File metadata and controls

134 lines (110 loc) · 6.59 KB
Raw

Benchmark: priorityjudge vs single-shot Gemini vs default Cursor's Claude

The quest brief asks for a head-to-head comparison against default Cursor. This file runs two comparisons:

  1. Primary: a single-shot Gemini baseline. Same model (gemini-3.1-pro-preview), same rubric, same calibration set, same metric. Only the multi-pass scaffolding is removed. This isolates the contribution of the architecture from the contribution of the model.
  2. Secondary: default Cursor's Claude. Captured manually from a fresh Cursor chat with no rules attached. This is the comparison the brief asks for, but it confounds model and scaffolding so we treat it as a sanity check rather than the headline.

Headline number

priorityjudge scored 9887 / 10000 on the 6-plan calibration set with 3 repeats per plan (18 total runs).

Component Weight Result Points
Agreement with human ranking (Spearman ρ) 5000 1.0000 (perfect rank match) 5000
Citation precision (verified quotes) 3000 0.9649 2895
Test-retest stability (1 − mean CV) 2000 0.9962 1992
Total 10000 9887

The agent ranked all 6 plans in the exact same order as the human grader, hallucinated only ~3.5% of cited evidence (caught by the deterministic citer pass), and was effectively deterministic across repeats (mean CV < 0.4%). Run on gemini-3.1-pro-preview with temperature=0.

Per-plan results

Plan Tier Human total Agent mean (3 runs) Δ CV
s01_aspirational (vision-only) bad 400 100 −300 0.0%
s02_flat_p0 (everything is P0) bad 1400 1300 −100 0.0%
s05_metrics_only (numbers, nothing else) bad 2200 2400 +200 0.0%
s03_no_risks (good but no risks) medium 6300 6383 +83 1.6%
s06_dep_blind (looks complete, dependency-blind) medium 6600 8050 +1450 0.6%
s04_strong (reference-quality) good 9200 9900 +700 0.0%

The agent is slightly harsher than the human on weak plans and slightly more generous on strong ones, but the rank-order is preserved exactly. The largest absolute disagreement (s06_dep_blind, +1450) comes from the agent giving partial credit on dependency awareness where the human gave nearly none — a known calibration gap that a larger calibration set would narrow.

Protocol

We feed the same calibration plan to three judges:

  1. priorityjudge (this repo). Multi-pass: extractor → 5 scorers → citation verifier → synthesizer. Gemini 3.1 Pro Preview under the hood, controlled by src/priorityjudge/llm.py.
  2. Single-shot Gemini baseline. Same model, same rubric, same calibration set. One LLM call per plan that returns the full scorecard JSON. Run by benchmarks/run_baseline.py; prompt template at prompts/baseline_singleshot.md. Its citations are still verified by the same deterministic citer, so the baseline's citation_precision measures how often single-shot scoring fabricates line numbers.
  3. Default Cursor's Claude (manual). A new Cursor chat in this repo, with no rules attached, no MCP tools, and the message:

    Rate the priority-definition quality of <plan> on a 1-10000 scale. Score it across actionability, priority clarity, risk coverage, measurability, and dependency awareness. For each dimension, cite the exact lines you used. Output JSON. Cursor's response goes into benchmarks/cursor_default/<plan_id>.md verbatim. The same scoring harness reads it.

Comparisons (1) vs (2) isolate the contribution of the multi-pass architecture: same model, same rubric, same calibration. Comparisons (1) vs (3) match the brief literally but confound model and scaffolding, so we treat them as a sanity check.

Reproducing the numbers

python benchmarks/run_benchmark.py --repeats 3   # priorityjudge
python benchmarks/run_baseline.py --repeats 3    # single-shot Gemini baseline
python benchmarks/compare.py                     # head-to-head table

Run on 2026-05-01. 18 priorityjudge repeats completed, ~22 minutes wall-clock, ~290k input tokens, ~50k output tokens. Baseline numbers land on the overnight run scheduled for the HP / WSL host.

Head-to-head — baseline run pending

Metric priorityjudge Single-shot Gemini Default Cursor's Claude
Agreement with humans 1.0000 baseline pending capture pending
Citation precision 0.9649 baseline pending capture pending
Test-retest stability 0.9962 baseline pending capture pending
Final score (1-10000) 9887 baseline pending capture pending
Mean wall-clock per plan ~73 s baseline pending capture pending
Mean tokens per plan ~57k baseline pending capture pending

The baseline row will be filled in by the overnight benchmark run on the HP / WSL host (see openclaw/skill/ for the OpenClaw integration that triggers it). The default-Cursor row stays a manual capture; protocol is in benchmarks/cursor_default/README.md.

Why we expect priorityjudge to win

Three structural advantages:

  1. Forced rubric adherence. Each of priorityjudge's scorers sees only one dimension's definition. A single-shot judge holds all five in the same context window and tends to anchor them to each other.
  2. Citation enforcement. priorityjudge re-checks every citation in pure Python after the LLM returns. A single-shot judge has no such checker; its citations land in the output unaudited. Our 9887 score already lost 105 points to citation hallucinations that this pass caught — those points would have shipped silently in a single-shot system.
  3. Test-retest stability. Independent per-criterion calls reduce the variance of the total. A single-shot total is a single sample of a noisier random variable. Our measured CV across repeats is 0.6% on average; we expect a single-shot judge to be 5-15% noisier.

Limitations

  • The calibration set is small (currently 6 synthetic plans; target is 10 real + 20 synthetic). A larger set would tighten the agreement estimates.
  • Both judges are LLM-based and inherit some shared failure modes. priorityjudge's structural guards reduce, but do not eliminate, this.
  • The human grader for the calibration set is a single person (the author). A second rater on the same plans would let us report inter-rater agreement and bound the ceiling more honestly.