CHI-Bench Leaderboard

Public, data-only record of benchmark submissions for CHI-Bench.

We accept submissions via pull request. The full audit packet (manifest + per-trial verifier evidence + compressed trajectories) lives in git so reviewers can inspect any submission directly from the PR diff.

The rendered leaderboard lives at actava.ai/benchmarks/leaderboards — and reads results.csv files out of this repo.

Benchmarks tracked

Benchmark	Producer repo	Current version	Submissions
chi-bench	actava-ai/chi-bench	chi-bench-v1.0.0	see submissions/

Submit a result

Tip

Prefer reading on the web? The same flow with collapsible step UI lives at actava.ai/benchmarks/submit, and a deeper reference is at actava.ai/benchmarks/docs/leaderboard.

Two repos, one handoff:

1. Producer — run your trials and produce a packet with the benchmark's tooling. For chi-bench (see actava-ai/chi-bench for the full lifecycle, or the web docs):

   uv run cb submission prepare -f configs/submissions/<id>.yaml
   # → logs/submissions/<id>/packet/YYYY-MM-DD-<id>/

   See chi-bench's submission packet contract for the directory shape.

2. Leaderboard (this repo) — fork this repo and open a PR adding that packet. Two paths below (Quick / Manual) run the same CI validation (.github/workflows/validate.yml).

3. CI labels the PR valid-submission / invalid-submission / needs-review and posts a sticky report; a maintainer reviews and merges. Verified runs land on the live leaderboard within one business day.

Important

Each submission PR must touch only one new directory under benchmarks/<bench>/submissions/<YYYY-MM-DD>-<slug>/. See CONTRIBUTING.md for the scope rule and reviewer policy.

One-time setup

Before your first submission you need a fork of this repo. Either:

# Option A — via the gh CLI (recommended; also handles auth)
gh auth login                          # authenticate to GitHub
gh repo fork actava-ai/leaderboard --clone=false
git config --global user.email "you@example.com"   # if not already set

…or open https://github.com/actava-ai/leaderboard in your browser and click Fork at the top-right.

After forking, clone your fork (not actava-ai's directly):

git clone https://github.com/<you>/leaderboard && cd leaderboard

Subsequent submissions reuse this same fork — no need to re-fork.

Claude Code

This repo ships a submit-to-leaderboard skill that wraps the helper flow above with preflight checks, partial-failure recovery, and pointers to producer-side fixes when the validator complains. Open the repo in Claude Code, point it at your packet, and ask it to submit — it invokes the skill automatically.

> /submit-to-leaderboard /abs/path/to/packet/2026-05-12-<slug>/ to the leaderboard

Codex

Codex reads AGENTS.md at the repo root, which points at the same submit-to-leaderboard skill (the file is plain markdown — instructions are platform-agnostic; only the helper tool names differ from Claude Code's). From a Codex session in the repo:

> submit /abs/path/to/packet/2026-05-12-<slug>/ to the leaderboard

Codex picks up the AGENTS.md pointer, reads the skill, and follows the same flow.

Helper script

python scripts/submit.py /path/to/packet/2026-05-12-<slug>/

The helper:

1. Reads the packet's submission.json:dataset.name to route it to the right benchmark subtree.
2. Runs scripts/validate.py against the proposed directory.
3. Copies the packet into benchmarks/<bench>/submissions/<dir>/.
4. Creates a branch sub/<bench>/<dir>, commits, pushes to your fork (or directly, with --no-fork).
5. Opens a PR via gh pr create. Prints the PR URL.

Flags: --no-fork, --no-open-pr, --on-conflict {abandon,replace,bump-date}, --leaderboard-repo <slug>.

Manual submission

From your fork clone (see One-time setup above):

cp -r /path/to/packet/2026-05-12-<slug>/ benchmarks/<bench>/submissions/
python scripts/validate.py benchmarks/<bench>/submissions/2026-05-12-<slug>/
git checkout -b sub/<bench>/2026-05-12-<slug>
git add benchmarks/<bench>/submissions/2026-05-12-<slug>/
git commit -m "<bench>: <team> · <agent> · <model>"
git push origin sub/<bench>/2026-05-12-<slug>
gh pr create -R actava-ai/leaderboard --base main

Both flows go through the same CI validation (.github/workflows/validate.yml).

Pre-PR sanity check

scripts/validate.py is a thin shim around the CI validator — same code path. Run it locally before opening a PR to catch problems early:

python scripts/validate.py benchmarks/chi-bench/submissions/2026-05-12-<slug>/

Exit 0 = validation passed. Exit 1 = errors (printed); fix and rerun.

Inspecting a submission

Submission directories are plain files. Click into any one on GitHub to see the manifest, headline metrics (in the auto-generated README), and the per-trial tree. For a sortable, filterable view of every merged submission with PA/UM/CM breakdown columns, see the live leaderboard at actava.ai/benchmarks/leaderboards.

Trajectories are zstd-compressed JSONL:

zstdcat benchmarks/chi-bench/submissions/<dir>/trials/<domain>/<trial_id>/agent/trajectory.jsonl.zst | jq .

Reviewer / maintainer notes

See CONTRIBUTING.md.

See also

• Live leaderboard — sortable table of every merged submission.
• CHI-Bench overview & authors — what the benchmark measures and why.
• Submission walkthrough — the producer + leaderboard flow with collapsible step UI.
• Web docs · Leaderboard repo — the same workflow this README describes, with deeper reviewer notes.
• Web docs · Run experiments — producer-side commands that build the packet you submit here.
• Browse all 75 tasks — the task explorer.
• Producer repo (actava-ai/chi-bench) — the benchmark code that produces packets accepted here.

License

Apache-2.0. See LICENSE.