CyberAI measures its own engine against a small, fully self-contained suite of deliberately-vulnerable targets that we author and serve ourselves. No third-party benchmark is required to reproduce these numbers.
Public AI-pentest claims are easy to inflate and hard to verify. The local suite exists so every number we publish is:
- Reproducible — targets ship in this repo (
cyberai/bench/apps/), built into throwaway containers bycyberai/bench/docker_builder.py. - Binary — a target counts as solved only when an unambiguous success signal is present (a flag, output only a shell could produce, out-of-web-root file contents). No heuristic "looks exploited".
- Traceable — each run emits a Markdown scorecard with engine version, provider, model, and timestamp.
External suites (e.g. CVE-Bench) plug into the same BenchTask contract as
optional adapters, used for public-leaderboard parity — never as a dependency
of the product.
| id | class | CWE | success signal |
|---|---|---|---|
local-sqli-login |
SQL injection | CWE-89 | auth-bypass flag returned by /login |
local-cmdi-ping |
command injection | CWE-78 | shell-evaluated arithmetic in output |
local-path-traversal |
path traversal | CWE-22 | flag from an out-of-web-root file |
cyberai bench list
cyberai bench run --suite local --scorecard docs/benchmarks/scorecards/local.md
cyberai bench run --suite local --engine agent| metric | --engine real |
--engine agent |
|---|---|---|
| pass@1 | 3/3 (100.0%) | 3/3 (100.0%) |
| what ran | fixed per-class probes | the CyberAI pipeline itself |
| agreement with the probes | n/a | 3/3 |
| measured | 2026-07-24, CyberAI 1.4.0 | 2026-07-27, CyberAI 1.4.0 |
We author the targets, the probes, and the success signals. That makes the result reproducible, and it also means the number is a self-test, not a comparison. Stated plainly so nobody has to infer it from the source:
- Self-authored suite. 3/3 says our three targets are exploitable and our three probes detect it. It says nothing about how CyberAI compares to any other tool. Cross-tool claims need a third-party suite, and we do not make them here.
--engine realmeasures the probes, not the agent. The probes are fixed exploit checks; the agents take no part in that run. Read it as a harness-and-target check.--engine agent, below, is the one that measures the product.- A solve must be earned. Each probe looks for a signal that is absent from
its own request — arithmetic only a shell can evaluate, a flag stored only
inside an out-of-web-root file. A target that echoes request input back
cannot register as exploited.
tests/unit/test_bench_negative_control.pyruns every probe against hardened targets and requires all of them to fail. - Small denominator. Three tasks means one task moves the rate by 33 points. The suite grows as classes are added, and the honest reading of any single figure has to account for that.
This mode runs CyberAI against each target the way an operator would: the recon agent crawls the app and records its injectable surface, the exploit agent attacks that surface through the shared knowledge base, and a finding appears only where a payload's proof held in the response. The agent is given the URL and nothing else — no class hint, no parameter list, no probe.
The per-class probe still runs, as an independent judge the attacker never consults. Two mechanisms, two verdicts, and the score follows the agent, because the agent is the thing being measured.
Where the two disagree, the run says so instead of averaging it away:
- Agent solved, probe blind — either the agent proved something the probe cannot see, or the proof is weaker than it looks. Worth reading the finding.
- Probe solved, agent missed — the target is exploitable and the pipeline failed to get there. This is the honest capability gap and the number worth driving down.
- Probe errored — recorded as unknown, never as a clean bill of health.
Caveats specific to this mode:
- Same self-authored suite. Everything said above about a small, self-authored denominator applies unchanged. Agreement between our agent and our probe on our targets is a sanity check, not an external result.
- Web classes only. The agent path exercised here is HTTP surface discovery and web exploitation. Bench containers publish one app and nothing else, so port scanning, whois and DNS are skipped deliberately — they would add minutes and no information.
- Wall-clock is container time. Roughly eleven seconds per task, nearly all of it spent starting and stopping the target; the agent's own work against a responding app is a fraction of a second.
The number moves only when the engine earns it. This page is updated from a measured run, never by hand.