Auto-generated by
make scorecard. Do not edit by hand. Source:benchmarks/results/latest.json(produced bymake benchmark).
- Harness version:
1.2 - Seed:
42 - Rank cutoff
k:5 - Token estimator:
CharDivFourEstimator(deterministic, no model dependency) - Answer-phase budget:
6000tokens
All numbers below are reproducible deterministically by running
make benchmark && make scorecard from a clean checkout. Hardware and
Python version affect latency only; recall, drops, dedup, and token
counts are environment-independent.
Gold dataset: 200 hand-curated queries (benchmarks/routing_gold.json)
covering all 8 catalog namespaces. Each query is repeated three times
for latency percentile stability.
| catalog_size | queries | precision@5 | recall@5 | MRR | p50 (ms) | p95 (ms) | p99 (ms) |
|---|---|---|---|---|---|---|---|
| 50 | 131 | 0.1191 | 0.5649 | 0.4978 | 0.420 | 0.514 | 0.650 |
| 83 | 200 | 0.0800 | 0.3825 | 0.3242 | 0.691 | 0.841 | 1.205 |
| 1000 | 200 | 0.0310 | 0.1475 | 0.1456 | 37.086 | 40.610 | 47.855 |
Generated by make benchmark-matrix (issue #208). The matrix lets
you compare the bundled tfidf baseline against the optional
bm25 ([bm25] extra), fuzzy ([retrieval] extra), and embedding
backends (embedding_hashing stdlib baseline, embedding_st
[embeddings] extra — #266) across three catalog sizes.
Skipped backends are recorded explicitly.
| backend | catalog_size | queries | precision@5 | recall@5 | MRR | p50 (ms) | p95 (ms) | p99 (ms) | status |
|---|---|---|---|---|---|---|---|---|---|
| bm25 | 100 | 200 | 0.0800 | 0.3825 | 0.3399 | 5.394 | 7.591 | 8.399 | ok |
| bm25 | 500 | 200 | 0.0470 | 0.2250 | 0.2165 | 28.707 | 36.881 | 41.865 | ok |
| bm25 | 1000 | 200 | 0.0330 | 0.1575 | 0.1525 | 71.638 | 89.514 | 112.387 | ok |
| embedding_hashing | 100 | 200 | 0.1050 | 0.5175 | 0.4360 | 0.907 | 6.953 | 7.218 | ok |
| embedding_hashing | 500 | 200 | 0.0560 | 0.2700 | 0.2674 | 10.070 | 39.180 | 43.171 | ok |
| embedding_hashing | 1000 | 200 | 0.0420 | 0.2000 | 0.1931 | 37.094 | 94.277 | 102.979 | ok |
| embedding_st | 100 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| embedding_st | 500 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| embedding_st | 1000 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| fuzzy | 100 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| fuzzy | 500 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| fuzzy | 1000 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| tfidf | 100 | 200 | 0.0800 | 0.3825 | 0.3220 | 0.949 | 1.082 | 1.193 | ok |
| tfidf | 500 | 200 | 0.0490 | 0.2325 | 0.2314 | 9.652 | 10.105 | 10.803 | ok |
| tfidf | 1000 | 200 | 0.0310 | 0.1475 | 0.1456 | 36.419 | 39.120 | 42.986 | ok |
Generated by python benchmarks/benchmark.py --matrix --mixed-shapes
(issue #277). The mixed-shape catalog has one head namespace
(analytics_xl, 200 items), two mid-weight namespaces, four small
operational namespaces, and a 100-namespace long tail (one item
each) — a deliberately asymmetric distribution that contrasts
with the uniform 8-namespace pool used by the headline matrix.
Numbers here are intentionally lower: gold queries only cover
the natural 8 namespaces, so the synthetic head-and-tail tools
act as targeted noise items.
| backend | catalog_size | queries | precision@5 | recall@5 | MRR | p50 (ms) | p95 (ms) | p99 (ms) | status |
|---|---|---|---|---|---|---|---|---|---|
| bm25 | 500 | 200 | 0.0260 | 0.1200 | 0.1065 | 20.256 | 24.963 | 26.822 | ok |
| embedding_hashing | 500 | 200 | 0.0340 | 0.1550 | 0.1376 | 10.680 | 40.380 | 42.665 | ok |
| embedding_st | 500 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| fuzzy | 500 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| tfidf | 500 | 200 | 0.0260 | 0.1200 | 0.1010 | 9.912 | 10.626 | 13.045 | ok |
Breakdown of recall@k by gold-set namespace (issue #209). Useful for identifying which tool namespaces benefit most from a given scoring backend.
| backend | catalog_size | namespace | queries | recall@5 |
|---|---|---|---|---|
| bm25 | 100 | admin | 25 | 0.9200 |
| bm25 | 100 | analytics | 25 | 0.7200 |
| bm25 | 100 | billing | 25 | 0.1200 |
| bm25 | 100 | comms | 25 | 0.4000 |
| bm25 | 100 | crm | 25 | 0.1200 |
| bm25 | 100 | docs | 25 | 0.2800 |
| bm25 | 100 | infra | 25 | 0.0800 |
| bm25 | 100 | search | 25 | 0.4200 |
| bm25 | 500 | admin | 25 | 0.4600 |
| bm25 | 500 | analytics | 25 | 0.2400 |
| bm25 | 500 | billing | 25 | 0.5800 |
| bm25 | 500 | comms | 25 | 0.3600 |
| bm25 | 500 | crm | 25 | 0.0400 |
| bm25 | 500 | docs | 25 | 0.0800 |
| bm25 | 500 | infra | 25 | 0.0400 |
| bm25 | 500 | search | 25 | 0.0000 |
| bm25 | 1000 | admin | 25 | 0.5800 |
| bm25 | 1000 | analytics | 25 | 0.4000 |
| bm25 | 1000 | billing | 25 | 0.2400 |
| bm25 | 1000 | comms | 25 | 0.0000 |
| bm25 | 1000 | crm | 25 | 0.0400 |
| bm25 | 1000 | docs | 25 | 0.0000 |
| bm25 | 1000 | infra | 25 | 0.0000 |
| bm25 | 1000 | search | 25 | 0.0000 |
| embedding_hashing | 100 | admin | 25 | 0.4000 |
| embedding_hashing | 100 | analytics | 25 | 0.5200 |
| embedding_hashing | 100 | billing | 25 | 0.3200 |
| embedding_hashing | 100 | comms | 25 | 0.4400 |
| embedding_hashing | 100 | crm | 25 | 0.6400 |
| embedding_hashing | 100 | docs | 25 | 0.5600 |
| embedding_hashing | 100 | infra | 25 | 0.6800 |
| embedding_hashing | 100 | search | 25 | 0.5800 |
| embedding_hashing | 500 | admin | 25 | 0.5000 |
| embedding_hashing | 500 | analytics | 25 | 0.4400 |
| embedding_hashing | 500 | billing | 25 | 0.5800 |
| embedding_hashing | 500 | comms | 25 | 0.3200 |
| embedding_hashing | 500 | crm | 25 | 0.0400 |
| embedding_hashing | 500 | docs | 25 | 0.1600 |
| embedding_hashing | 500 | infra | 25 | 0.0400 |
| embedding_hashing | 500 | search | 25 | 0.0800 |
| embedding_hashing | 1000 | admin | 25 | 0.5800 |
| embedding_hashing | 1000 | analytics | 25 | 0.5200 |
| embedding_hashing | 1000 | billing | 25 | 0.3800 |
| embedding_hashing | 1000 | comms | 25 | 0.0000 |
| embedding_hashing | 1000 | crm | 25 | 0.0400 |
| embedding_hashing | 1000 | docs | 25 | 0.0000 |
| embedding_hashing | 1000 | infra | 25 | 0.0000 |
| embedding_hashing | 1000 | search | 25 | 0.0800 |
| tfidf | 100 | admin | 25 | 0.9200 |
| tfidf | 100 | analytics | 25 | 0.7200 |
| tfidf | 100 | billing | 25 | 0.1200 |
| tfidf | 100 | comms | 25 | 0.4000 |
| tfidf | 100 | crm | 25 | 0.1200 |
| tfidf | 100 | docs | 25 | 0.2800 |
| tfidf | 100 | infra | 25 | 0.0800 |
| tfidf | 100 | search | 25 | 0.4200 |
| tfidf | 500 | admin | 25 | 0.4600 |
| tfidf | 500 | analytics | 25 | 0.2000 |
| tfidf | 500 | billing | 25 | 0.5400 |
| tfidf | 500 | comms | 25 | 0.3600 |
| tfidf | 500 | crm | 25 | 0.0400 |
| tfidf | 500 | docs | 25 | 0.0800 |
| tfidf | 500 | infra | 25 | 0.0400 |
| tfidf | 500 | search | 25 | 0.1400 |
| tfidf | 1000 | admin | 25 | 0.5800 |
| tfidf | 1000 | analytics | 25 | 0.3600 |
| tfidf | 1000 | billing | 25 | 0.2000 |
| tfidf | 1000 | comms | 25 | 0.0000 |
| tfidf | 1000 | crm | 25 | 0.0400 |
| tfidf | 1000 | docs | 25 | 0.0000 |
| tfidf | 1000 | infra | 25 | 0.0000 |
| tfidf | 1000 | search | 25 | 0.0000 |
Reading the routing tables:
precision@5is bounded by1 / kwhen each query has a single expected tool, so the headline accuracy signal isrecall@kandMRR.- Recall degrades predictably as the catalog grows — noise items
compete with true matches; the routing-only experience for catalogs
larger than ~200 items benefits from one of the optional retrieval
backends (
bm25,fuzzy) configured viaRouter(scorer_backend=...). - p50/p95 latencies stay under a millisecond at catalog_size ≤ 83.
At catalog_size 1000 the p99 climbs into the tens of milliseconds
because the beam search has to evaluate substantially more children
per step; this is the regime where switching to a retriever-first
shortlist (the
Retrieverprotocol on theEngineRegistry) is the expected next step.
Reference event logs under benchmarks/scenarios/ are pushed through
ContextManager.build_sync(phase=Phase.answer). The firewall
intercepts every tool_result; large results become artifacts and the
prompt sees their summaries instead.
| scenario | events | included | dropped | dedup | tokens | budget | util % | artifacts | compaction |
|---|---|---|---|---|---|---|---|---|---|
| large_catalog | 60 | 60 | 0 | 0 | 1480 | 6000 | 24.7% | 15 | 1.00x |
| long_conversation | 82 | 82 | 0 | 0 | 2500 | 6000 | 41.7% | 21 | 1.41x |
| mixed_payload | 15 | 15 | 0 | 0 | 488 | 6000 | 8.1% | 4 | 1.64x |
| short_conversation | 18 | 18 | 0 | 0 | 487 | 6000 | 8.1% | 4 | 1.00x |
| stress_conversation | 147 | 136 | 11 | 4 | 6590 | 6000 | 109.8% | 32 | 3.29x |
| tiny_payload | 20 | 20 | 0 | 0 | 256 | 6000 | 4.3% | 5 | 1.00x |
Reading the table:
dropped > 0meansselect_and_packhad to evict candidates to stay under the answer-phase budget — thestress_conversationscenario is sized to force this so the budget-driven selection stage shows up in benchmark output (#181).dedup > 0proves the Jaccard near-duplicate stage actively removed redundant context.compaction > 1.0×is the average ratio of raw artifact bytes to injected summary bytes — that's the firewall's load-bearing job.compaction == 1.00×rows (tiny_payload,short_conversation,large_catalog) are the firewall correctly no-op'ing on tiny inputs — everytool_resultstill passes through the envelope, but the summary equals the raw text so no token cost is saved or added (#271). The headline reduction figures further down reflect the firewall's value when payloads are actually large; small-payload rows guard against the impression that the firewall always wins.
Token cost and coverage proxy against the no-op baseline of
concatenating all tool schemas + the full conversation history.
The naïve token count is measured with tiktoken.cl100k_base; the
coverage proxy is items_included / event_count (deterministic,
no LLM judge — see scripts/baseline_naive.py).
| scenario | naive_tokens | cw_tokens | token reduction | coverage proxy |
|---|---|---|---|---|
| large_catalog | 2767 | 1480 | 46.51% | 100.00% |
| long_conversation | 4365 | 2500 | 42.73% | 100.00% |
| mixed_payload | 2277 | 488 | 78.57% | 100.00% |
| short_conversation | 1946 | 487 | 74.97% | 100.00% |
| stress_conversation | 17482 | 6590 | 62.30% | 92.52% |
| tiny_payload | 1704 | 256 | 84.98% | 100.00% |
- Deterministic seeds. All catalog generation, scenario loading,
and beam-search tie-breaking is seeded; identical inputs always
produce identical outputs (
make benchmarkis a no-op on a fresh re-run for routing accuracy + context metrics; only latency varies with hardware). - No LLM calls by default. The harness is pure-Python, stdlib +
minimal core deps. The token estimator is
CharDivFourEstimatorso the headline numbers do not depend ontiktoken's cached encoding state. The optional end-to-end real-model section (#269) is off by default and never runs in CI. - No network access by default. The benchmark is safe to run
in air-gapped CI environments. The end-to-end real-model capture
(#269) and the
tiktokenparity check (#268) require network or a cached encoding respectively, and degrade to askippedrow when unavailable. - Hardware variance. Latency numbers are measured on the runner
that produced
latest.json. Treat them as ordering, not absolutes: the relative cost between catalog sizes is portable; the absolute microsecond count is not. See the Hardware reference rig section below for the canonical reference machine (#267).
Reference rig (numbers in this scorecard are calibrated for):
- GitHub Actions ubuntu-latest (2-core x86_64)
- System:
Linux - Machine:
x86_64 - CPU logical cores:
2 - Python:
3.10+
Absolute latency on other hardware will differ; the relative cost between catalog sizes and backends is portable.
Measured on (host that produced the current latest.json):
- System:
Linux - Machine:
x86_64 - Processor:
x86_64 - CPU logical cores:
4 - Python:
CPython 3.11.15 - Platform:
Linux-6.18.5-x86_64-with-glibc2.39
Quantifies the drift between CharDivFourEstimator (used everywhere
above) and tiktoken.cl100k_base on the gold-query corpus. This is
measurement of the estimator, not a routing metric — when the
two diverge, the headline token figures above are still self-
consistent, but readers comparing the numbers against an OpenAI
tokenizer expectation should apply the drift below (#268).
Token-estimator parity check is unavailable on this run (status: skipped: ProxyError); the headline CharDivFourEstimator numbers elsewhere in this scorecard are still valid — they do not depend on tiktoken at runtime.
Off by default. When enabled (--with-real-model flag plus
CW_BENCH_LLM_PROVIDER + CW_BENCH_LLM_API_KEY env vars) the
harness sends ≤5 gold queries through an OpenAI-compatible chat
endpoint and records prompt/completion tokens, USD cost, and
round-trip latency. Pure end-to-end, no fan-out, no retries; the
section exists so deployment-cost readers can sanity-check the
harness's offline token estimates against real provider usage (#269).
Status: skipped: offline by default.
Set CW_BENCH_LLM_PROVIDER + CW_BENCH_LLM_API_KEY and pass --with-real-model to enable. See benchmarks/README.md.
See benchmarks/README.md for the full harness reference
and the per-scenario notes.
make benchmark # writes benchmarks/results/latest.json
make scorecard # writes benchmarks/scorecard.md from latest.json
git diff --quiet benchmarks/scorecard.md # passes on clean re-runPer-backend matrices are generated via make benchmark-matrix (#208)
and now cover tfidf, bm25, fuzzy, embedding_hashing, and
embedding_st (#266). Weekly scheduled regeneration runs out of
.github/workflows/benchmark-scorecard-weekly.yml (#207).
Optional capture flags:
--mixed-shapes— emit the head-heavy + long-tail catalog matrix (#277).--no-tiktoken-parity— disable the estimator parity check (#268).--with-real-model— run the end-to-end real-model capture (#269).