Open benchmark of LLM tokenization — offline vs empirical calibration deltas across five providers.
llm-tokens-atlas is a reproducible, open dataset and analysis pipeline that measures how offline tokenizers (e.g. tiktoken proxies, the published BPE vocabularies) compare against the empirical token counts returned by each provider's own API or OSS tokenizer. The current dataset covers 5 providers (Anthropic claude-opus-4-7, Google gemini-2.5-pro, OpenAI gpt-4o, Mistral mistral-large-latest, Cohere command-r-08-2024), 5 prompt formats (Markdown, XML, JSON, YAML, plain text), and 12,500 real-world prompt requests (500 unique prompts × 5 formats × 5 providers; n=2,500 per provider) drawn from open corpora.
The output is a per-provider, per-format calibration delta distribution — so anyone estimating cost or context-window budgets ahead of an API call can quantify the bias of their offline counter instead of treating it as exact.
This project builds on tokenometer, which surfaced the underlying methodology and motivated the larger provider- and format-stratified sweep published here.
v0.2.0 candidate — 5-provider coverage (Anthropic + Google + OpenAI + Mistral + Cohere). Schema, drivers, and data are stable for the five-provider sweep.
n = 12,500 rows (2,500 per provider). Detailed numbers live in analysis/results.json.
| Provider | Model | Median offline-vs-empirical delta | OLS slope | R² |
|---|---|---|---|---|
| anthropic | claude-opus-4-7 |
+41.3% (cl100k_base underestimates) | 1.611 | 0.9956 |
gemini-2.5-pro |
+4.5% (mixed format-dependent bias) | 1.165 | 0.9771 | |
| openai | gpt-4o |
0.0% (tiktoken-as-truth oracle, mean +3.0%) | 1.024 | 0.9986 |
| mistral | mistral-large-latest |
−0.1% (mistral-tokenizer-js, mean +1.9%) | 1.016 | 0.9993 |
| cohere | command-r-08-2024 |
+3.2% (mixed format-dependent bias) | 1.130 | 0.9771 |
The Anthropic row is the headline: the publicly-recommended offline tokenizer
underestimates real claude-opus-4-7 cost by ~41% across thousands of prompts,
and 100% of rows underestimate (no exact / overestimate cases). OpenAI and
Mistral are baselines confirming the offline-vs-empirical pipeline is calibrated
correctly when the provider's own tokenizer is the oracle. Google and Cohere
show lower median error than Anthropic but wider format-dependent bias.
The recommended local workflow uses uv:
uv sync
make installFor library use from a project environment:
pip install llm-tokens-atlasLoad the published dataset from Hugging Face:
from datasets import load_dataset
dataset = load_dataset("faraa2m/llm-tokens-atlas")
df = dataset["train"].to_pandas()
anthropic = df[df["provider"] == "anthropic"]
print(anthropic["delta_pct"].median())Run a small credentials-free reproduction locally:
make reproduce-tinyRun the full pipeline with the default provider set:
make reproduceUse the Python bridge when another analysis script needs token counts through the same Tokenometer path as the published dataset:
from llm_tokens_atlas.tokenometer_bridge import count_offline
result = count_offline(
text="Summarize this support ticket.",
provider="openai",
model="gpt-4o",
format="markdown",
)
print(result)See docs/REPRODUCING.md for provider keys, expected
runtime, generated files, and CI-sized runs.
Use Atlas when an offline tokenizer needs a correction factor before a large batch job:
- Claude budgeting — the Anthropic sweep shows systematic undercounting versus empirical provider counts, so production budgets should include a provider-specific calibration margin.
- OpenAI sanity checks — the
gpt-4orow acts as an oracle-style baseline foro200k_basecounting. - Mistral validation — the Mistral row validates the OSS tokenizer path for SentencePiece-family models.
- Google and Cohere guardrails — the
gemini-2.5-proandcommand-r-08-2024rows expose format-sensitive mixed bias that should be calibrated rather than treated as exact.
Generated result tables live in analysis/results.json
when the analysis pipeline has been run. Generated figures are expected under
analysis/figures/.
make reproduceThis regenerates the dataset from scratch. Tokenizer and provider API versions are pinned (see data/lockfile.json once published).
See docs/REPRODUCING.md for full instructions — required API keys per provider, expected runtime at each scale, output sizes, and a CI-friendly tiny variant (make reproduce-tiny).
Atlas reuses tokenometer's
multi-provider tokenizer logic instead of reimplementing it in Python. The
integration lives in a single module:
llm_tokens_atlas/tokenometer_bridge.py— Python facade over the tokenometer CLI. Exposescount_offline,count_empirical,list_providers,list_models,list_formats, plus acount_offline_batch/count_empirical_batchpair for the high-throughput atlas pipeline.llm_tokens_atlas/install_tokenometer.sh— idempotent installer;make installruns it. Finds tokenometer via (1)tokenometeron PATH, (2) a sibling../tokenometer/repo build, (3) builds the sibling if source is present, or (4) fails with an install hint.
Any new Python code that needs token counts should import from
llm_tokens_atlas.tokenometer_bridge. Do not invoke the tokenometer CLI
directly from other modules.
The canonical home for the dataset is
https://huggingface.co/datasets/faraa2m/llm-tokens-atlas. The Hugging Face
dataset card lives at data/README.md. The upload
script is llm_tokens_atlas/publish_to_hf.py; set
HF_TOKEN in your env and run it with --dataset llm-tokens-atlas.
docs/REPRODUCING.md—make reproducemechanics + expected runtime.
Cite the version you used. Until the paper is on arxiv, cite the GitHub repo and the HuggingFace dataset directly:
@misc{llm-tokens-atlas-2026,
author = {Faraazuddin Mohammed},
title = {{llm-tokens-atlas}: An Open Benchmark of LLM Tokenization Calibration},
year = {2026},
version = {v0.2.0},
howpublished = {\url{https://github.com/faraa2m/llm-tokens-atlas}},
note = {5-provider coverage (Anthropic, Google, OpenAI, Mistral, Cohere). Companion arxiv preprint forthcoming.}
}- Code: Apache-2.0
- Data (everything under
data/): CC-BY-4.0