A benchmarking report is a standard data format describing the cluster configuration, workload, and results of a benchmark run. The report acts as a common API for different benchmarking experiments. Each supported harness in llm-d-benchmark creates a benchmark report upon completion of a run, in addition to saving results in its native format.
There are two versions of the benchmark report, 0.1 and 0.2. Both reports are generated by the llm-d-benchmark harness pod, but new applications consuming benchmark data should use version 0.2 reports.
A benchmark report describes the inference service configuration, workload, and performance results. Individual traces from single inference executions are not captured, but time-series or statistical results may be captured.
See br_v0_2_example.yaml for a dummy example report (values may be non-sensical). A JSON Schema for the benchmark report version 0.2 is at br_v0_2_json_schema.json.
The top-level fields for the benchmark report are version, run, scenario, and results.
The version field (0.2) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.
The run field contains details about the benchmark run. Only the unique ID run.uid is required, with other fields including start/end times and duration, user name, and additional IDs for the experiment (to group benchmark reports), cluster, and pod.
scenario describes the inference stack configuration (under scenario.stack) and workload details (under scenario.load). These are the inputs to a benchmark experiment, and ideally will be sufficiently populated to ensure the benchmark is reproducible and leaves no question as to what exactly is being measured.
scenario.stack consists of a list of components such as inference engine instances (vLLM, SGLang...), routers, and any other part of the system used to support the inference endpoint. No specific component is required, so this may be customized to the needs of a particular experiment.
Each component has a metadata, standardized, and native field. The native subsection can store any arguments, environment variables, and configuration details in the native format of a particular component. The standardized subsection stores details in a single standardized schema for that component, regardless of the specific vendor or version used. For example, an inference engine component could be vLLM, SGLang, etc. but they will always certain configuration options in common albeit defined in different ways. The standardized subsection for an inference engine contains common details like LLM model name, accelerator hardware model and count, and parallelisms used.
The workload details under scenario.load follow the same methodology as components, with metadata, standardized, and native subsections.
The results field contains outputs from a benchmarking experiment. This may include any one of request-level performance metrics, observability data, stack component health, or profiling.
A benchmark report describes the inference service configuration, workload, and aggregate results. Individual traces from single inference executions are not captured, rather statistics from multiple traces of identical scenarios are combined to create a report.
See br_v0_1_example.yaml for a dummy example report (values may be non-sensical). The JSON Schema for the benchmark report version 0.1 is at br_v0_1_json_schema.json. The report has three top-level fields, version, scenario, and metrics.
While each of these fields is required, some subfields may be optional or not apply to the specific benchmark being performed. For example, some metrics may not be captured or supported by a certain benchmarking toolset. In cases where one desires to capture information that is not part of the standard benchmark report schema, a metadata field may be placed almost anywhere under scenario or metrics to add arbitrary data.
The version field (0.1) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.
The scenario describes precisely what was measured. This includes details about the inference platform (full stack, including versions and the important runtime arguments), cluster configuration (like GPUs and parallelism utilized), and workload. The content in this field should be detailed enough that it is sufficient to launch a repeat benchmarking experiment that will yield similar results (within some reasonable bound of variability).
Note
With future revisions the scenario field could be used as an input format for executing benchmark runs. For this to be practical we would first need to standardize the definition of a workload, specifically in scenario.load.args which is currently the exact arguments (non-standard) sent to a particular harness. In addition, the input format would need a way to describe swept parameters. A benchmark report currently describes a single point in any sweep.
The metrics field contains all of the results for the report. This does not include individual trace details, rather statistics for all runs that were captured in benchmarking for a particular scenario. This includes request-level performance metrics (like latencies and throughput), details about the inference service (like request queue lengths and KV cache size), and hardware metrics (such as GPU compute and memory utilization). Some of the underlying fields, such as the hardware metrics, more traditionally fall in the category of "observability" rather than "benchmarking".
The schema for a benchmarking report is defined through Python classes using Pydantic in schema_{version}.py. Instantiating an instance of a benchmark report class includes various checks, such as ensuring compliance with the schema, proper use of units, and defining all required entities.
numpy>=2.3.1
pydantic>=2.11.7
PyYAML>=6.0.2
A BenchmarkReport class may be instantiated from a JSON/YAML file with import_benchmark_report(), or from a JSON/YAML string with yaml_str_to_benchmark_report(). Both of these functions will detect the appropriate version of the benchmark report.
To generate a JSON or YAML string from a BenchmarkReport instance, use the get_json_str() and get_yaml_str() methods. To save as a JSON/YAML file, use the export_json() or export_yaml() methods.
import benchmark_report
# Instantiate a BenchmarkReport instance from a YAML file
br=benchmark_report.import_benchmark_report("benchmark_report/br_v0_2_example.yaml")
# Create a YAML string from the BenchmarkReport
br_yaml_str = br.get_yaml_str()
# Print the string
print(br_yaml_str)
# Save the BenchmarkReport as a YAML file
br.export_yaml("br_test.yaml")An instance of BenchmarkReport may be created directly, for example:
import benchmark_report
br = benchmark_report.BenchmarkReportV02(**{
"run": {
"uid": "38b1f169ca178b756f7483523b17de61",
},
"scenario": {
"stack": [
{
"metadata": {
"schema_version": "0.0.1",
"label": "vllm-svc-0",
"cfg_id": "cc73fc6b51a1d3b8128f312d70476d7c",
},
"standardized": {
"kind": "inference_engine",
"tool": "llm-d",
"tool_version": "ghcr.io/llm-d/llm-d-cuda:0.3.1",
"role": "decode",
"replicas": 2,
"model": {
"name": "Qwen/Qwen3-0.6B",
},
"accelerator": {
"model": "NVIDIA-H100-80GB-HBM3",
"count": 8,
"parallelism": {
"tp": 8,
},
},
},
"native": {},
},
],
"load": {
"metadata": {
"schema_version": "0.0.1",
"cfg_id": "a4e18f265cc33786a42b8a3f7ac2edcb",
},
"standardized": {
"tool": "inference-perf",
"tool_version": "0.3.0",
"input_seq_len": {
"distribution": "fixed",
"value": 2148,
},
"source": "sampled",
},
"native": {},
},
},
"results": {
"request_performance": {
"aggregate": {
"latency": {
"request_latency": {
"units": "s",
"mean": 0.0368578234128654,
},
},
},
},
},
})
print(br.get_yaml_str())The native formats returned by different harnesses may be converted to a benchmark report using functions in native_to_br0_2.py, or using the CLI defined in cli.py. This CLI may be executed with python -m benchmark_report.cli ... at the root of ths repository, and can import native results data of a harness and print to stdout a benchmark report, or save a report to file if a second argument is provided.