paq / Benchmarks
This document outlines the process for creating reproducible and comparative benchmarks for paq.
Benchmark executable scripts are located in the bin directory of this repository.
Reproducibility relies on four main tools:
- AWS EC2: For a consistent compute environment.
- Git: For hash-backed data source snapshots.
- Nix: For pinned software versions and dependencies.
- Hyperfine: For standardizing benchmark execution.
To prioritize accessibility and ease of reproduction, benchmarks are executed on cloud computing infrastructure rather than bare-metal hardware.
EC2 serves as the compute provider. To ensure result consistency across runs, the instance is provisioned with a strict configuration via CloudFormation.
The environment can be reproduced by creating a new CloudFormation stack using the ec-benchmark-template.yaml template.
The Go programming language repository serves as the data source for directory hashing benchmarks.
To ensure consistency, the benchmarks target the specific version tag go1.25.0 (commit hash: 6e676ab2b809d46623acb5988248d95d1eb7939c). Clone and checkout steps are defined in the ec2-benchmark-template.yaml template.
Note: The data source repository's
.gitdirectory is deleted prior to execution. This eliminates variability caused by version control metadata.
The Nix package manager is used to pin software builds and third-party tools to specific, hash-verified versions.
The benchmark compute instance relies on the paq flake.nix configuration to set up the environment.
Benchmarks are executed using hyperfine.
The bin directory contains a helper script, comparison.sh, which invokes hyperfine to run comparative benchmarks against other tools.
Hyperfine benchmark commands starting with find use the following command with various <hashsum> implementations:
find ./go -type f -print0 | LC_ALL=C sort -z | xargs -0 <hashsum> | <hashsum>Benchmarks are used to ensure that paq release candidates have equal or better performance than past releases.