Skip to content

Commit 1631e38

Browse files
mfleaderclaude
mfleader
and
claude
committed
feat(benchmarking): add Bayesian model fitting and CI workflow
Fit a Wald (Inverse Gaussian) latency model with HSGP (Hilbert Space Gaussian Process) temporal adjustment via nutpie. Detect regressions using posterior predictive quantile decisions with false positive calibration. Add MCMC diagnostics and a daily CI workflow comparing latest release against main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew F Leader <mleader@redhat.com>
1 parent b5829f2 commit 1631e38

14 files changed

Lines changed: 1890 additions & 20 deletions

File tree

‎.github/workflows/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ OGX uses GitHub Actions for Continuous Integration (CI). Below is a table detail
3131
| Build, test, and publish packages | [pypi.yml](pypi.yml) | Build, test, and publish packages |
3232
| Integration Tests (Record) | [record-integration-tests.yml](record-integration-tests.yml) | Auto-record missing test recordings for PR |
3333
| Release Branch Scheduled CI | [release-branch-scheduled-ci.yml](release-branch-scheduled-ci.yml) | Scheduled CI checks for active release branches |
34+
| Response Latency Regression Benchmark | [response-latency-regression-benchmark.yml](response-latency-regression-benchmark.yml) | Benchmark comparison ref vs baseline ref (defaults: main vs latest release) |
3435
| Check semantic PR titles | [semantic-pr.yml](semantic-pr.yml) | Ensure that PR titles follow the conventional commit spec |
3536
| Stainless SDK Builds | [stainless-builds.yml](stainless-builds.yml) | Build Stainless SDK from OpenAPI spec changes |
3637
| Close stale issues and PRs | [stale_bot.yml](stale_bot.yml) | Run the Stale Bot action |

‎.github/workflows/response-latency-regression-benchmark.yml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
name: Response Latency Regression Benchmark
2+
3+
run-name: Benchmark ${{ inputs.comparison_ref || 'main' }} vs ${{ inputs.baseline_ref || 'latest release' }}
4+
5+
on:
6+
schedule:
7+
- cron: '0 2 * * *' # Daily at 2 AM UTC
8+
workflow_dispatch:
9+
inputs:
10+
baseline_ref:
11+
description: 'Baseline git ref (default: latest release tag)'
12+
required: false
13+
type: string
14+
default: ''
15+
comparison_ref:
16+
description: 'Comparison git ref (default: main)'
17+
required: false
18+
type: string
19+
default: 'main'
20+
replicates:
21+
description: 'Number of replicates per group'
22+
required: false
23+
type: number
24+
default: 3
25+
run_duration:
26+
description: 'Run duration in seconds'
27+
required: false
28+
type: number
29+
default: 10
30+
31+
permissions:
32+
contents: read
33+
34+
jobs:
35+
benchmark:
36+
runs-on: ubuntu-latest
37+
timeout-minutes: 60
38+
concurrency:
39+
group: benchmark-${{ inputs.baseline_ref || 'latest' }}-${{ inputs.comparison_ref || 'main' }}
40+
cancel-in-progress: false
41+
steps:
42+
- name: Checkout
43+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
44+
with:
45+
fetch-depth: 0
46+
fetch-tags: true
47+
48+
- name: Install uv
49+
uses: astral-sh/setup-uv@1e862dfacbd1d6d858c55d9b792c756523627244 # v7.1.4
50+
with:
51+
python-version: '3.12'
52+
53+
- name: Install Rust toolchain (for nutpie)
54+
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
55+
56+
- name: Install dependencies
57+
run: uv sync --group benchmark-regression
58+
59+
- name: Run benchmark
60+
run: |
61+
uv run python -m benchmarking.api_latency_comparison.experiment.benchmark \
62+
--baseline-ref "${{ inputs.baseline_ref }}" \
63+
--comparison-ref "${{ inputs.comparison_ref || 'main' }}" \
64+
--replicates "${{ inputs.replicates || 3 }}" \
65+
--run-duration "${{ inputs.run_duration || 10 }}" \
66+
--results-dir ${{ runner.temp }}/benchmark-results
67+
68+
- name: Upload results
69+
if: always()
70+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
71+
with:
72+
name: benchmark-results-${{ github.run_id }}
73+
path: |
74+
${{ runner.temp }}/benchmark-results/analysis/
75+
${{ runner.temp }}/benchmark-results/runs/
76+
${{ runner.temp }}/benchmark-results/experiment-matrix.csv
77+
${{ runner.temp }}/benchmark-results/environment.txt
78+
${{ runner.temp }}/benchmark-results/run-log.csv

‎benchmarking/api_latency_comparison/README.md‎

Lines changed: 59 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,9 @@
11
# API Latency Comparison Benchmark
22

3-
Measures per-request latency of two OGX versions under a controlled
4-
agentic workload. Compares an older release against a newer commit by
5-
running both through a mocked agentic workload and recording
6-
per-request response times.
7-
8-
Analysis and model fitting are added in a follow-up PR.
3+
Detects latency regressions between two OGX versions using a Bayesian
4+
hierarchical model. Compares an older release against a newer commit by
5+
running both through a mocked agentic workload and fitting a Wald
6+
(Inverse Gaussian) latency model to the per-request response times.
97

108
## Overview
119

@@ -24,7 +22,9 @@ The three trials are:
2422

2523
Each run starts a fresh OGX server against a mock backend, sends
2624
agentic requests (with web_search tool calls) via Locust for a fixed
27-
duration, and records per-request latencies.
25+
duration, and records per-request latencies. A Bayesian model estimates
26+
the version effect on mean latency. If the false positive control fires
27+
(same code shows a difference), the experiment is unreliable.
2828

2929
Components:
3030

@@ -33,18 +33,22 @@ Components:
3333
- **Experiment orchestrator** (`experiment/benchmark.py`): run execution with CPU pinning
3434
- **Worktree setup** (`experiment/setup-worktree.sh`): isolated git worktrees per version
3535
- **Design matrix** (`experiment/generate_design_matrix.py`): randomized experiment design
36+
- **Model fitting** (`analysis/fit_resp_latency_model.py`): Wald (Inverse Gaussian) model + diagnostics
3637

3738
## Prerequisites
3839

3940
```bash
40-
# Benchmark experiment dependencies (Locust, mirakuru)
41-
uv sync --group api-latency-comparison
41+
# Full benchmark dependencies (experiment + analysis)
42+
uv sync --group benchmark-regression
43+
44+
# Rust toolchain (required for nutpie compilation)
45+
rustup show # or: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
4246
```
4347

4448
## Quick Start
4549

46-
The orchestrator handles worktree setup, matrix generation, and
47-
experiment execution in one command:
50+
The orchestrator handles worktree setup, matrix generation, experiment
51+
execution, and model fitting in one command:
4852

4953
```bash
5054
uv run python -m benchmarking.api_latency_comparison.experiment.benchmark \
@@ -53,6 +57,46 @@ uv run python -m benchmarking.api_latency_comparison.experiment.benchmark \
5357

5458
Output lands in an auto-timestamped directory under `results/`.
5559

60+
## GitHub Actions
61+
62+
The workflow at `.github/workflows/response-latency-regression-benchmark.yml`
63+
runs daily comparing the latest release tag against main. Manual dispatch
64+
accepts custom refs, replicates, and run duration.
65+
66+
```bash
67+
gh workflow run response-latency-regression-benchmark.yml \
68+
-f replicates=10 \
69+
-f run_duration=10
70+
```
71+
72+
## Interpreting Results
73+
74+
The fit script prints parameter estimates, diagnostics, and a summary:
75+
76+
```text
77+
============================================================
78+
RESULTS (regression threshold: 1.0ms)
79+
============================================================
80+
comparison vs baseline
81+
mean latency: +1.7ms [+1.5, +2.0]
82+
p50: +1.7ms [+1.4, +2.1] REGRESSION
83+
p95: +2.0ms [+1.3, +2.7] REGRESSION
84+
p99: +2.0ms [+0.7, +3.3] no regression
85+
============================================================
86+
```
87+
88+
- **mean latency**: posterior mean shift with 99% HDI
89+
- **p50/p95/p99**: posterior predictive quantile contrasts
90+
- **REGRESSION**: P(contrast <= 1ms) < 0.05
91+
- **False positive check**: same-code control must show no difference
92+
93+
Diagnostics include MCMC health (divergences, E-BFMI, ESS), posterior
94+
correlations, prior-to-posterior contraction, Pareto k analysis, and
95+
residual autocorrelation checks.
96+
97+
Output files: `decisions.csv`, `fp-results.json`, and `idata.nc`
98+
(full InferenceData for offline analysis).
99+
56100
## Configuration
57101

58102
| Environment variable | Default | Description |
@@ -68,6 +112,10 @@ Output lands in an auto-timestamped directory under `results/`.
68112

69113
## Implementation Notes
70114

115+
**Data filtering**: The first and last observation of each run are
116+
dropped before fitting. The first is a client warmup artifact (Locust
117+
connection setup). The last is frequently elevated (edge-of-window effect).
118+
71119
**CPU pinning**: Processes are pinned via `os.sched_setaffinity()` in
72120
`preexec_fn` callbacks, applied at fork before exec. Pinning is verified
73121
per run via `os.sched_getaffinity(pid)` after each server start.

‎benchmarking/api_latency_comparison/analysis/MODEL.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Response Latency Model
2+
3+
[Wald (Inverse Gaussian)](https://en.wikipedia.org/wiki/Inverse_Gaussian_distribution)
4+
hierarchical model with Hilbert Space Gaussian Process (HSGP) temporal
5+
adjustment. Detects mean latency shifts between OGX versions at a 1ms
6+
threshold.
7+
8+
Response times are strictly positive and right-skewed (occasional slow
9+
requests). The Wald (Inverse Gaussian) captures this naturally, while a Normal would allow
10+
impossible negative times.
11+
12+
## Likelihood
13+
14+
```text
15+
y_i ~ Wald/InvGaussian(mu_i, lambda_i)
16+
```
17+
18+
## Mean (Location) Structure
19+
20+
```text
21+
mu_i = mu_version[g_i] + beta_drift * d_i + delta_run[t_i] + f(x_i)
22+
```
23+
24+
## Shape Structure
25+
26+
Lambda is the Wald (Inverse Gaussian) shape parameter: higher values produce a narrower
27+
peak and lighter tails. Spread scales as mu^3 / lambda, so higher
28+
lambda means more consistent response times at the same mean latency.
29+
30+
```text
31+
lambda_i = exp(log_lambda_run[t_i])
32+
log_lambda_run[t] ~ Normal(log_lambda_bar, sigma_lambda)
33+
```
34+
35+
## Run Intercepts
36+
37+
Each run gets its own intercept (delta_run) to account for
38+
run-to-run variation (e.g., different thermal state, background load).
39+
Within each version group, the intercepts are independently constrained
40+
to sum to zero via [ZeroSumNormal](https://www.pymc.io/projects/docs/en/stable/api/distributions/generated/pymc.ZeroSumNormal.html)
41+
so that the version mean latency (mu_version) cleanly represents
42+
each version's average response time.
43+
44+
## Experimental Design (RCBD)
45+
46+
The run order follows a Randomized Complete Block Design. Each block
47+
contains exactly one run of each version (baseline, comparison,
48+
comparison_ctrl) in a randomly permuted order. Blocks execute in
49+
temporal sequence (block 1 first, block N last).
50+
51+
Blocking prevents temporal drift from aliasing with the treatment
52+
effect: any monotonic trend affects all three versions within a block
53+
approximately equally. Within-block randomization prevents position
54+
effects (e.g., cold-start penalty for the first run in a block)
55+
from systematically favoring one version.
56+
57+
beta_drift operates on the global chronological run position d_i,
58+
scaled to [0, 1] across the full experiment. It captures smooth
59+
linear drift that the block structure does not resolve (e.g.,
60+
gradual thermal ramp within a block). The two mechanisms are
61+
complementary: blocking removes arbitrary between-block shifts,
62+
beta_drift removes smooth within-experiment trends.
63+
64+
## Gaussian Process Temporal Adjustment
65+
66+
Sequential observations within a run share transient system state
67+
(GC pressure, connection pool warm-up, event loop saturation),
68+
producing temporal autocorrelation. The GP captures this within-run
69+
dependence so that credible intervals on the version effect (beta_v)
70+
reflect the true effective sample size. Run intercepts (delta_run)
71+
capture between-run level shifts; the GP captures within-run
72+
temporal structure by operating on the sequence number of each
73+
request within a run (1st request, 2nd request, ...).
74+
75+
The [HSGP (Hilbert Space Gaussian Process)](https://www.pymc.io/projects/docs/en/stable/api/gp/generated/pymc.gp.HSGP.html)
76+
is an approximation to a full Gaussian Process that scales
77+
linearly in the number of observations, making it practical for
78+
the large datasets typical of automated performance experiments.
79+
[Matern 3/2](https://en.wikipedia.org/wiki/Mat%C3%A9rn_covariance_function)
80+
kernel: nearby observations are correlated, correlation decays
81+
smoothly with distance.
82+
83+
```text
84+
f ~ HSGP(Matern32, m=20, c=1.5, noncentered, drop_first=True)
85+
```
86+
87+
## Priors
88+
89+
```text
90+
mu_version[g] ~ Normal(25, 10), for each version group
91+
beta_drift ~ Normal(0, 2)
92+
sigma_run ~ Exponential(1)
93+
delta_run[t] ~ ZeroSumNormal(sigma_run), independently per group
94+
log_lambda_bar ~ Normal(7, 2)
95+
sigma_lambda ~ Exponential(1)
96+
eta_gp ~ HalfNormal(0.25)
97+
ell_gp ~ InverseGamma(mu=6, sigma=3)
98+
```
99+
100+
## Derived Quantities
101+
102+
```text
103+
beta_v[g] = mu_version[g] - mu_version[baseline]
104+
```
105+
106+
## Definitions
107+
108+
A bare index like `g` or `t` ranges over all values (used in priors).
109+
A subscripted index like `g_i` or `t_i` is a lookup: it maps
110+
observation `i` to its group or run.
111+
112+
The `_bar` suffix denotes a population average: the center of the
113+
distribution that individual values are drawn from.
114+
115+
| Symbol | Known/Estimated | Code variable | Description |
116+
|---|---|---|---|
117+
| i | index | — | Observation index |
118+
| g | index | — | Version group index (baseline, comparison, comparison_ctrl) |
119+
| t | index | — | Run index |
120+
| y_i | known | `y` | Response time for observation i (ms, positive) |
121+
| g_i | known | `x_group` | Version group that observation i belongs to |
122+
| t_i | known | `x_run` | Run that observation i belongs to |
123+
| d_i | known | `x_drift` | Run's chronological position in the experiment, scaled to [0, 1] |
124+
| x_i | known | `x_time` | Observation sequence number within a run |
125+
| mu_version[g] | estimated | `mu_version` | Mean latency for version group g (baseline, comparison, comparison_ctrl) (ms) |
126+
| beta_drift | estimated | `beta_drift` | Linear drift over run ordering (ms) |
127+
| sigma_run | estimated | `sigma_run` | Hierarchical standard deviation for run intercepts (ms) |
128+
| delta_run[t] | estimated | `delta_run_full` | Run-level deviation from group mean (ms), ZeroSumNormal within each group |
129+
| log_lambda_bar | estimated | `log_lambda_bar` | Population average log shape across runs |
130+
| sigma_lambda | estimated | `sigma_lambda` | Between-run standard deviation in log shape |
131+
| log_lambda_run[t] | estimated | `log_lambda_run` | Per-run log shape (centered parameterization) |
132+
| lambda_i | deterministic | `lambda_obs` | exp(log_lambda_run[t_i]). Wald (Inverse Gaussian) shape parameter |
133+
| eta_gp | estimated | `eta_gp` | How far the GP can shift latency within a run (ms) |
134+
| ell_gp | estimated | `ell_gp` | How many observations the GP effect stays correlated over |
135+
| f(x) | estimated | `f` | HSGP Matern32 within-run temporal adjustment |
136+
| beta_v[g] | deterministic | `beta_v` | How much slower (positive) or faster (negative) version g is compared to baseline, in ms |
137+
138+
## Data Filtering
139+
140+
First and last observation per run are dropped. The first is a
141+
Locust client warmup artifact (+20ms). The last is an edge-of-window
142+
effect.
143+
144+
## Code Reference
145+
146+
| What | File | Function |
147+
|---|---|---|
148+
| Model definition and data filtering | `fit_resp_latency_model.py` | `build_model()` |
149+
| Fitting and LOO | `fit_resp_latency_model.py` | `fit_and_diagnose()` |

‎benchmarking/api_latency_comparison/analysis/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copyright (c) The OGX Contributors.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the terms described in the LICENSE file in
5+
# the root directory of this source tree.

0 commit comments

Comments
 (0)