feat(benchmarking): add Bayesian model fitting and CI workflow#6089
Draft
mfleader wants to merge 2 commits into
Draft
feat(benchmarking): add Bayesian model fitting and CI workflow#6089mfleader wants to merge 2 commits into
mfleader wants to merge 2 commits into
Conversation
mfleader
force-pushed
the
api-latency-analysis
branch
2 times, most recently
from
1631e38 to
c66502c
Compare
This was referenced Jun 12, 2026
mfleader
force-pushed
the
api-latency-analysis
branch
from
c66502c to
c3d0510
Compare
Adds a data collection pipeline under benchmarking/api_latency_comparison/ for comparing per-request API latency between two OGX versions. The orchestrator sets up git worktrees for each version, generates a randomized complete block design experiment matrix, starts servers with CPU pinning via mirakuru, runs Locust against each version, and records per-request response times. A third "comparison control" group runs the same code as comparison to catch false positives from environmental noise. First of two commits. Follow-up adds model fitting and a CI workflow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew F Leader <mleader@redhat.com>
Fit a Wald (Inverse Gaussian) latency model with HSGP (Hilbert Space Gaussian Process) temporal adjustment via nutpie. Detect regressions using posterior predictive quantile decisions with false positive calibration. Add MCMC diagnostics and a daily CI workflow comparing latest release against main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew F Leader <mleader@redhat.com>
mfleader
force-pushed
the
api-latency-analysis
branch
from
c3d0510 to
05d9a6a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds Bayesian model fitting, diagnostics, and a CI workflow on top of the experiment pipeline from #6088. Estimates changes in API response latency (mean, p50, p95, p99) with uncertainty intervals. The false positive detection runs a negative control (same code as comparison, run independently) to verify the experiment isn't producing spurious differences.
analysis/fit_resp_latency_model.py— Wald (Inverse Gaussian) model with HSGP (Hilbert Space Gaussian Process) temporal adjustment, fitted via nutpieanalysis/diagnostics.py— analysis quality assessment: estimation reliability, model assumptions, data fitanalysis/decisions.py— quantile decisions and false positive detectionanalysis/wald_numba.py— numba dispatch shim so nutpie can compile the Wald distributionanalysis/smoke_test.py— smoke test for the analysis dependencies (nutpie, NetCDF, LOO)analysis/MODEL.md— model specification.github/workflows/response-latency-regression-benchmark.yml— daily CI comparing latest release vs mainTest Plan
test_benchmark.pyruns the full pipeline (experiment + model fitting) and verifies analysis artifacts:decisions.csv,fp-results.json, andidata.nc.