Skip to content

Commit 4e8ae22

Browse files
pimenovolegpimenovoleg
committed
chore(adr): added Perf Bench description
1 parent f243c92 commit 4e8ae22

1 file changed

Lines changed: 304 additions & 0 deletions

File tree

‎docs/adr/0009-performance-benchmarks-vitest-browser-mode.md‎

Lines changed: 304 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,304 @@
1+
# ADR 0009: Performance Benchmarks via Vitest Browser Mode (base-ui–style harness, Angular-native metrics)
2+
3+
- Status: Proposed
4+
- Date: 2026-06-12
5+
- Decision owners: Radix NG maintainers
6+
- Related: new project `apps/radix-perf-testing` (consumers: all primitives; pilot: `checkbox`, `select`), `.github/workflows/`, `tools/scripts/benchmark/`
7+
8+
## Context
9+
10+
The library has no performance testing of any kind today — no harness, no bench files, no CI step.
11+
Regressions in mount cost (DI context creation, `effect()` setup, host bindings) or in large-list
12+
scenarios (Select/Autocomplete popups with hundreds of options) are only caught by accident.
13+
14+
[Base UI](https://base-ui.com/) — our primary reference — runs per-PR performance tests and posts a
15+
comparison comment on every PR:
16+
17+
```
18+
Performance
19+
Total duration: 1,277.17 ms -177.08 ms (-12.2%) | Renders: 50 (+0) | Paint: 1,917.67 ms -266.72 ms (-12.2%)
20+
21+
Test Duration Renders
22+
Checkbox mount (500 instances) 66.82 ms ▼ -47.27 ms (-41.4%) 1 (+0)
23+
11 tests within noise — details
24+
```
25+
26+
### How base-ui actually does it (verified against source)
27+
28+
- Bench files live in [`test/performance/tests/*.bench.tsx`](https://github.com/mui/base-ui/tree/master/test/performance);
29+
the whole `vitest.config.ts` is one line: `createBenchmarkVitestConfig()` from
30+
[`@mui/internal-benchmark`](https://github.com/mui/mui-public/tree/master/packages/benchmark) (lives in `mui/mui-public`).
31+
- The runner is **Vitest Browser Mode with Playwright (real Chromium)** — not jsdom, not Tachometer,
32+
and **not** Vitest's built-in `bench()`/Tinybench.
33+
- Their `benchmark(name, render, interaction?, options?)` helper does 10 warmup runs + 20 measured
34+
runs (configurable), with IQR-based outlier removal.
35+
- Metrics:
36+
- **Duration / Renders** — `React.Profiler` on React's profiling build;
37+
- **Paint** — the **Element Timing API**: an invisible sentinel element with an `elementtiming`
38+
attribute observed via `PerformanceObserver`, capturing the time until the browser actually
39+
paints the frame. This part is framework-agnostic.
40+
- CI (CircleCI): `BENCHMARK_UPLOAD=true pnpm test:benchmark` uploads result JSON to S3 keyed by SHA.
41+
Baseline comes either from a **same-job run of the base branch** (`BENCHMARK_BASELINE_PATH` —
42+
same runner for both sides, which is also their noise mitigation) or as a fallback from S3 by
43+
merge-base SHA.
44+
- The PR comment is **compact**; its "details" link points to MUI's
45+
[code-infra dashboard](https://github.com/mui/mui-public/tree/master/apps/code-infra-dashboard)
46+
(`code-infra-dashboard.onrender.com/benchmark-details/...?sha=…&base=…&prNumber=…`), which fetches
47+
`benchmark.json` for both SHAs from their **private S3 bucket** (`mui-org-ci`) and renders the
48+
full comparison. The dashboard itself is MUI-internal ("highly opinionated", external feature
49+
requests not accepted) and reads only their bucket — **not reusable by third parties**.
50+
- The comparison logic, however, is open source —
51+
[`compareBenchmarkReports.ts`](https://github.com/mui/mui-public/blob/master/apps/code-infra-dashboard/src/lib/benchmark/compareBenchmarkReports.ts):
52+
"within noise" is a plain relative-difference threshold (`NOISE_THRESHOLD = 0.2`, i.e. ±20%; no
53+
confidence intervals or t-tests), severity is error/success only beyond it, totals are simple
54+
sums compared with the same rule, and per-test rows show **mean ± stdDev** plus an outlier count.
55+
56+
### What does and does not transfer
57+
58+
| base-ui piece | Transfers? | Angular-native replacement |
59+
| ------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
60+
| Vitest Browser Mode + Playwright | ✅ as-is | We already use Vitest + the AnalogJS Angular Vite plugin |
61+
| `benchmark()` with warmup/runs/IQR | ✅ design as-is | Own ~150-line harness (no React dependency in the algorithm) |
62+
| `React.Profiler` for Renders | ❌ | Count change-detection cycles by wrapping `ApplicationRef.tick`; optionally Angular's DevTools profiler hook (`ɵsetProfiler`) later |
63+
| Element Timing sentinel for Paint | ✅ as-is | Same `elementtiming` attribute + `PerformanceObserver`, nothing React-specific |
64+
| S3 upload (private bucket `mui-org-ci`) | ❌ overkill | Same-runner baseline via `git worktree` at merge-base; result JSONs kept as workflow artifacts |
65+
| code-infra dashboard (details page) | ❌ MUI-internal | GitHub Actions **job summary** (`$GITHUB_STEP_SUMMARY`) with the full table; the PR comment links to it |
66+
| `compareBenchmarkReports.ts` logic & table format | ✅ port it | `tools/scripts/benchmark/compare.mjs` ports the diff/severity/totals logic and rendering |
67+
| CircleCI + bot comment | ❌ | GitHub Actions + sticky PR comment |
68+
69+
## Decision
70+
71+
Mirror base-ui's benchmark design 1:1 on our stack:
72+
73+
1. A dedicated Nx project **`apps/radix-perf-testing`** (naming mirrors `apps/radix-ssr-testing`)
74+
running **Vitest Browser Mode with the Playwright provider (Chromium)**. It is **not** part of the
75+
regular `test` target and is excluded from CI's normal test/build runs the same way
76+
`radix-ssr-testing` is.
77+
2. An own **`benchmark()` harness** (in `apps/radix-perf-testing/src/harness/`) implementing
78+
warmup + measured runs + IQR outlier removal, producing a result JSON per run.
79+
3. **Angular-native metrics**: wall-clock mount duration, change-detection cycle count, and paint
80+
time via the Element Timing API.
81+
4. **CI comparison on the same runner**: benchmark the PR head, then the merge-base via
82+
`git worktree`, diff the two JSONs with a Node script, and post/update a **sticky PR comment**
83+
with the comparison table and a within-noise verdict per test.
84+
85+
## Alternatives considered
86+
87+
- **[Tachometer](https://github.com/google/tachometer)** — statistically rigorous, real browser,
88+
built-in two-build comparison. Rejected: requires standalone HTML harness pages per scenario
89+
(a parallel build pipeline next to Vitest), and its horizon for maintenance is unclear. Vitest
90+
browser mode gives us the same real-Chromium fidelity inside infra we already run.
91+
- **Vitest `bench()` (Tinybench) in jsdom** — cheapest to add, but jsdom has no layout/paint, so no
92+
Paint metric and unrealistic DOM costs; Tinybench also cannot express the renders/paint metrics,
93+
so we'd outgrow it immediately. Rejected as the primary harness (it was the original proposal
94+
before base-ui's actual setup was verified).
95+
- **Manual profiling with Angular DevTools** — not automatable, no PR gate. Not an alternative,
96+
just the status quo.
97+
98+
## Detailed design
99+
100+
### Directory layout
101+
102+
```
103+
apps/radix-perf-testing/
104+
project.json # Nx project, target: bench (@nx/vitest:test or run-commands)
105+
vite.config.mts # browser mode config (see below)
106+
tsconfig.json
107+
src/
108+
harness/
109+
benchmark.ts # benchmark() — warmup, runs, IQR, result collection
110+
metrics.ts # CD-cycle counter, Element Timing paint observer
111+
mount.ts # mountN() — createApplication + createComponent helpers
112+
reporter.ts # writes bench-results.json (BENCH_OUTPUT_PATH env)
113+
tests/
114+
checkbox.bench.ts # pilot: mount 500, toggle interaction
115+
select.bench.ts # pilot: open popup with 1000 options
116+
tools/scripts/benchmark/
117+
compare.mjs # head.json + base.json → markdown table + noise verdict
118+
```
119+
120+
### Vite/Vitest config
121+
122+
```ts
123+
// apps/radix-perf-testing/vite.config.mts
124+
import angular from '@analogjs/vite-plugin-angular';
125+
import { defineConfig } from 'vite';
126+
import tsconfigPaths from 'vite-tsconfig-paths';
127+
128+
export default defineConfig({
129+
plugins: [angular(), tsconfigPaths()],
130+
test: {
131+
include: ['src/tests/**/*.bench.ts'],
132+
browser: {
133+
enabled: true,
134+
provider: 'playwright',
135+
headless: true,
136+
instances: [{ browser: 'chromium' }]
137+
},
138+
// benchmarks are sequential by nature — one file at a time, no parallelism
139+
fileParallelism: false
140+
}
141+
});
142+
```
143+
144+
Notes:
145+
146+
- **Do not reuse `packages/primitives/test-setup.ts`** — it initializes the jsdom/TestBed
147+
environment. The perf app bootstraps real applications via `createApplication` instead.
148+
- `*.bench.ts` files use plain `it()`/`describe()` from Vitest (as base-ui does — their `.bench.tsx`
149+
files are regular browser-mode tests, the "bench" is the helper, not Vitest bench mode), so the
150+
`include` pattern above keeps them out of every other project's `**/*.spec.ts` globs.
151+
- The suite stays **zoneless**: `provideZonelessChangeDetection()` in every bootstrapped app.
152+
153+
### Harness API
154+
155+
```ts
156+
export interface BenchmarkOptions {
157+
runs?: number; // default 20
158+
warmupRuns?: number; // default 10
159+
}
160+
161+
export interface BenchmarkResult {
162+
name: string;
163+
duration: Stats; // ms, IQR-filtered: median, mean, stdDev, min, max, samples, outliersRemoved
164+
renders: number; // CD cycles during the measured section (must be stable across runs)
165+
paint?: Stats; // ms, time to sentinel paint (Element Timing)
166+
}
167+
168+
export async function benchmark(
169+
name: string,
170+
setup: () => BenchScenario, // fresh component/app per iteration
171+
interaction?: (ctx: BenchContext) => Promise<void>, // optional re-render scenario
172+
options?: BenchmarkOptions
173+
): Promise<void>;
174+
```
175+
176+
Per iteration:
177+
178+
1. `setup()` creates a fresh host element + `createApplication({ providers: [provideZonelessChangeDetection()] })`,
179+
then `createComponent()` × N into it. The harness appends its own sentinel
180+
`<div elementtiming="bench-sentinel">` after the mounted components (see risk 2 for the
181+
constraints on that element).
182+
2. `performance.mark()` before mount; `ApplicationRef.tick()` (wrapped to count CD cycles); await the
183+
sentinel's `PerformanceObserver` entry (type `element`) for the paint timestamp.
184+
3. If `interaction` is provided, measure it as a separate named section (mutate signals, await
185+
stabilization + a fresh sentinel paint).
186+
4. Destroy the `ApplicationRef`, remove the host element, `performance.clearMarks()`.
187+
188+
After all iterations: drop outliers outside `[Q1 - 1.5·IQR, Q3 + 1.5·IQR]`, compute stats, append to
189+
the in-memory result list. A Vitest `afterAll`/reporter hook writes the full
190+
`BenchmarkResult[]` JSON to `process.env.BENCH_OUTPUT_PATH ?? 'bench-results.json'`.
191+
192+
**Renders metric**: wrap `ApplicationRef.tick` (or subscribe to a counter incremented in an
193+
`afterRender` callback registered on the app injector) and report the count per iteration. It must be
194+
deterministic; the harness should warn if the count varies across runs. `ɵsetProfiler`-based template
195+
profiling is explicitly out of scope for v1 (private API; revisit only if CD-cycle counts prove too
196+
coarse).
197+
198+
### Pilot bench files
199+
200+
- `checkbox.bench.ts`:
201+
- `Checkbox mount (500 instances)` — direct analog of base-ui's test;
202+
- `Checkbox toggle (500 instances)` — interaction: flip all `checked` signals, one CD pass expected.
203+
- `select.bench.ts`:
204+
- `Select open (1000 options)` — mount closed, interaction opens the popup; this covers collection
205+
(DOM-order `contentChildren`), popper positioning, and portal cost — our highest-risk path.
206+
207+
Scenario components live next to the bench files, are standalone, and use **no styles** (headless —
208+
paint cost is dominated by DOM size, which is exactly what we want to track).
209+
210+
### Scripts and targets
211+
212+
- `project.json` target `radix-perf-testing:bench` (vitest run with the config above).
213+
- Root `package.json`: `"primitives:bench": "nx run radix-perf-testing:bench"` under the
214+
`----Radix Primitives---` section.
215+
- Nx: the target must **not** be cacheable (timings are not reproducible artifacts).
216+
217+
### CI: comparison job and PR comment
218+
219+
New workflow `.github/workflows/benchmark.yml`:
220+
221+
- Trigger: `pull_request` with a `paths` filter on `packages/primitives/**` (plus the perf app and
222+
the workflow itself). Add a `workflow_dispatch` for manual runs.
223+
- Steps:
224+
1. Checkout with `fetch-depth: 0`; `pnpm install`; install Playwright Chromium.
225+
2. Bench **head**: `BENCH_OUTPUT_PATH=head.json pnpm primitives:bench`.
226+
3. `git worktree add ../base $(git merge-base HEAD origin/main)`; `pnpm install` in the worktree;
227+
bench **base** there with `BENCH_OUTPUT_PATH=base.json`, **reusing the head worktree's harness
228+
if the perf app doesn't exist at merge-base yet** (bootstrap period: skip comparison and post a
229+
"baseline unavailable" comment instead of failing).
230+
4. `node tools/scripts/benchmark/compare.mjs head.json base.json` produces **two outputs**:
231+
a compact `comment.md` and a full per-test breakdown appended to `$GITHUB_STEP_SUMMARY`
232+
(our substitute for base-ui's dashboard details page). `compare.mjs` **ports the diff /
233+
severity / totals logic and table format** from base-ui's open-source
234+
[`compareBenchmarkReports.ts`](https://github.com/mui/mui-public/blob/master/apps/code-infra-dashboard/src/lib/benchmark/compareBenchmarkReports.ts)
235+
rather than inventing its own.
236+
5. Post/update a sticky comment (`marocchino/sticky-pull-request-comment` or
237+
`actions/github-script` with a hidden marker), formatted like base-ui's:
238+
totals line, per-test table (duration delta with ▲/▼ and %, renders delta), collapsed
239+
`<details>` for within-noise tests, and a **"details" link to the workflow run's job summary**.
240+
6. Upload `head.json` / `base.json` as **workflow artifacts** (the repo-local substitute for
241+
base-ui's S3-by-SHA storage; subject to the default artifact retention window).
242+
- **Noise rule (v1)**: follow base-ui's actual rule — _within noise_ when
243+
`|relative Δ of the median| ≤ 20%` (`NOISE_THRESHOLD = 0.2`; they apply it to means, we apply it
244+
to IQR-filtered medians). Optionally tighten later (e.g. require non-overlapping IQR ranges) once
245+
real runner variance is known — base-ui's generous 20% reflects how noisy CI timings actually are.
246+
- The job is **informational only** (no required status) until variance on GitHub-hosted runners is
247+
characterized; same-runner head/base execution is the primary mitigation.
248+
249+
## Consequences
250+
251+
Positive:
252+
253+
- Per-PR regression visibility for mount cost and large-list popup scenarios, with real-browser
254+
paint numbers — matching the reference project's developer experience.
255+
- The harness algorithm (warmup/runs/IQR, Element Timing) is copied from a battle-tested design
256+
instead of invented.
257+
- No external storage or services; the whole pipeline is repo-local.
258+
259+
Negative / accepted costs:
260+
261+
- GitHub-hosted runners are noisy; even with same-runner comparison, small regressions (<5%) will
262+
hide in noise. Accepted for v1; thresholds are tunable and the job is non-blocking.
263+
- The CI job installs dependencies twice (head + merge-base worktree). Accepted: pnpm store makes
264+
the second install cheap.
265+
- Renders metric is coarser than React's Profiler (CD cycles, not per-component render timings).
266+
Accepted; can be deepened later via the profiler hook.
267+
268+
## Risks and open questions
269+
270+
1. **AnalogJS Angular plugin under Vitest Browser Mode** — the main technical risk; it is known to
271+
work in jsdom and in Storybook's Vite pipeline, but browser mode must be verified first.
272+
**Mitigation / go-no-go**: phase 1 is a spike that mounts one Checkbox in browser mode before any
273+
harness work. Fallback if it fails: precompile scenarios with a small Vite build (the Storybook
274+
builder path) — decided only if needed.
275+
2. **Element Timing in headless Chromium** — paints do occur in headless mode, but the observer
276+
wiring must be verified in the spike. The entry type `element` requires the `elementtiming`
277+
attribute on a rendered, non-`display:none` element — the harness appends a 1×1 transparent
278+
sentinel element itself (scenario components don't need to know about it), positioned offscreen
279+
rather than hidden.
280+
3. **Runner variance** — collect ~2 weeks of runs before considering making the check blocking or
281+
alerting on it.
282+
283+
## Rollout plan (implementation phases)
284+
285+
1. **Spike (go/no-go)**: `apps/radix-perf-testing` skeleton + browser-mode config + one trivial test
286+
mounting `RdxCheckboxRootDirective`-based component and observing a sentinel `element` timing
287+
entry. Verifies risks 1 and 2.
288+
2. **Harness + pilots**: `benchmark()` with warmup/runs/IQR, CD-cycle counter, JSON reporter;
289+
`checkbox.bench.ts` and `select.bench.ts`; `primitives:bench` script; local numbers reviewed for
290+
stability (run 5× locally, compare medians).
291+
3. **CI**: `benchmark.yml`, `compare.mjs`, sticky comment, noise thresholds; non-blocking.
292+
4. **Later (separate decisions)**: more primitives (Autocomplete virtualized, Menu, Slider drag),
293+
profiler-hook render timings, blocking thresholds, historical tracking.
294+
295+
## References
296+
297+
- base-ui bench file: <https://github.com/mui/base-ui/blob/master/test/performance/tests/checkbox.bench.tsx>
298+
- base-ui perf suite: <https://github.com/mui/base-ui/tree/master/test/performance>
299+
- `@mui/internal-benchmark`: <https://github.com/mui/mui-public/tree/master/packages/benchmark>
300+
- base-ui CI job (`test_benchmark`): <https://github.com/mui/base-ui/blob/master/.circleci/config.yml>
301+
- MUI code-infra dashboard (benchmark details page; MUI-internal, reads their private S3): <https://github.com/mui/mui-public/tree/master/apps/code-infra-dashboard>
302+
- Comparison/noise logic to port into `compare.mjs`: <https://github.com/mui/mui-public/blob/master/apps/code-infra-dashboard/src/lib/benchmark/compareBenchmarkReports.ts>
303+
- Element Timing API: <https://developer.mozilla.org/en-US/docs/Web/API/PerformanceElementTiming>
304+
- Vitest Browser Mode: <https://vitest.dev/guide/browser/>

0 commit comments

Comments
 (0)