Skip to content

Commit d8a30b9

Browse files
committed
docs(perf): retain rejected actual-server PGO screens
1 parent 15673a5 commit d8a30b9

12 files changed

Lines changed: 1212 additions & 10 deletions

File tree

scripts/perf/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -857,3 +857,14 @@ The next step is fresh training and qualification of the actual Darwin server
857857
with its release profile, allocator, features and build-std settings. Do not
858858
reuse the probe profile as shipping evidence or turn PGO on unconditionally.
859859
[Sanitized measurements and limits](evidence/native-pgo-current-2026-09-07/README.md).
860+
861+
The separately matched [counter-only HTTP PGO screen](evidence/server-pgo-counter-http-4059/README.md)
862+
did not meet its predeclared continuation threshold and retained a diagnostic
863+
mismatch despite exact geometry and data-model payloads. It was rejected before
864+
repeated qualification; no flags shipped. This does not dispose of full-value
865+
PGO, which requires its own fresh training, compatibility audit and endpoint
866+
qualification. Do not repeat the same counter-only profile as a hidden win.
867+
868+
### Rejected full-value actual-server PGO (#4059)
869+
870+
Fresh full-value profiles from a generator-only graceful shutdown did not qualify the shipping Darwin server. The held-out screen missed its continuation threshold, retained a strict CSG diagnostic mismatch and exposed a largest-model time/memory regression. The separate native processing-probe result does not transfer automatically to the actual endpoint. Stop both server variants without enabling PGO or adding repeated qualification; do not substitute the training cohort or drop failed diagnostics to promote them. The [full-value evidence](evidence/server-pgo-full-value-http-4059/README.md) preserves source/profile provenance, compiler compatibility limits and all corpus outcomes.
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Rejected actual-server counter-only PGO screen (#4059)
2+
3+
Do not ship or expand this candidate. The held-out screen falls below its predeclared continuation threshold, and a CSG diagnostic comparison remains failed. No five-pair cohort was started. This verdict applies to the continuous counter-only candidate, not PGO in general.
4+
5+
[screen.json](./screen.json) retains every anonymous model, fixture hash, pair order, cold HTTP readiness, startup/upload/cache milestones, geometry counts, sampled RSS and exact gate. The primary held-out timing aggregate includes the diagnostic-failed model explicitly; the separately reported qualified subset excludes it. The subset is not a replacement primary result. Training fixtures remain marked and are not given extra weight.
6+
7+
Both artifacts use the corrected server JSON cache implementation from #4064, the Darwin downloadable release/build-std configuration, default allocator/features and identical source hashes. The candidate adds the fresh continuous IR counter profile. Its value-site warnings were retained and checked against matching LLVM source: branch weights are applied, but value profiling is absent. This is distinct from the earlier native full-value processing probe, which did not measure this HTTP endpoint or this artifact.
8+
9+
All raw geometry batch sequences and data-model bytes match across cold arms and cache replay. The complete semantic comparison fails for model-18 because its CSG failure count differs; metadata, symbolic data and both cache replays remain exact. [#4067](https://github.com/LTplus-AG/ifc-lite/issues/4067) tracks the source audit: the current KernelError emitter records accepted open topology, while shared mesh caches omit diagnostics on hits and allow concurrent rebuilds. That mechanism can vary counts without changing mesh bytes, but the retained aggregate logs do not identify the extra operation. This explanation does not waive the failed gate.
10+
11+
The timed boundary includes raw upload, complete geometry SSE and data-model receipt. It excludes offline verification and client reconstruction. Fresh server processes and empty caches were used for each arm. Both captures and process cleanup completed before the isolated offline verifier; verified APFS copy-on-write retention then ran before the next pair. Each pair passed its free-space reserve check. Interrupted earlier observer pilots were excluded rather than pooled or reconstructed.
12+
13+
RSS is sampled process residency, not physical footprint. Very short requests can finish before a representative sample, so their peaks and a cross-corpus memory aggregate are unqualified. Startup remains separately visible. The screen does not establish browser readiness, server-restart cache behavior, other-platform performance or absence of regressions.
14+
15+
A successor actual-server full-value PGO experiment remains pending. Its separate graceful-generator mechanism permits normal profile flushing; neither its eventual outcome nor the native processing-probe result should be attributed to this rejected counter-only screen.
16+
17+
The separate [full-value actual-server screen](../server-pgo-full-value-http-4059/README.md) is now complete and also rejected under its frozen gates. No results are pooled and no shipping compiler flags change.

scripts/perf/evidence/server-pgo-counter-http-4059/screen.json

Lines changed: 292 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,40 @@
1-
# Actual Darwin server PGO qualification: pending
1+
# Actual Darwin server PGO: both screens rejected
22

3-
The native processing probe has a qualified corpus result in the [separate record](../native-pgo-current-2026-09-07/README.md). The actual HTTP server screen is still running. No actual-server gain, universal improvement, shipping readiness or profile-use flag change is claimed here.
3+
The native processing probe has a qualified corpus result in the [separate record](../native-pgo-current-2026-09-07/README.md). The [counter-only HTTP screen](../server-pgo-counter-http-4059/README.md) did not meet its continuation threshold and retained one failed diagnostic gate. It did not continue to five pairs. The distinct [full-value generator-only-shutdown screen](../server-pgo-full-value-http-4059/README.md) also failed its continuation gates; its results are not pooled with this profile. No shipping readiness or profile-use flag change is claimed here.
44

55
## What has passed
66

7-
Five fixed public fixtures trained a fresh instrumented server after the JSON cache roundtrip correction (#4064). Each completed cold geometry/data-model delivery and exact same-process cache replay. Profiles are continuous counter-only, with no value-profile data. All three compiled-source hash maps match despite a later documentation-only commit. Artifact, compiler, source, training and profile evidence is supplied separately in [stack PR #4070](https://github.com/LTplus-AG/ifc-lite/pull/4070), including `pre-screen-provenance.json`; it is not part of this harness-only commit.
7+
Five fixed public fixtures trained a fresh instrumented server after the JSON cache roundtrip correction (#4064). Each completed cold geometry/data-model delivery and exact same-process cache replay. Profiles are continuous counter-only, with no value-profile data. All three compiled-source hash maps match despite a later documentation-only commit. Artifact, compiler, source, training and profile evidence is in `pre-screen-provenance.json`.
88

9-
LLVM21 value-site warnings do not discard branch profiles: `populateCounters()` and `setBranchWeights()` precede `annotateValueSites()`, whose mismatch returns only from the value annotation helper. The [matching upstream source](https://github.com/llvm/llvm-project/blob/llvmorg-21.1.5/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp) and its digest/line references are recorded. Missing-function diagnostics and value-site warnings remain visible; this is not a warning-free build claim.
9+
LLVM 21 value-site warnings do not discard branch profiles: `populateCounters()` and `setBranchWeights()` precede `annotateValueSites()`, whose mismatch returns only from the value annotation helper. The [matching upstream source](https://github.com/llvm/llvm-project/blob/llvmorg-21.1.5/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp) and its digest/line references are recorded. Missing-function diagnostics and value-site warnings remain visible; this is not a warning-free build claim.
1010

1111
## Measurement limits
1212

1313
Cold means a fresh server process, empty application cache and raw IFC upload. The operating-system page cache is not purged; preflight fixture hashing can warm it. Local loopback HTTP includes upload, wire persistence and observer overhead; it does not measure internet latency or physical-disk cold reads. Cold readiness ends at complete data-model receipt; cache publication/replay has separate timestamps. Memory is sampled server RSS, not physical footprint.
1414

15-
Both timed arms finish and their server processes are reaped before offline decoding or evidence retention. The offline child exits before the next pair. One operator owns the timing host; no concurrent builds, profiling, compression or heavy hashing. A single pair is a screen, not a qualified five-pair estimate. The held-out22 cohort is primary; all27 and fixed training5 are secondary.
15+
Both timed arms finish and their server processes are reaped before offline decoding or evidence retention. The offline child exits before the next pair. One operator owns the timing host; no concurrent builds, profiling, compression or heavy hashing. A single pair is a screen, not a qualified five-pair estimate. The held-out 22 cohort is primary; all 27 and fixed training 5 are secondary.
1616

1717
## Reproduction source
1818

1919
`reproduce/` contains the actual HTTP transport, full wire/data-model witnesses, tests and paired driver, plus parameterized build/training/profile tools. `harness-source-map.json` identifies exact private-origin hashes and published hashes, including explicit path-parameter adaptations. These are manual evidence tools, not production entry points or CI performance gates. User IFC files and generated wire evidence are never committed.
2020

21-
Use Python3.9 or newer and the recorded dependencies in `reproduce/requirements.txt`; use the repository-pinned Rust toolchain and its matching `llvm-profdata`. Darwin training additionally needs the documented16KiB profile section alignment. Do not reuse profiles across source changes or infer other target/compiler behavior.
21+
Use Python 3.9 or newer and the recorded dependencies in `reproduce/requirements.txt`; use the repository-pinned Rust toolchain and its matching `llvm-profdata`. Darwin training additionally needs the documented 16 KiB profile section alignment. Do not reuse profiles across source changes or infer other target/compiler behavior.
2222

23-
Provide an experiment `plan.json` with sourceCommit, profileKind and exactly five training rows `{label, path, publicSha256}`. The fixture schema examples are supplied by [stack PR #4070](https://github.com/LTplus-AG/ifc-lite/pull/4070), including `corpus.example.json`; they are not included in this harness-only commit. Replace example paths with local files. The driver requires exactly27 unique fixture hashes and a training subset of5. The same follow-up supplies `capacity-projection.example.json` with label/SHA-linked budgets. Supply an explicitly reviewed PROJECTION file; storage requirements must be re-evaluated for different fixtures.
23+
Provide an experiment `plan.json` with sourceCommit, profileKind and exactly five training rows `{label, path, publicSha256}`. `corpus.example.json` documents the fixture manifest schema; replace example paths with local files. The driver requires exactly 27 unique fixture hashes and a training subset of 5. `capacity-projection.example.json` contains measured label/SHA-linked capacity budgets; storage requirements must be re-evaluated for different fixtures.
2424

25-
From this directory, with SOURCE, EXPERIMENT, COMMIT, CORPUS, PROJECTION and PROF_DATA_TOOL set explicitly by the operator:
25+
From this directory, with SOURCE, EXPERIMENT, COMMIT, CORPUS and PROF_DATA_TOOL set explicitly by the operator:
2626

2727
```bash
2828
python reproduce/build_server.py --source "$SOURCE" --experiment "$EXPERIMENT" --expected-commit "$COMMIT" control
2929
python reproduce/build_server.py --source "$SOURCE" --experiment "$EXPERIMENT" --expected-commit "$COMMIT" generate
3030
python reproduce/train.py --experiment "$EXPERIMENT"
3131
python reproduce/audit_profile.py --experiment "$EXPERIMENT" --llvm-profdata "$PROF_DATA_TOOL"
3232
python reproduce/build_server.py --source "$SOURCE" --experiment "$EXPERIMENT" --expected-commit "$COMMIT" use
33-
python reproduce/screen_http_v2.py --manifest "$CORPUS" --training-plan "$EXPERIMENT/plan.json" --base-provenance "$EXPERIMENT/control-provenance.json" --candidate-provenance "$EXPERIMENT/use-provenance.json" --projection "$PROJECTION" --out "$EXPERIMENT/http-screen" --cpu-window-confirmed
33+
python reproduce/screen_http_v2.py --manifest "$CORPUS" --training-plan "$EXPERIMENT/plan.json" --base-provenance "$EXPERIMENT/control-provenance.json" --candidate-provenance "$EXPERIMENT/use-provenance.json" --projection capacity-projection.example.json --out "$EXPERIMENT/http-screen" --cpu-window-confirmed
3434
```
3535

3636
These commands intentionally refuse existing build targets/profiles/output directories. Keep failures instead of silently retrying or overwriting. The APFS retention helper only operates on explicitly released generated evidence; it verifies exact bytes and keeps distinct paths/inodes. Run its tests and inspect capacity permits before use on another filesystem.
3737

3838
Run witness tests with `python -m unittest discover -s reproduce -p 'test_*.py'`; retention tests separately use `-s reproduce/retention`. They encode actual Arrow/Parquet data and mutate payload fields. No source-text assertions or private IFC fixtures are required.
3939

40-
The eventual HTTP verdict will be appended only after all retained output, memory and timing gates have been reviewed. Issue#4059 remains open while that work is active.
40+
The counter-only verdict is recorded separately, with all failures. The bounded qualification in #4059 is complete with both actual-server candidates rejected. `pre-screen-provenance.json` is the immutable initial counter-only preparation snapshot, including its then-pending status.

0 commit comments

Comments
 (0)