additional quantization and throughput work - #4
Merged
Conversation
kwindla
commented
Aug 15, 2026
kwindla
commented
Collaborator
- Function head now quantized as GPTQ-W8.
- Tool call loop issue fixed
- Startup race condition fixed
- Test improvements
Skip unnecessary acknowledgement audio for fast tools and support concise, client-authored model context while retaining full tool results. Add benchmark and qualification coverage for the tool path, strengthen function-cycle lifecycle handling, and prevent pre-ready text input from deadlocking Pipecat startup.
Synthesis document covering the VoiceChat 11B model architecture (verified against checkpoint configs and the pinned NeMo Speech source) and every change this repository makes to sustain realtime inference on GB10, with Mermaid/UML diagrams and a browser wrapper (md-block + mermaid via CDN; serve docs/ over HTTP to view). Adversarially reviewed against primary sources; all 25 findings applied (docs/reviews/model-runtime-overview-adversarial-review.md). README no longer presents retired conditional PAD drafting as a current mechanism and attributes the first-open-full-duplex-tool-calling claim to NVIDIA. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Delivered speech-generation frames run ~83-86 ms against the 80 ms media budget on the current sequential image; the plan records the corrected history (PAD-pair retirement regressed generation ~11 ms/frame and was justified by the wedge alone), a normative measurement schema, release- aware playout gates verified to accept the retained pair-era schedule and reject the sequential one, and an ordered step plan with preregistered gates. Three review rounds to READY-TO-IMPLEMENT; all measured evidence appendices included. The step-5 and step-8 negative-control reports are copied in from the source-artifact tree byte-identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… report
Plan Steps 0-1. Offline analyzer implementing the plan-normative
populations/clocks with membership manifests and source hashes, the
release-aware playout replay (normative) plus zero-reserve debt
(informational), the literal K=12 structural gate, a 67-anchor verifier
with negative integrity coverage, and the checked observed-playout-trace
consumer. Step 1 corpus report over 14 retained artifacts generated
solely by the analyzer.
Qualified through five adversarial review rounds (step0 r1-r3, step1
r1-r2). The 38 MB machine-readable companion
reports/step1-retained-evidence-20260812.json is not committed: it
regenerates byte-exactly (proven in review) via
stratified_latency_analyzer.py analyze <14 pinned artifacts> \
--step1-details --markdown-output ...
SHA-256 9e25ffe7c664aa93509b28cd45aa11cf67122439fa728bc595a547841a13cc17
(pre-lint-format regeneration remains byte-identical).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan Step 2 (code phase). (2a) VOICECHAT_QUAL_NO_TEXT_WATCHDOG_OVERRIDE_FRAMES unblocks long-response latency fixtures behind a positive fail-closed qualification identity: explicit VOICECHAT_QUALIFICATION_MODE label (grammar-checked verbatim, 'production' refused) plus an environment equal to a checked-in candidate TOML except an explicit allowlist of deltas; authority is a consume-once validator-owned registry entry enforced at the engine boundary; visible in /health and per-frame watchdog snapshots. (2b) NEMOTRON_VOICECHAT_PLAYOUT_TRACE: observational playout JSONL (receipt clock at the WebSocket iterator, lifecycle filter before decode, bounded batched writer, atomic no-replace link publication with staging quarantine); AudioWorklet browser capture in the Playwright harness with batched export and per-worklet flush-ack ordinal verification; checked producer-exact trace consumer in the Step 0 analyzer. Off paths are functionally byte-identical (verified by code-object comparison in review). Four adversarial review rounds to PASS (docs/reviews/step2-implementation-review-r1..r4.md); live phase gated on the r4 preconditions and evidence-retention manifest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live fixture driver for the preregistered Step 2 campaign: canonical
hash-pinned eight-fixture plan, Pipecat-trace response attribution with
{connection}-tokenized per-session artifacts, minimal-anchor RTVI
lifecycle matching, descriptor-anchored session reservation, quiescence
barriers, and strict artifact-reconstructed summary validation. Seven
offline adversarial review rounds (docs/reviews/step2-driver-review-r1..r7)
plus eight live-discovered fake-vs-real fixes, each regression-pinned
against its retained real artifact (rounds r8+ and the implementation log
record the full arc).
Campaign completed on attempt 10 (session-20260813T083303Z): 8/8 fixtures
with terminal validation; stochastic long-script truncation confirmed as
model-level; I1 cancel-then-replace complete; first long-form reserve
sweep shows fixed reserves cannot clean long renditions (Step 3/5 input).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate 2 PASS (no no-text watchdog interference); I1 replacement PASS; Gate 1 FAIL campaign-wide per the preregistered rule (audio-complete long renditions L1 1/2, L2 2/2, capture-off 0/1 - stochastic model-level truncation). Text-vs-audio divergence demonstrated in both directions, validating the audio-adjudication requirement. Bulky raw artifacts (playout/browser JSONLs, WAVs, server traces) remain on-host under the sealed 35-entry SHA-256 manifest committed here. Step 2 of docs/generation-frame-latency-plan.md is complete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ad A/B Descriptive Step 4 inputs, admissible after three measurement-validity review rounds: thermal/DVFS/memory-pressure confounds show no supported association with frame cost (pinned cores flat at 3.9 GHz, no GPU clock collapse, associations vanish under detrending; single-sequence caveats recorded); client-layer trace overhead is a conditional bounded result and server trace cost is UNMEASURED pending the designed qualification trace-mode experiment. Corrected counterbalanced designs for 4c/4d recorded in the reports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ent) Preregistered counterbalanced incidence measurement across direct WebSocket, native Pipecat, and browser transports with the L1 long-PAD-tail fixture. Corrected valid population after three review rounds: 0 EarTTS-high in 764 delivered-nonBOS rows (direct 0/458, native 0/306, browser blocks invalid on fixture fidelity and registered as a prospective contract limitation). The historical spike mode did not recur within the preregistered exposure cap; verdict INSUFFICIENT with the retained-state forensics pivot standing. Demotes EarTTS-high elimination as a Step 5 candidate pending the pivot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sition The long-standing 'flaky' deferred-tool-timeout test was a real runtime liveness gap: a client observing the fatal terminal edge could cancel the endpoint before the outer cleanup submitted engine.abort(), losing the abort on the single-session stateful engine. Three review rounds produced the final design: every model-owning fatal branch routes through a transferred terminal-owner task that shields and drains the executor-side abort to actual completion before any observable terminal effect, safe under direct, two-branch-race, and all-task shutdown cancellation. The endpoint test asserts abort-before-publication causality at the send boundary; nothing was loosened. 10-consecutive-run green streaks and 67 anchors throughout. Step 4d: preregistered Nano interface decomposition on captured matched states - ten-block corrected result: content 53.56 ms wall / 45.85 ms CUDA-graph envelope / 7.71 ms residual (85.6%/14.4%); PAD 53.60/45.58/8.02 (85.1%/14.9%). Descriptively admissible; aggregate INCOMPLETE because the exact-output gate exposed fresh-engine graph-path replay divergence at call 39, which remains a prerequisite for any kernel-class candidate ranking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewed synthesis of all admitted Step 4 evidence with per-candidate evidentiary standing. No candidate holds ranking authority yet; the recommendation orders prerequisites: W8 graph-path reproducibility first, pairing wedge-closure as a parallel design-only track, Step 5 selection deferred until a row turns measured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of the reviewed 4c native-Pipecat runner (missed in the 4c commit): NEMOTRON_VOICECHAT_QUALIFICATION_FIXTURE selects the fixture system instruction; default behavior unchanged when unset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
REAL STATE DEFECT (preregistered classification, two review rounds): fresh graph-enabled engine construction for the released hybrid W8 Nano is bistable - engines land in one of two internally-stable states (38/56 production graph blocks exact; 18 identical call-39 divergences at a greedy boundary), while eager mode was 18/18 exact. Seed not isolated; confirmation cell recorded as prevalence/epoch evidence. Consequence: kernel-class ranking row remains blocked pending an exact-gate repair at the graph-construction level; bounded-divergence gate excluded; long- lived production instances carry a construction-time state qualification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Track 2 design study reviewed to PASS across two rounds: options (a) and (c) NO-GO on static fork-code grounds; option (b) wedge root-cause is the only bounded route with repaired preregistered experiment designs. Plan log records both concluded tracks and the three open funding axes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… costs 7.3ms/frame Preregistered (sealed 0f63864456889ad12b8758624036609f1911b2dba5d189af...) three-phase descriptive study of the Nano graph envelope. Measured GB10 ceiling 239.6 GB/s; the envelope streams ~10.1 GB/step at near-ceiling throughput, so intra-graph idle is 1.048 ms/frame and the wrapper graphing/fusion row screens OUT under its preregistered 1.5 ms floor. Dominant recoverable cost: the GPTQ-excluded modules run as bf16 cuBLAS GEMVs (9 instances/step) - function_head 7.292 ms median, attention projections ~1.9 ms - yielding a NEW quantization-extension candidate worth ~4.6 ms/frame (function_head W8 at measured Marlin throughput) to ~5.5+ ms/frame. Profile attempt 1 retained-invalid (call-39 divergent basin, further Step 5 defect prevalence); attempt 2 exact-basin 12/12 designated. Full 300MB raw evidence retained on disk; manifest committed (199 files). Service restored and identity-verified on stable fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g row Preregistered campaign (sealed 40a36a2c...) for the greenlit quantization extension. G0: RTN-exact GPTQ-W8 packing in shipped lm_head conventions; 100% argmax agreement on all 1,172 retained-capture hidden states; loader proves fail-closed on partial routing (two retained attempts; candidate definition amended to include modules_in_block_to_quantize). G1: zero decision flips - baseline exact-basin lane 15/15 vs retained capture, and matched-divergent design (recorded deviation) with 439,500 paired decisions / 0 flips across five candidate engines. New Step 5 defect evidence: candidate constructions selected the call-39 divergent basin 5/5 (checkpoint layout influences basin selection). G2: -4.437 ms/frame traced (54.986 -> 50.549), function_head bf16 GEMV (-7.21 ms) replaced by one Marlin instance (+2.51 ms), no other family regressed. Projected untraced content wall ~49.1 ms. G3 live/behavioral promotion still required before production. 289MB raw evidence retained on disk; manifest committed (161 files). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… rounds) B0-B9 boundary trace (mmap ring, 78.94ns disabled), atomic hash-verified patch-on-copy for the qualified vLLM escrow, and the v3 CPU protocol race campaign: exact qualified async topology incl. FCFS admission, 829,062 distinct realized interleavings across 1M schedules, genuine operation- loss injection 10/10 and record-suppression self-test 10/10, zero lost completions. v1 retracted (synchronous), v2 superseded (six confirmed defects); both retained. Reviews r1 (FAIL), r2 (FAIL), r3 (PASS) committed. Phase 1 GPU discrimination may now be designed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Candidate-backed service (earned-reproducibility candidate manifest, double-build byte-identical 13/13) ran the canonical 8-fixture browser campaign: 8/8, zero terminalization errors, categorical parity with the retained Step 2 baseline on every gate field. Function-calling 20/20 = 20/20 identical decisions vs baseline. Descriptive latency favorable (5/8 fixtures faster; long fixtures ~1-1.5s faster to first audio). Release service restored and stable-field identity verified. Remaining before production: counterbalanced A/B latency qualification + explicit promotion decision. Full session evidence retained on disk; manifest committed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Counterbalanced R1,C1,C2,R2 campaign, canonical 8-fixture plan per session, all first-try clean. Delta 5.452 ms/frame (84.12 -> 78.67), response-clustered bootstrap 95% CI [3.55, 7.20], seed 20260814. The structural K=12/960ms realtime gate flips REJECT -> ACCEPT on the candidate. Recorded deviation: normative analysis on pipecat playout traces matching the retained baseline artifact choice; server-clock analyses retained with identical contrast. Full promotion contract (G0-G3 + A/B) complete; promotion awaits explicit user decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promoted service: frozen production-candidate-1 env (61 keys + launcher HF cache keys), generation-step2 image, qualified fhw8 manifest/nano. First launch failed on the omitted launcher-added HF cache env (pocket worker asset path); retained, fixed, relaunched clean. Production identity + fhw8 provenance verified in /health. Pipecat bot on 0.0.0.0:7860, ngrok tunnel active. Follow-up recorded: signed release-chain assembly requires HF publication (user-owned). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…entical New tool_call_behavior_suite: replays retained live-session audio over the direct WS with production session config (turn segmentation, protocol-aware pacing, variation-tolerant loop/repetition metrics). Deterministic 3+3 A/B on the loop session's audio: candidate and release byte-identical in text, tool decisions, and the reproducible UserEouSettlementFailure fatal. Recorded defects for follow-up: greedy repetition/tool-loop under overlapped speech during function cycles; client-reachable settlement fatal on model self-start before EOU commit. fhw8 stays promoted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two live defects from manual testing, both with deterministic reproductions via the tool-call behavior suite: 1. Function-repetition watchdog in AgentSilenceEosWatchdog: counts model-level <TOOLCALL> emissions per response (the pipeline self-feeds tool responses, so the published-call budget saw 1 of 391); requests graceful agent EOS beyond 4. Unit-tested; limit 0 disables. 2. Pre-EOU activity recovery ladder ahead of the strict settlement fatal: dangling self-interrupted responses close administratively (fail_active_response), actively-producing responses drain to natural terminal (bounded), then bounded cancel escalation; the fail-closed rejection remains the backstop. The replay that deterministically fataled at turn 10 now completes 3/3 with zero fatals. Suite gains protocol-aware turn pacing, model_output negotiation, and graceful fatal reporting. defect-fixes-1 overlay image serves the promoted fhw8 production stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ding The typed input path owns deliberate cancel-then-replace interruption semantics, so the settlement recovery applies only at the microphone commit site where the original fatal occurred. Chrome-path validation: S1/L1/L2 6/6 on the fixed image; the I1 timeout reproduces identically on the unmodified generation-step2 control (fixes exonerated) and on a fresh service (age refuted) - isolated as a pre-existing I1 anchor-order sensitivity (original-answer early stop vs interruption marker, doubled bot-interrupted signature) and recorded as an open item with retained evidence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The I1 predicate required the original answer's stop to follow the interruption marker; a stochastically truncated original answer can stop one event before the marker registers (retained 2026-08-14 stream), making completion unmatchable despite both answers finishing. The matcher now accepts either ordering while still requiring two full answer lifecycles and the replacement to follow both the marker and the original stop. All 19 retained-trace I1 tests pass unchanged plus the new inversion regression from the real failing stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Order-tolerant anchors moved the failure to the driver's strict one-edge/clear-release lifecycle contract: early-truncated original answers leave no mid-speech interruption edge. Recorded as a qualified-driver follow-up coupled to the stochastic-truncation known limitation. Defect-fix browser validation stands: 6/6 non-interruption fixtures, 3/3 WS fatal reproduction, control-exonerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ergence Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase-targeted FC interruption harness (pre-response / during-injection / during-ack / control x3): every cell one bounded tool call, zero fatals, second answer lifecycle after every barge-in. First live exercise of the settlement recovery across FC phases. Negative control: mismatched tool declined 12/12 under the production instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real executor stack (5 adversarial review rounds), A-lane and post-drain KV adjudications, five-construction calibration (fail-closed REJECTED), and the two sealed findings that disqualify pairing-as-retained from exact-replacement revival: packed-scan recurrent drift and the new post-rejection drain committed-KV non-equivalence bug-class. Phase 1 campaign unscheduled; pairing revival now requires a pair-implementation drain-path repair + requalification. Large binary evidence retained on disk; small artifacts and all seals committed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Manual testing PASSED the promotion candidate. This commit makes the repository state complete: - deploy/promotion-candidate/: the actual production start scripts (defect-fixes-1 fhw8 script now serving; step2-image and release variants; qualification v4; overlay Dockerfile) with SHA manifest - previously only in session scratchpad. Note: the fixed script hash (5ff1efaa...) supersedes the promotion record's 060cf095... after the defect-fixes-1 image tag swap. - Final wedge trace/patcher source revisions from the executor rounds. - 222 small retained-evidence files (session health/exit/trace-list records, adjudication receipts, preflight identities, campaign summaries) across the A/B, G0-G3, toolloop, browser-validation, final-e2e, and step5-wedge trees. Large binary evidence (audio, nsys, sqlite, batch ledgers) remains on disk, pinned by committed manifests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HF release revision fdf70edf9d8b... (537MB new LFS data after dedupe); inventory verified with the bootstrap's own verify_release before upload and by download round-trip after (release_sha256 426eecc0..., nano composite dd4ce6a9... matching the qualified candidate manifest). Adds production-candidate-3.toml + fhw8 release pointer, portable launch scripts, the deterministic derive-fhw8-candidate.sh with fail-closed qualified pins, the inventory builder, and the runbook. Follow-up: fresh-machine candidate-3 bootstrap-state swap test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.