More EOU settlement work (and other stuff) - #5
Merged
Conversation
Fence-on-real-audio + revocable early arming in the settlement path, with the RNNT fence target and every terminal contract unchanged. Queued sub-gate microphone audio now advances the blank fence (barge-in stops and requeues fail-closed); blank frames observed between speech-stop and commit pre-arm the fence; explicit model-input provenance confines arming to complete pre-commit microphone frames (round-1 review's commit-edge padded-frame defect fixed, its reproduction committed as a regression test). Rapid-turn gates (regenerated post-fix): zero empty responses in 80/80; six-turn median commit-to-first-audio flat at 894->753 ms vs baseline 861->2,919 ms; 78/80 settlements on real queued audio; full battery and canonical browser plan 8/8 green. Two adversarial review rounds; r2 PASS. Evidence under reports/fence-latency/ (manifest 4956d89c...). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The raw baseline/candidate-r2 session streams (~5MB each) join the already-committed lossless projections and manifests so the complete V1/V2 evidence travels with the repository per the round-1 review requirement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
G0 exact, G1 zero flips (10,548 decisions), G2 -1.546 ms/frame - but the interruption matrix elicited 0/12 tool calls (fhw8: 12/12, identical carrier), deterministic: RTN-W8 attention quantization collapses tool-call propensity while frozen-corpus decisions stay exact. Candidate retired per the sealed preregistration. Finding: the tool-affordance decision is attention-precision-sensitive, and the G1 corpus cannot gate it (no tool contexts) - future interior campaigns need tool-decision replay up front. Calibrated GPTQ remains an unsealed future option for the ~1.5 ms. Builders + full evidence committed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ence Plan sealed after 3 review rounds; campaign stopped before Seal-1 with no Hessian, candidate, or gate result. Two findings: 1. Pooled selection (steered mid-campaign) validated 48/48 scenarios in ONE baseline (24/24 positive, 24/24 no-call, min similarity 0.889): the 39 prior Seal-0 revisions were ASR transcript-exactness artifacts, not tool-propensity fragility, and sequential replacement was an adaptive-selection bias the sealed pool removes. 2. NEW: the same audio is an exact no-call under the packed production runtime but emits function tokens under eager reconstruction (a2-nocall-r25-01) - eager/graph categorical disagreement on tool decisions, invalidating eager-mode activation capture as a basis for calibrating tool-relevant behavior and joining the Step 5 basin defect family. Three prospective criterion deviations recorded and explicitly submitted for results-review adjudication. Full evidence + seals committed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The committed start script pinned defect-fixes-1, a local-only tag built before the fence work - following the runbook would have launched WITHOUT the qualified settlement changes. Adds build-promotion-image.sh (builds from the current checkout via container/build-public-runtime.sh, tags a canonical promotion-candidate image, and fails closed unless the built image's server.py hash matches the working tree) and repoints the start script at that tag, overridable via VOICECHAT_PROMOTION_IMAGE. README now states plainly that the qualified behavior is in the runtime source, not the weights. No weight change: fhw8 is already published (HF fdf70edf, pinned by config/production-candidate-3.toml on main) and this branch qualified no new artifact - both attention-W8 candidates were retired. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces get_current_utc_time with two tools: - get_current_time: reports the time in the process-wide reporting zone (default UTC), spoken 12-hour with zone abbreviation - set_timezone: sets that zone for later calls, accepting IANA names or spoken aliases (Pacific, Eastern, Tokyo...), case-insensitive; unknown zones return an ok:false result the model can recover from conversationally rather than failing the turn State is process-global per the demo's design, so a set_timezone call is observed by any later get_current_time call. tool_freshness_contract.py keeps its frozen v1 get_current_utc_time definition (retained qualified campaign artifact, deliberately untouched). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… blindness Root cause (pre-existing): no_text_frames accumulated while EarTTS was still rendering audible audio, so responses whose audio tail exceeded the 30-frame text-silence deadline were cut off mid-speech. 4 of 5 firings in the reported session were acoustically live. All qualification ran with the 160-frame override, so no gate exercised production behavior. Repair: audible audio now resets the text counter (two-dimensional liveness), with a real enforced ResponseWallClockDeadline (30s, derived as 2.29x the longest observed valid response) as the global bound. Shipped as a new versioned production-hotfix-notext-watchdog-v1 identity; the frozen candidate-1/2/3 TOMLs are byte-identical to HEAD and the published fhw8 pin is untouched. Test matrix: response_completion_gate.py (strict total parse, fail-closed schema, source-derived event vocabulary with drift regression) FAILS the retained pre-fix trace and PASSES all eight repaired traces; browser adjudication reports a real aggregate and classifies every mismatch so watchdog truncation cannot be absorbed as model-silence; production-value runs; fail-closed promotion manifest verifier over 8 launch files. Six adversarial review rounds; each found a real in-scope defect. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… gate From the user's manual session (trace c42c6701), four issues triaged: I1 (fixed): the completion gate failed on legitimate disconnects. Open responses are now recorded as session_terminated only on POSITIVE evidence - a matching producer terminal claim after BOS, or the explicit unclaimed transport disconnect lifecycle. Unknown/typo reasons, clean stops, duplicate stops, malformed lifecycle and post-stop records fail closed (review caught an earlier not-equal-to-client_stop formulation as fail-open). I2 (fixed, root-caused): responses opening after a tool call carried empty user_text - exact correlation, 39/40 preceding frames in an active function cycle. The committed turn transcript lived only in the mutable RNNT accumulator, which resets for the next turn before the deferred post-function-cycle BOS fires. The runtime now freezes transcript and source identity at commit, and clears the superseded binding at the locked unpublished-cycle interruption edge (review caught that the first version could bind the WRONG turn's text - worse than the original bug). Endpoint tests pin deferred-FC -> interruption -> pre-commit BOS, typed and microphone function output through settlement/EOTR/forced BOS, and a two-tool continuation. I3 (measured, model behavior): all three mid-sentence endings classify as model_silence or model_turn_taking; zero runtime-caused. No boundary contract was loosened. I4 (NOT fixed): most-recent-result instructions added to both the runtime tool grounding and demo prompt; directive fixture shows 0/3 passing in BOTH arms, i4_fix_supported=false. The stale-result behavior is model behavior in the documented confabulation family. Acceptance harness: session_replay_browser_suite.py replays the retained session audio through real Chromium with a fake mic and gates completion, attribution, audible-watchdog, runtime-I3, and qualified-ASR WER (corrected run passes all five, WER 0.075). Tool-sequence/freshness are reported as propensity metrics, not gated, because they measure model behavior: notably, fixing I2 changed observable tool-calling on casual re-asks (the live session's third call fired on a turn with EMPTY text; with text correctly bound the model answers from memory instead). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Independent review (Codex R2) found the I2 repair incomplete: the typed request path had no ownership transition, so a typed request arriving while a function cycle was deferred-but-unpublished was accepted, mutated the model without interrupting the worker, and its BOS consumed the superseded microphone turn's binding (reproduced through the real endpoint). The typed path now takes the same locked transition: interrupt the unpublished cycle and clear the superseded binding, leaving the published-call rejection and publication-race branch unchanged, with an endpoint test driving microphone commit -> deferred FC -> typed request -> typed BOS. Also: re-promoted the runtime image so host and in-image server hashes match (the prior promoted image omitted the final repair), regenerated browser capture/manifest records to hash their FINAL reports (one additional stale pair found), and replaced the helper-only manifest regression with a mocked end-to-end run_asr test proven to kill the old-generation-order mutation. Reviews: Codex R2 (FAIL, found the typed hole), Claude R2 (PASS - wrong, correction recorded in the review file), Codex R3 (I2 + promotion confirmed correct). Two known suite failures: the non-evergreen sealed revision guard and the intermittent real-Chromium round trip (passes in isolation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Disconnecting and pressing Connect again without reloading the page never produced a bot-ready line. The browser sat on "Connecting..." forever. Root cause is client-side, not server-side. pipecat-ai-prebuilt 1.0.5 ships @pipecat-ai/small-webrtc-transport 1.10.6, whose DailyMediaManager.connect() stores its promise resolver in _connectResolve, awaits initialize(), and then returns without invoking it. The only call to _connectResolve() sits behind a _mediaStreamRecorder.begin(track) guard, and SmallWebRTC constructs the manager with recording disabled, so that path is unreachable and the promise stays pending. SmallWebRTCTransport._connect() therefore never reaches startNewPeerConnection(), so no peer connection is built and no SDP offer is ever sent. The first connect works because the UI initializes devices on mount. Disconnect clears the manager's _initialized flag while the client's device state stays granted, so needsInit() is false, initDevices() is not called again, and the next connect takes the broken branch. Instrumented Playwright runs against the live stack confirmed the server was never at fault: every disconnect logged "Discarding peer connection for pc_id", cancelled the worker, and no reconnect took the connection-reuse path. An earlier server-side change built on that mistaken diagnosis was reverted. Apply the upstream correction to the installed asset before starting the bot, pinned by SHA-256 and idempotent. An unrecognised bundle raises rather than patching unknown code, so an upstream change surfaces as a hard failure instead of a silent browser regression. Verified end-to-end in real Chromium: connect, disconnect, wait 5s, connect again without reloading now yields two bot-ready messages and two server sessions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The A0 campaign's narrative was committed through r40 (seal0-revision reason notes for r2-r40), but the per-round structured evidence stopped at r10. Commit the missing 468 records for r11-r40 so the retained trail matches the campaign it documents. Also stop tracking the raw inputs those records were distilled from: ~90 GB of per-call tensor dumps, profiler traces, run logs, and event streams under reports/attnw8-gptq. They were never tracked, but nothing ignored them, so git status was unusable and `git add -A` would have committed 90 GB. The ignore rules are scoped to this campaign rather than written as global extension patterns. attnw8-qualification, session-repair, and the step4/step5 data sets deliberately track 541 .nsys-rep/.sqlite/.log/.pt files as evidence, and reports/attnw8-gptq/**/tool-capture/raw/metadata.json is itself retained evidence living inside an otherwise-raw directory, so blanket patterns would have contradicted the existing retention policy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Audio crackles during long bot responses. Two timestamp-based models disagreed about why by nearly two orders of magnitude: mine predicted one ~161 ms dropout per bad response, the independent Codex assessment predicted ~62 discrete 10 ms silence ticks. Sixty small splices is what crackling sounds like; one gap is a dropout. Neither model observed an actual underflow, because RawAudioTrack keeps no counter. Count them directly. The wrapper reproduces recv()'s own pacing wait before sampling the queue: recv sleeps until the frame is due and only then tests the queue, so sampling earlier would count chunks that arrive during the sleep. Silence emitted between responses is expected and must not be counted -- a naive counter reports ~100% underflow on any session where the bot is quiet, which the first version of this did. Only a queue that empties while audio is still streaming splices silence into speech, so that is tallied separately and grouped into runs, since two four-tick runs are two audible artifacts rather than eight. Also reclaim lead that was being discarded before audio became playable, neither costing latency: - audio_out_10ms_chunks 4 -> 1. BaseOutputTransport forwards only complete chunks, stranding up to 40 ms of generated audio in its byte buffer. - Keep the SOXR stream warm. Its clear_after_secs default of 0.2 s is always exceeded between responses, so every response resampled on a fresh stream and re-paid the initial filter latency exactly when lead is scarcest: the first two 80 ms deltas expose 135.5 ms rather than 160 ms, and the 40 ms chunker then exposes only 120 ms. Measured, not assumed, from a traced session: the 160 ms prebuffer is confirmed active and re-armed on every one of 9 responses (reason=threshold, frames_released=2, sample_count=3528). Every response opens with exactly 3 silent model frames, so the prebuffer always holds pure silence -- but queue depth at speech onset is 150-170 ms, so trimming that silence buys nothing. Internal silence available for catch-up is zero frames. The stateless-resampler click hypothesis was refuted independently by both reviews. Telemetry is opt-in via NEMOTRON_VOICECHAT_TRACK_UNDERFLOW_TELEMETRY so production behaviour is unchanged by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit set audio_out_10ms_chunks=1 on the reasoning that
BaseOutputTransport strands up to 40 ms of generated audio in its byte buffer.
A/B against 90 s of identical replayed speech says that was wrong:
chunks=1 -> 233 in-speech ticks, 19 runs, depth histogram {1: 2168}
chunks=4 -> 215 in-speech ticks, 14 runs, depth histogram {1:667 2:667 3:668 4:668}
Smaller chunks are mildly worse, not better. The value is not only how much
audio waits in the byte buffer; it is also the only slack absorbing timing
jitter at the hand-off to the WebRTC track, and cutting it from 40 ms to 10 ms
removed that slack. Restore Pipecat's default and expose the knob via
NEMOTRON_VOICECHAT_OUTPUT_10MS_CHUNKS so the tradeoff stays measurable.
The warm SOXR resampler is unaffected and stays: it reclaims filter latency
without touching the jitter budget.
Recorded so it is not re-derived: the direct counter shows 215-233 in-speech
ticks per 90 s session, while the model-side generation deficit over the same
sessions accounts for only 35-56. The excess is not yet explained, and
BaseOutputTransport does not pace its own output (no mixer configured, so
_next_frame yields on arrival), which rules out drift between an output clock
and the track clock.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rawn The independent R2 assessment (docs/reviews/audio-crackle-root-cause-r2.md) found a deterministic boundary error in the counter added two commits ago. It called any zero within a fixed 0.3 s grace period of the last chunk "in speech", which is not a speech state machine: after the final chunk of every response the queue stays empty, so the entire grace period was counted. That manufactured ~30 ticks and one run per response, roughly half of every number reported, including the 722 ticks measured on a human session. A zero is now attributed to a gap only once more real audio arrives to close it. Terminal drains are never followed by more audio in the same stream, so they are excluded by construction rather than by a timing heuristic, and are reported separately as tail_ticks_discarded. Re-measured on the same 90 s replay: 2410 tail ticks removed and in_speech_runs falls from 14 to 6. Also withdraw the chunk-size claim from the previous commit. R2 shows the n=1 A/B was confounded: the arms differ in response count, length, and generation debt, and the two traces predict a ~21 tick difference before any chunk-size effect is credited, which exceeds the 18 tick gap observed. The revert to Pipecat's default stands, but on the grounds that changing to 1 was unjustified, not that 4 was shown better. A sound test needs frozen output PCM replayed with identical arrival timestamps, paired AB/BA order, and repeats. The counter is still not trustworthy enough to tune production against. The remaining 6 runs may include inter-response gaps shorter than the 1 s window. Per R2, the fix is explicit response lifecycle and content state rather than another timing heuristic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The counter's job is to adjudicate whether the WebRTC output track splices silence into speech. Every revision so far decided that from timing, and every one inflated the total. R2 found the first class (a fixed grace period counted each response's terminal drain). Requiring audio to resume fixed that and left a second: a short inter-response gap promoted by the next response's first chunk. Replacing timing with explicit response state exposed a third, which no timing rule could have caught. BaseOutputTransport.handle_tts_stopped flushes the trailing partial chunk padded with silence and queues it ahead of the stop frame. It arrives 550-670 ms after the response's real audio has drained -- within 9-10 ms of response.done in all four observed instances -- and is indistinguishable from audio resuming, from inside the same response. So the epoch is fenced at every boundary of a contiguous TTS stream, not just at response start: the tagger sits upstream of the output transport and sees TTSStopped before the flush it triggers is enqueued. A run of zeros now counts only when audio resumes under the same epoch. The time window survives as a backstop against a missing epoch signal and fires in neither run. Measured on a 95 s replay of a captured session, twice: zero in-speech ticks, zero runs, a clean 1/2/3/4 depth sawtooth. Transport underflow is not the cause of the crackling on this path. A zero needs a control, so there is one: withholding audio from the track for 200 ms every 20 writes yields 26 runs of 19-20 ticks against 26 injections. The instrument reads zero because there is nothing there, not because it cannot read. The replay is headless against 127.0.0.1 and exercises the server-side playout path only. The network path and the browser's own jitter buffer and playout are untested, and are now the leading candidates.
A live session produced two responses the user heard as incomplete. Pinned offline ASR over the exact emitted PCM confirms it: response 2 spoke 22 of 41 generated words, response 3 spoke 68 of 86. Both losses are clean tail truncations at a sentence boundary. An earlier characters-per-second argument pointed the same way but is superseded and should not be cited. This is not the no-text watchdog repaired in 9c1b87b. At all three firings the no-text and no-audio predicates stood at 20/30 and never reached threshold; the decoded-silence predicate fired at 20/20, which that fix deliberately left alone. Nor is watchdog firing equivalent to truncation: response 5 fired the same watchdog and is complete. In observed order the watchdog is downstream of the loss. What the retained evidence cannot settle is whether EarTTS would ever have resumed past 20 silent frames, which is the counterfactual that decides whether the guard is also a truncating timeout. The paired analysis adds two corrections to my first pass: the tail RMS shape does not identify which mechanism closed a response, and response 5 was not in a tool-wait state, so the truncation and the missing tool calls do not share a cause. Raw acoustic codes were not retained, so the final split between a model silence decision and a lower decoder/codec fault is not yet measurable. The clips are kept against the repo's blanket *.wav rule because the trace they were cut from lives outside the repo in state that rotates.
… trace Two outages have now wiped the scratch copy of this page. It reads only the retained playout trace and the committed transcripts, so it can be regenerated from evidence instead of from a scratch directory that does not survive a reboot. The rendered page itself stays untracked; it is derived.
One session established that truncation happens; it cannot establish how often. With a two-in-seven observation, a lucky run would look like a fix, so any repair needs a baseline to beat. This drives repeated sessions, transcribes what each response actually said with the pinned offline evaluator, and diffs that against the generated text, reporting counts rather than a rate from a single run. Responses the user interrupted are excluded; only a completed response that lost its tail counts as a defect. The first run exposed a fault in the harness rather than the stack. The replay ran 95 s against an 88.8 s fixture, so the microphone fell silent, the pipeline idle timeout closed the session, and the final response was cut short by the measurement itself. The replay is now clamped to the fixture length, and the probe treats a self-ended session as a normal finish instead of an error.
…e measured Root-causing the two responses that lost their spoken tails, by elimination under measurement rather than argument. Each layer below the model is now cleared: the WebRTC transport underran zero times with a negative control proving the counter can see a fault; the 20-frame decoded-silence guard never withheld recoverable audio, since no response resumed even with thresholds raised to 100; the PAD-tail policy fires 81 frames after the audio is already gone; the CPU codec reproduces production bit-exactly and renders the same silence under pointwise-linear off, persistence off, and the native reference; and the text feed delivers every response token-for-token, missing suffixes included, with delivery frames identified. What remains is EarTTS's PAD-conditioned acoustic continuation ending an utterance with rendering still owed. There is no alignment pointer, finished flag, or rendering-debt output in the deployed model, so a runtime detect-and-recover mitigation has nothing to trigger on. Acoustic sampling is also stochastic, via Gumbel-Max and MaskGIT noise on a worker-wide RNG with no per-response seed, which is why the same prompt truncates only sometimes and why no single response can support a claim about any candidate cause. The production baseline is 1 truncation in 7 completed responses, and 1 in 3 long responses. Long responses are where the risk concentrates, and this is the number a fix has to beat. Raw campaign captures are scoped out; the conclusions, summaries and per-case verdicts are kept, along with the tooling that regenerates the rest.
Tooling built to root-cause the response tail truncation. All of it is diagnostic-only and cannot alter production: every path is gated on environment variables absent from the frozen contract, the guard and deadline overrides fail closed unless their attestation is supplied, and the diagnostic manifest kinds are accepted only when the matching config hashes are present. The acoustic capture clones the true raw generated code before the first PAD policy runs. That distinction is the whole point: the baked Speech capture receives its code only after the policy may have substituted silence, so its historical raw_generated_code argument is not raw, and any earlier diagnosis reading it as the model's output was reading post-policy data. The replay driver gains a closed-loop mode. The full-precision checkpoint needs it: at a p95 of 139.5 ms against an 80 ms frame budget the model cannot be driven by a wall clock, and a wall-paced run would trip the response wall deadline and manufacture terminations that mimic the defect being measured. Also corrects a launcher default that silently set the decoded-silence guard to 100 frames whenever capture was enabled. Two early capture campaigns ran under it; capture now changes no guard unless an override is passed explicitly.
Long responses lose their final sentences. The cause is EarTTS acoustic generation entering a silence regime with text it has already received still unrendered, and it is not something this project can fix. Everything below that boundary was eliminated by measurement: the WebRTC transport never underran while speech was live, with a negative control proving the counter can see the fault it reports as absent; the decoded-silence watchdog withholds nothing, since no response resumed even at a 100-frame threshold; the PAD-tail policy substitutes 81 frames after the audio is already gone; the codec reproduces production bit-exactly and renders the same silence under every variant including the native reference; and the text feed delivers each response token-for-token, missing suffixes included. Quantization was the last hypothesis we controlled, and the user was right to ask for it. The stock public checkpoint at full precision, both components, truncated 17 of 20 story responses against a released-stack envelope of 20/40. It is exonerated as a necessary cause, and it is not a fallback either: full precision runs at 1.74x the frame budget. No runtime mitigation is available because EarTTS exposes no rendering-debt signal at all - no text cursor, finished flag, or outstanding-text output - so detection has nothing to trigger on. The document reports counts rather than rates throughout, since acoustic sampling is stochastic with no per-response seed, and it records the caveats that limit each claim, including two early campaigns that ran a non-production decoded-silence guard and the pacing variable that remains uncontrolled. Raw campaign bulk is scoped out; the conclusions, per-case verdicts and the tooling that regenerates the rest are kept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.