Behavior fixes, throughput improvement, and startup time improvements - #1
Merged
Conversation
kwindla
commented
Aug 10, 2026
kwindla
commented
Collaborator
- Post-function-call pair recovery. Fixed bug that could lead to model state corruption after a tool call.
- Dedicated PAD-pair CUDA graph. Added a hash-qualified FULL CUDA graph for the exact Nano PAD-pair shape. This is the two-token speculative decoding code path that gets us down to <160ms average inference time for two 80ms audio input chunks, on DGX Spark.
- Exact EarTTS constants. Precomputed exact constants for EarTTS (stage A, pre-padded rvq_embs), qualified by A/B comparison. (Stage B, cached norms, was excluded because it regressed.)
- Baked cold-start cache. The assembled image ships a freshly generated vLLM compile cache (built by cold-booting the assembled sources, ~365 s, then baking the cache tree into the image) under an explicit capture contract, with a hashed manifest and provenance records. This cuts startup time from approximately 7 minutes to approximately 5 minutes.
Hash-qualified vLLM patch for a query_len=2 FULL CUDA graph, container build, and bitwise gate tooling. Checkpointed as-is after the 2026-08-07 host OOM killed the first gate run (gpu_memory_utilization=0.55 default, no container memory limit); gate has not yet passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.