v0.19.0 — PPL parity proven, long-context speculation, 1173 tok/s @16

Highlights since v0.18.1 (2026-07-10):

Quality / correctness

• First cross-engine PPL-parity measurement (GOAL release bar 1, docs/audit/ppl_parity_2026_07_12.md): with the LM-head opt-out, imp is at parity (−0.8…+0.2%) with llama.cpp b9976 on every comparable GGUF hero. Two fixes fell out: the qwen35 pre-tokenizer routed to the gpt2 fallback (+13% tokens on Qwen3.5/3.6 GGUF prompts — now byte-identical to llama.cpp) and the NVFP4-LM-head opt-outs were dead on GGUF checkpoints. The gemm.nvfp4_lm_head speed/quality trade is now listed in the GOAL bar-1 trades (#982 tracks the default-gate question).

Performance

• Dense n-gram speculation now WINS long context (#964 stages 1+2): +45% at 512 ctx, +27% at 13k, neutral at 16k (was −8…−62%).
• FP8 KV auto-enables on hint-less Qwen3 dense/MoE GGUFs: +41% decode at 16k ctx, PPL-gated neutral.
• Batched serving with 16 concurrent requests: 861 → 1173 tok/s sustained (above the published vLLM reference at TPOT parity), streaming +6% via the one-step-in-flight decode pipeline.
• Suspend/resume (/admin/suspend), on-disk warm weight cache (cold boot 68→21 s), KV-floor fix (35B at 16k ctx without streaming: 72.7→264.3 tok/s).

Competitive (re-swept 2026-07-12 vs llama.cpp b9976 e3546c794, same day, pp512/tg128)

• Dense GGUF decode +42–48%, Qwen3.6-35B hybrid UD-Q4KM +18%, Gemma-4-26B MoE +21%; NVFP4 SafeTensors uncontested (30B-class MoE 271–390 tok/s).
• Honest note: llama.cpp's MXFP4/MoE-GGUF decode caught up — gpt-oss-20b is now a statistical tie (#984).

Full details in CHANGELOG.md and docs/BENCHMARKS.md (commit-anchored).

v0.18.1 — decode-graph ctx-topology wedge fix + FP8 SSM decode sidecar (Qwen3.6-35B +19%)

Fixed

• Decode CUDA graph re-derives its launch topology when the context high-water mark grows — a long-prompt request after short ones no longer wedges the engine with an illegal memory access; the full degeneration suite passes against the Qwen3.6-35B server for the first time (#948, #950).
• /v1/responses streaming: metrics + idle keepalive (#941); packing-aware pre-upload KV reserve for 4-bit KV dtypes (#942); workspace_estimate() no longer charges the S-matrix on FA2-served configs (#943) — all via #945.
• Streaming drift bugs surfaced by the SSE-loop unification (#951): responses emitted empty buffered tool-call argument deltas and hardcoded reasoning_tokens: 0; messages streams missed the inter-token metric, streamed-args bookkeeping, and request cancel on a dead keepalive; think-budget enforcement never engaged on messages/responses.
• Explicit enable_thinking: true is honored on templates that default to a closed think block (#939 chain).

Added

• gemm.fp8_ssm_proj (default ON): FP8 E4M3 per-row-scale decode sidecar for native-precision GDN/SSM projections on NVFP4 hybrids — Qwen3.6-35B decode +19% (tg256 268.6 → 320.3), PPL flat (#949).

Changed

• Shared per-token SSE stream driver: the triplicated outer streaming loop (chat/messages/responses) and the 4× hand-copied imp::Request mapping are single-sourced; net −732 LOC (#951). Follow-up single-sourcing: ModelConfig::ssm_conv_channels() (9 sites) + once-per-init native-cache-demand scan (#952).
• Docs consolidated under docs/; root holds only standards (#946). Four unreferenced March-era one-off tools removed (#947). Structural audit #6 landed with its cleanup (#944).
• Maintainer home paths sanitized from 24 tracked files ($HOME/Path.home()); scripts/check-release.sh fully green again (#953).

Full changelog: CHANGELOG.md

v0.18.0 — HD=256 FA2 default-on + Qwen3.5-4B-mxfp4 correctness fixes

Minor release: the headline is a default-on feature — attention.fa2_hd256 now defaults true, so head_dim=256 models (Qwen3.6 hybrids, gemma-3-class) route prefill through the register-resident f16-QK FA2 kernel (pp4096 +26%). Plus a batch of Qwen3.5-4B-mxfp4 correctness fixes.

Changed

• HD=256 FA2 default-on + FP8-KV deterministic forcing lifted (stage 3) (#932). Single-shot FA2 for uniform GDN/Mamba2 hybrids at any head_dim; learned-sink (gpt-oss) and heterogeneous (gemma-4) models keep cuBLAS.

Added

• Stage-1 HD=256 FA2 port attention.fa2_hd256 (#930): register-resident hd=256 kernel, 4.3× vs WMMA FMHA, e2e prefill +10.6% pp4096 / +24.8% pp8192, no PPL loss.
• Graph-captured verify for hd=256 GDN hybrids (#933).

Fixed

• Qwen3.5-4B-mxfp4 answer trapped in reasoning_content (#937): enable_thinking is now reconciled against what the chat template actually rendered into the prompt tail (open <think> → on; pre-closed block → off). Genuine reasoning models unaffected.
• Qwen3.5-4B-mxfp4 CUDA-graph capture status-14 abort (#937): the capture-safe sm_120 WMMA GEMM now accepts narrow N (the GDN N=32 projection) instead of falling to cuBLASLt.
• MXFP4-GDN !!! garbage when VRAM is tight (#935): the VRAM budget now reserves the mandatory MXFP4→FP16 decode fallback, and an oversubscribe fails loud at load instead of serving garbage.
• Deterministic cuBLAS GEMM validates its algo choice; a totally-failed GEMM is now fatal (#929).

Full detail: CHANGELOG.md.

v0.17.3 — reserve mandatory NVFP4 decode caches before workspaces + KV

Native-NVFP4 serving fix release: the mandatory decode caches are now physically reserved before the elastic VRAM consumers, so large NVFP4-prequant MoE models reach full decode-cache coverage + captured decode graphs under pure default config.

Fixed

• VRAM ordering: mandatory NVFP4 decode caches reserved before workspaces and the KV pool (#926). The CUTLASS SfAtom SF slab (~2 GB) and the nvfp4_moe decode cache were built last from already-starved free VRAM; partial caches abort decode CUDA-graph capture (one uncovered MoE layer → host-args path → capture throws), pinning Qwen3.6-35B-A3B-NVFP4 at 26–40 tok/s under default config. A balloon allocation right after weight upload holds the exact demand (new compute_native_cache_demand, sized with cutlass_nvfp4_sf_size and now including the GDN/SSM projections the old estimate missed) until the cache build, and the phase-3 budgets are floored at the balloon-backed guarantee (live cudaMemGetInfo lags async frees). Default config now reaches full caches + captured decode graph: 247–249 tok/s with a 138k-token KV pool (was 26–40). Non-prequant (GGUF/FP16) budget arithmetic is unchanged (pinned by test); escape hatch [vram] native_cache_reserve (default on). A post-build coverage log states FULL/PARTIAL cache status and remedies.
• Loud WARN when the KV pool collapses below its token floor (#927). With an oversized max_batch_size the batch-scaled workspaces can shrink the KV pool to the 16-block minimum; longer requests were silently cancelled at admission while /v1/models kept advertising the full context. The budget planner now warns with the real pool size and remedies. Log-only.

Perf baseline untouched (no perf change for baseline GGUF models).

v0.17.2 — context-window auto-detection probes

Small server-compatibility release. The served context window is now discoverable through the three field conventions OpenAI-compatible clients already probe, so they can auto-detect it instead of keeping a hard-coded table. Server-only, no functional change to inference; perf baseline untouched.

Added

• Context-window auto-detection across the three live conventions (#921):
  GET /v1/models carries the context length as vLLM's max_model_len and llama.cpp's meta.n_ctx_train on the model object (plus created for OpenAI compliance); new GET /props (llama.cpp shape — n_ctx) and GET /info (TGI shape — max_total_tokens / max_input_tokens). All three report the same engine-detected max_seq_len.

Docker: docker pull ghcr.io/kekzl/imp:0.17.2 (also tagged 0.17, 0, latest).

v0.17.1 — adopt C++23 idioms (to_underlying, deducing this, static operator())

Follow-up to the v0.17.0 C++23 toolchain bump: now that the whole tree builds as C++23, this patch release adopts the C++23 language idioms that genuinely fit the codebase. Behavior-neutral — identical values, same accessors, same functors; decode verified coherent, perf baseline untouched.

Changed

• Adopted C++23 idioms across the tree (#919):

  • std::to_underlying at ~67 real enum-to-underlying cast sites (<utility> added per TU — it is a hard compile error on non-enums, so the build itself confirms every site).
  • deducing this collapses four duplicated const/non-const accessor pairs into one overload each (Model::layer, SchemaConstrain::top, jinja Value::as_object, WeightRegistry::handle).
  • static operator() on six stateless functors (two host hash functors + four __device__ activation functors — nvcc 13.3 accepts static operator() in device code).

  Deliberately not adopted: std::expected (throw-based error model), std::mdspan/std::print/std::generator (device-side nvcc limits), [[assume]] (proven-inert codegen on the NVFP4 GEMV decode path).

Tests

• Cover format_tool_response + reconstruct_tool_call_output (#914).

No functional change, no perf movement — tests/perf_baseline.json untouched.

v0.17.0 — C++23 / Ubuntu 26.04 toolchain + FP8 tile attention + accumulated fixes

Toolchain-modernization release: the engine now builds as C++23 on an
Ubuntu 26.04 / GCC 15.2 / CUDA 13.3 base (was C++20 / Ubuntu 24.04 / GCC 13).
The standard bump changes no default-path behavior and no perf (decode verified
neutral). Ships alongside two FP8 tile decode-attention kernels (large
long-context wins), the MLA/MTP RoPE correctness fixes, an async-mempool teardown
fix, and the server-hardening / config / VRAM cleanups from the 2026-07-07
structural audit.

Changed

• C++ standard raised to C++23 (host + CUDA). CMake's NVIDIA-CUDA module has no
  CUDA23 dialect flag, so the build teaches it -std=c++23 explicitly (shim to drop
  once CMake ships a native mapping). No source changes were required (#916).
• Build toolchain → Ubuntu 26.04 / GCC 15.2 (CUDA stays 13.3); the Dockerfile and
  both CI compile containers moved, which catches the GCC-15 missing-include class in
  CI. Note: nvcc silently drops -std=c++23 on a host compiler older than GCC 14,
  so dev/profiling images must be on this base — the impdev:ncu recipe is now
  committed at tools/Dockerfile.ncu (#907).
• Retired the legacy config surface: env-var seeding (down to IMP_DETERMINISTIC +
  IMP_FMHA_FA2), turboquant aliases, and dead flags (#879); imp.conf.example,
  --help, and config comments synced to parser reality (#878).
• VRAM-layer audit: dead modules removed, one reserve floor, honest budget logs (#877).
• Tokenizer: dropped the duplicated JSON parser in favor of shared model/json_util (#887).
• Analysis/roofline tooling: PTX survey scripts track the latest CUDA toolkit (#908);
  Python 3.14 plot env + roofline baseline re-pin (#904).

Added

• FP8 tile decode-attention kernels. Token-tiled FP8 split-K decode (K and V staged
  in one cp.async group) — long-context decode +51% (#899); a GQA-batched variant
  reads each KV head once across the warp group for a further +14% (#900).

Fixed

• MLA (DeepSeek-V2/V3) YaRN rope-mscale: the RoPE cos/sin were scaled by
  yarn_get_mscale(factor, mscale_all_dim) (=1.261 for V2-Lite) instead of the
  HF ratio yarn_get_mscale(factor, mscale) / yarn_get_mscale(factor, mscale_all_dim)
  (=1.0 when the two coincide, as in V2-Lite). imp was inflating the rotary
  embedding by 1.261×; the error compounds with position, so teacher-forced PPL
  degraded with sequence length. mscale and mscale_all_dim are now loaded
  separately: the softmax attention scale keeps mscale_all_dim² (unchanged),
  the rope factor uses the ratio. Same-corpus PPL vs HF bf16 on DeepSeek-V2-Lite:
  534-tok +24.4% → +2.75% (imp 7.78→6.43, HF 6.25); 196-tok +5.0% → +0.8%.
  The residual ~1-3% is F16-vs-bf16 compute precision. Applies to both
  DeepSeek-V2-Lite and DeepSeek-Coder-V2-Lite (same config); generalizes
  correctly to V3 (where the two mscales differ) (#880).
• MTP draft-head mrope now applies YaRN / rope-scaling (was plain NeoX RoPE), so the
  drafter no longer drifts from the verifier on rope-scaled models — speculative
  acceptance no longer degrades with position (#913).
• Async mempool is now trimmed on Model teardown, not only at the C-API boundary,
  releasing device memory between in-process model swaps (#915).
• Capture-poisoned engine wedge: a failed CUDA-graph capture no longer wedges the
  engine; plus planner-driven KV-pool sizing (#874, #875).
• GCC 15 build: added the <algorithm> / <numeric> includes that libstdc++15 no
  longer pulls in transitively (#903, #906).
• No-GPU audit sweep #888–#894: server admission control / observability / /health
  locking, embeddings, API strictness, and tool-call suppression, plus dead-code and
  doc/comment drift (#901).

v0.16.2 — FP4-attention research batch: quality-proven ThriftAttention promotion, program closed with measurements

FP4-attention research batch: the #846 program (SageAttention3 → ThriftAttention → KV-append-quant) is closed end-to-end with measurements on every branch. All new knobs are research scaffolds and ship default-off; no default-path behavior changes.

Added

• attention.mxfp4_promote_budget (default 0): ThriftAttention-style outlier block promotion (arXiv 2605.23081) in the MXFP4 FMHA — per q-tile, the top-scoring fraction of visible KV tiles (block-mean score Q̄·K̄ᵀ, sink + diagonal force-included) computes exactly instead of FP4. Takes the FP4 attention quality gate from +9.9%/+4.4% NLL (@1k/9.3k, prose) to −0.6%/−0.2% at 5% budget (#870).
• attention.mxfp4_paged_kv (default off): chunked-prefill continuation reads K/V directly from the paged NVFP4 KV cache (quantization paid once at append; no gather→FP16 pass, no in-kernel quant); the current chunk stays fresh FP16 via force-promoted tiles. Quality gate passes (+0.34% NLL @9.3k at 5% budget); kernel-level perf refuted — quality-validated scaffold (#872).

Findings (documented in MISSION_JOURNAL / #846)

• FP4-MMA delivers as advertised (tensor pipe 40.8% → 2.2% for the same math), but in-kernel K quantization costs 3.34× FA2's entire instruction budget; the smem-materializing kernel is latency-bound (pure paged-MMA floor 8.5× FA2) — register-resident FP4-QK port refuted (#871).
• Quantizing the RECENCY window is the entire quality cost of FP4 KV storage: stored-FP4 current chunk +3.7–5.4% NLL even with exact compute, stored-FP4 past ≈ free.
• Decode-recency probe: nvfp4-KV ~+0.8% NLL at decode-like granularity (no cliff, generation coherent); FP8 auto-default clean — nvfp4-KV quality claims need a small-chunk (≤64) PPL arm.

Full changelog: https://github.com/kekzl/imp/blob/main/CHANGELOG.md

v0.16.1 — spec-verify economics: +61% default speculation on Qwen3.6-27B

Highlights

Spec-verify chunk-path overhaul (#847 ladder) — the default suffix-speculation path gets dramatically cheaper verify cycles:

• Qwen3.6-27B prompt-echo: 81 → 131 tok/s (+61% vs v0.16.0); Qwen3.6-35B-A3B +10–15%
• Small-M NVFP4 GEMM: batched GEMV replaces the per-chunk full-weight dequant fallback (was 48% of GPU time on MTP verify) (#863)
• Small hd≠128 chunks route to the tiled FMHA — kills cuBLAS per-new-shape algo-selection churn (#865)
• Persistent K/V gather scratch for the eager chunked path (#866); batched verify/eval LM heads (#854, #857)

New speculation machinery

• SuffixDecoding-style suffix drafter with frequency voting + adaptive draft length is the default draft source (#848)
• Speculative decoding now engages on hybrid GDN/SSM models (recurrent-state snapshot/replay): Nemotron-3-Nano code-edit +60%, Qwen3.6-27B echo +156% (#852)
• Graph-captured verify chunk incl. hybrids — per-(bucket × KV-tier) CUDA graphs, device-side KV length (#856, #859, #861)
• MTP verify activation, opt-in (--mtp-spec-decode <k>): the trained MTP head drafts where the suffix matcher misses; device-side draft chain + NVFP4 chain lm_head + configurable economics guard (#852, #862, #864)

New model support

• nomic-bert encoder path + /v1/embeddings (nomic-embed-text-v1.5, HF-oracle cos ≥ 0.999) (#867)

Fixes

• Non-gated NVFP4 MoE da_cache never built → stack-UAF memcpy nodes in captured verify graphs (#861)
• MoE host-args launches + NVFP4 capture-refusal now fail loud under stream capture; hybrid conv-tail zero-fill bug on short chunks (#858, #859)
• Schema-constrained decoding: object keys reject backslash escapes (#851); tools+json_schema preamble slack (#842)
• Opt-in MXFP4 FMHA: chunked-prefill q_offset masking + fully-masked-row softmax guard (#868)

Research (default-off)

• NVFP4 attention compute spike per the SageAttention3 recipe (#846/#868): per-16-block scaling rescues FP4-QK from the catastrophic per-row failure mode, but residual noise compounds with context (+10% NLL @9k) — shipped as diagnostic knobs, quality-refuted for production; reopen path is ThriftAttention-style outlier promotion.
• Opt-in schema jump-ahead (#849, idea #844 — closed): forced-span draft chunks in the constrained pipeline; net-negative on BPE tokenization, kept as scaffold.

Full details in CHANGELOG.md.

v0.16.0 — hard VRAM budget for multi-server-per-GPU + load/teardown robustness

Multi-server-per-GPU (hard VRAM budget) + load/teardown robustness.

Added

• Hard per-process VRAM budget — --vram-budget <mb> (imp-server + imp-cli),
  [runtime] vram_budget_mb in imp.conf, and the previously-inert C-API
  ImpConfig.vram_budget_mb: every sizing decision (weight caches, KV clamp,
  expert offload, workspaces, upload gates — all 19 sites) sees a virtual GPU
  of the given size, so multiple imp-server processes can share one card.
  Baseline-delta semantics: a co-tenant's pre-existing usage never counts
  against this process's budget; concurrent neighbour allocations shrink the
  view conservatively. Verified with two simultaneously-started servers
  (9000 + 8000 MiB budgets) serving concurrently at 15.9 GiB device total.
  Best-effort cap — leave ~1 GiB real headroom between the sum of budgets and
  the card. Default 0 = uncapped passthrough (#838).

Fixed

• Model unload leaked weights-sized VRAM (~8.3 GiB per Qwen3-8B-Q8_0
  cycle): weights are cudaMallocAsync-allocated but were freed with plain
  cudaFree, which returns success WITHOUT returning the blocks to the async
  mempool on this stack — the pool double-booked old + new weights on reload
  and cudaMemPoolTrimTo could reclaim nothing. Freed with cudaFreeAsync
  everywhere (Model teardown + the Phase-3 MoE expert-source drops, whose
  "freed" VRAM was phantom for the same reason). The reload test now probes
  actual re-allocatability (WSL2/WDDM under-reports reclaimed pages in
  cudaMemGetInfo) (#834, #837).
• Encoder-only models are rejected at load on the SafeTensors/HF path too
  — is_encoder_only_arch was case-sensitive, so HF config.json class names
  (NomicBertModel, BertModel, XLMRobertaModel, …) slipped past the
  GGUF-only reject and ran a BERT encoder through the causal-LM prefill +
  sampler → CUDA illegal memory access on the first /v1/embeddings request.
  Both HF-config paths (architectures array + model_type fallback) now
  fail loudly at load (#818, #835).
• Second engine on the same loaded model handle no longer IMAs — for GGUF
  MXFP4 GDN models the first engine's pre-dequant consumes the model sources
  destructively (in-place MXFP4 raw-block compaction; GDN FP16 fallback
  re-points model tensors at executor-owned memory), so a create→free→create
  cycle rebuilt caches from dangling memory and poisoned the CUDA context.
  Engine::init now rejects a second engine on a consumed model with a clear
  "reload the model" error; models whose sources stay intact (dense Q8_0)
  keep supporting create/free/create on one handle (#830, #835).