OxiBonsai 0.2.3 Release

[0.2.3] - 2026-07-21

Large cross-cutting hardening pass following a 109-agent production-release audit (91 confirmed findings across 23 implementation packages plus documentation honesty). Full per-package evidence lives in TODO.md's "Production-Release Audit (2026-07-20)" section.

A second pass (2026-07-21, same branch) closed the remaining Metal GPU platform gaps (Q4_0/Q8_0/K-quant GEMV + model-layer dispatch, FP8 batch prefill), built and wired the OpenAI-compatible penalty/logprobs seam Round 1 had only validated-and-honestly-rejected, added NEON Q4_0/Q8_0 kernels, fixed a kernel-dispatch-threshold gap, and closed further GGUF/checkpoint/LoRA/tokenizer/server robustness and documentation-accuracy gaps. Full per-package evidence lives in TODO.md's "Production-Release Audit Round 2 (2026-07-21)" section; every superseded Round-1 claim is annotated in place there rather than silently rewritten.

Fixed

• CUDA prefill/decode KV-cache desynchronization (P0) — unified the batch-prefill KV cache with the per-token decode KV cache (cuda_prefill/state.rs now delegates to cuda_full_layer::acquire_kv_cache), fixing silent decode corruption on Q1 prompts ≥17 tokens (decode logit Δ collapsed from ~7.3 to ~0.001); fixed the missing per-token d_pos_seqlen upload in the Q1 prefill loop; re-enabled ternary CUDA batch prefill, which had been disabled since the same bug was found.
• InferencePipeline generation now honors GenerationStrategy, applies the attached TokenConstraint, wires token healing against the real model, and no longer panics on engine errors (previously used .expect()).
• Speculative decoding gained a real, non-fabricating two-engine entry point, SpeculativeDecoder::generate_verified, that drafts against the delta-KV path and verifies against a genuine separate target InferenceEngine; fixed a draft-reuse out-of-bounds panic.
• Tokenizer fidelity: special/added tokens are now protected from BPE/Unigram/WordPiece shredding at encode() dispatch; GPT-2 byte-level bytes↔unicode mapping is wired in (fixes CJK/emoji/accented-Latin encoding and whitespace-run preservation); BPE merge-rank lookup is now O(1) instead of an O(total-merges) scan.
• JsonConstraint/RegexConstraint now operate on real token text (via a new decoder-backed TokenTextIndex) instead of misinterpreting raw token IDs as Unicode codepoints.
• Server request validation: max_tokens/temperature/top_p/n are now validated with honest 400s (closes an unbounded-max_tokens OOM/DoS path); /rag/query performs a real encode→generate→decode instead of returning a fabricated single token; /v1/models and /admin/config//admin/cache-stats now report real, honest state instead of hardcoded/fake values; the admin router, chat web UI, CORS, request logging, and an opt-in rate limiter are now actually mounted on the served router.
• OpenAI-compatible API extensions: finish_reason is now derived rather than hardcoded; frequency_penalty/presence_penalty are now rejected with an honest 400 instead of being silently discarded (there is no penalty seam in the sampler to apply them to); logprobs on the extended chat endpoint now honestly returns null instead of a fabricated empty array; n>4 now 400s.
• GGUF parsing hardening: bounded Array-nesting recursion depth, checked/saturating tensor offset/size/shape arithmetic, and general.alignment validation against untrusted/adversarial GGUF files; the existing compat-report checker is now wired into the real load path.
• Quantization/export correctness: fixed a Q1_0_g128 export sign-bit inversion (a real bit-level correctness bug, not just a doc issue); Int8PerChannel export now refuses instead of silently mislabeling its bytes as F32; guarded a division-by-zero in per-channel quantization; bounded Checkpoint::read_from's eager allocations against corrupted headers; extended f32 tensor loading to cover Q4_K/FP8 E4M3/E5M2.
• oxibonsai-serve: max_concurrent_requests/per_request_timeout_ms are now genuinely enforced (tower admission stack: overload→503, timeout→408); tokenizer.kind is validated against a real backend whitelist; CLI-flag precedence over TOML/env is now correct even when the CLI value matches the built-in default; bearer-token comparison is now constant-time.
• Grammar/JSON-Schema engine: the JSON-Schema→BNF compiler now rejects previously-silently-ignored keywords (minimum/maximum/minLength/maxLength/minItems/maxItems/uniqueItems) and implements real const support; object schemas now support optional (non-required) properties; tool-calling's select_tool now validates arguments against the tool's schema instead of only checking well-formed JSON; the Earley allowed_tokens cache is now keyed by (state_hash, vocab_size), closing a latent wrong-length-mask risk.
• Prefix-cache engine / semantic cache: PrefixCachedEngine now keeps a persistent seeded sampler across calls instead of discarding RNG state every call; SemanticCache::refit_embedder re-embeds existing entries instead of leaving stale-dimension zombies; max_entries == 0 no longer produces undefined eviction behavior; all server-reachable mutex sites recover from poisoning instead of propagating a panic.
• oxibonsai quantize now performs a real dequantize→re-encode→write pipeline instead of fabricating size/ratio estimates from the format string alone.
• TruthfulQA / multiple-choice evaluators: empty or all-NaN logits no longer silently default to "index 0 is correct"; ChunkConfig now rejects min_chunk_size > chunk_size, which previously guaranteed permanently-empty chunking on every RAG indexing path.
• KvCache::store_key/store_value gained real release-mode bounds checking (previously debug_assert!-only, a no-op in release builds); PagedKvCache::sequence_length() now returns the actual highest-written position instead of block-rounded capacity.
• wasm32-unknown-unknown build restored — engine_pool (which unconditionally used tokio) is now correctly cfg-gated off the wasm target, matching the sibling async_engine module.
• oxibonsai facade crate: the server/full Cargo features now actually compile (previously missing dep:oxibonsai-serve).
• kernels-cpu: guarded AlignedBuffer/AlignedBlocks allocation sizing against usize overflow; removed dead AVX-512 gather/scatter helpers and wired a previously-dead AVX-512 streaming-store GEMV kernel into the dispatch tier; PlatformProfile's tuned thresholds now genuinely govern GEMV/GEMM parallel dispatch.
• kernels-meta: build.rs's embedded-metallib fast path pointed at a deleted source file and was permanently dead; it now reads the current kernel_sources/ module layout, with a loud panic on future kernel-whitelist drift instead of a silently empty metallib.
• Image pipeline: fixed a TileBoundary::AfterMid no-op and added TE GPU weight residency on CUDA.

2026-07-21 second pass:

• repetition_penalty was a silent no-op (oxibonsai-runtime) — the default-enabled SamplingParams::repetition_penalty (1.1) is now genuinely applied in the native decode loop via a new Sampler::sample_with_history + PenaltyParams (frequency_penalty/presence_penalty, OpenAI formula) seam; the no-penalty case remains bit-identical to prior output. Also added GGUF-resolved eos_token_id() (was always the hardcoded fallback) and a logits-capturing generate_with_logprobs.
• OpenAI-compatible penalties/logprobs are now genuinely applied, not honestly rejected: base /v1/chat/completions and the extended endpoint both apply frequency_penalty/presence_penalty ([-2.0, 2.0]) for real (Round 1's 400-rejection is retired) and return real per-token logprobs/top_logprobs (was a validated-but-null placeholder on the extended endpoint); /v1/completions now also honors temperature/top_p/penalties and reports the real loaded-model id (was hardcoded "bonsai-8b"). A mid-stream SSE failure now emits a terminal {"error":...} event instead of a bogus clean finish; the base endpoint now parses tool_calls from generated text (previously extended-only); <|...|> ChatML markers in message content are now stripped by default before prompt assembly (prompt-injection/turn-boundary-forgery fix, opt-out OXI_DISABLE_PROMPT_SANITIZATION); every route now returns a unified OpenAI-style {"error":{...}} envelope; the server now serves with real peer connect-info so per-peer rate limiting is live end-to-end.
• Metal GPU platform gaps closed: new Metal GEMV kernels + model-layer dispatch for Q4_0/Q8_0/K-quant (Q2_K–Q8_K) — Linear*::forward() now tries Metal first (mirroring the existing CUDA short-circuit) before falling back to CPU; a hybrid GPU-linear-projections/CPU-attention Metal FP8 batch prefill (Phase 28.B) that avoids the CUDA-FP8-style split-KV-cache bug by construction. Both parity-validated on Apple-Silicon M3.
• NEON GEMV kernels for Q4_0/Q8_0 (oxibonsai-kernels) implemented and wired into the runtime dispatcher — was silently falling back to the scalar reference on AArch64 CPUs.
• Kernel-dispatch thresholds: the adaptive tile-size selector (parallel_tiled.rs::select_gemv_strategy) and L1 tile sizing (tiled.rs::optimal_tile_rows) now read PlatformProfile::global_thresholds() instead of hardcoded constants — closes a gap in the earlier "thresholds now govern dispatch" claim, which had only covered parallel.rs.
• Metal decode robustness: context-length guards on all six Metal prefill/greedy entry points (was an out-of-bounds RoPE-table panic on an over-long prompt); a coherent Metal→CPU mid-stream fallback in generate_greedy_gpu that rebuilds the CPU KV cache before continuing (was silent corruption reading a stale/zero CPU cache); gpu_cache.rs's Q1 GPU-handle fallbacks are now hard errors with a CPU fallback instead of silently defaulting to handle...

OxiBonsai 0.2.2 Release

[0.2.2] - 2026-06-08

Added

• oxibonsai repl interactive image REPL (oxibonsai-image, oxibonsai-cli): ImageSession
  loads the DiT, VAE, and text encoder once and renders many prompts without re-paying the
  load/dequant cost. StageTimings and RenderOutcome surface per-stage wall-clock splits.
  The session puts the text encoder in resident mode (TeWeights::set_resident) so the
  dequantised f32 weights (~16 GB) stay cached across renders on high-memory machines.
  On Ghostty the rendered image is shown inline via the kitty graphics protocol; on other
  terminals the PNG is written to a file. Runtime commands: :steps, :seed, :size,
  :fast (2-step 384×384 preview), :hq (8-step 512×512), :out, :open, :help, :quit.
• TeWeights::set_resident(on: bool) (oxibonsai-image): controls whether the Mlx4bit
  source caches dequantised f32 tensors across forwards. Off by default (preserves the one-shot
  CLI low-RAM profile); turned on by ImageSession for the REPL use-case.
• Kitty graphics protocol support (src/cli/term.rs): pure-Rust base64 encoder and inline
  PNG display for Ghostty terminals (kitty_supported() auto-detects via GHOSTTY_* env vars
  and TERM/TERM_PROGRAM).
• GPU acceleration flags documented in .env.example: OXI_DIT_ATTN_GPU (flash-attention,
  default-ON on Apple Silicon), OXI_VAE_GPU (convolutions, default-ON on Apple Silicon), and
  OXI_TE_GPU (text-encoder GPU, default-OFF — CPU SIMD wins on Apple Silicon; may help on
  Windows/NVIDIA CUDA) with platform-specific comments.
• CUDA TQ2 GEMV parity test (oxibonsai-kernels): isolated probe cuda_tq2_gemv_parity.rs
  for Blackwell GPU output validation; compile-gated behind cfg(feature = "cuda").

Changed

• decoded_chw_to_rgb8 extracted as shared helper (oxibonsai-image/pipeline.rs): CHW→HWC
  f32-to-u8 conversion factored into a pub(crate) function, shared by both text_to_image and
  ImageSession::render to guarantee byte-identical pixel output from both paths.
• oxionnx-proto bumped 0.1.3 → 0.1.4 (Cargo.toml workspace dependencies).

Full Changelog: v0.2.1...v0.2.2

OxiBonsai 0.2.1 Release

Changed

• Raised compile optimization for the test and dev profiles (Cargo.toml): added [profile.test] opt-level = 2 and [profile.dev.package."*"] opt-level = 3, so test binaries and all dependencies (including workspace path-deps like oxibonsai-model / oxibonsai-kernels when built as deps of another crate's tests) are optimized and autovectorized. At the default opt-level = 0 the workspace's tests run real numeric work unoptimized — the parity golden references and model forward passes took minutes (e.g. the DiT-shape joint-attention CPU reference ~14.5 GFLOP, and the speculative decoder's ~240 forward passes over the 151936-row vocab). The crate under active edit stays at dev opt-level 0, so incremental compiles of your own code remain fast. Float results are unchanged: Rust does not enable fast-math, so opt-level does not reassociate reductions — parity gates (cos≥0.999) stay bit-stable.
• Bumped oxiarc-deflate 0.3.2 → 0.3.3 (Cargo.toml, [workspace.dependencies]): tracks the latest OxiARC release per the COOLJAPAN Latest-crates policy. oxiarc-deflate provides the Pure-Rust DEFLATE backend for PNG output in oxibonsai-image; the substantive fixes in the 0.3.3 OxiARC release land in sibling crates (oxiarc-brotli high-entropy round-trip), so for OxiBonsai this is a version-tracking bump with no change to DEFLATE/PNG behavior.

Fixed

• VAE precheck rejected a valid .safetensors file (oxibonsai-image, src/pipeline.rs): the text-to-image precheck used is_dir(), so a valid .safetensors FILE passed via --vae / OXI_VAE_WEIGHTS was rejected with "VAE weights dir not found" before any loading — even though VaeWeights::open and the docs both accept a file. The precheck now accepts a file or a directory (is_file() || is_dir()), error wording is corrected, and the stale doc-comment is fixed. Added regression tests test_issue_9_*. (#9)

Documentation

• Corrected stale HuggingFace asset paths (docs/IMAGEN.md, crates/oxibonsai-image/README.md): the DiT lives under transformer-packed-mflux/, and the text encoder + tokenizer ship inside the main prism-ml/bonsai-image-ternary-4B-mlx-2bit repo under text_encoder-mlx-4bit/ (the standalone prism-ml/text_encoder-mlx-4bit repo does not exist). Consumer paths updated to match hf download --local-dir layout, and the bundled vs. gated black-forest-labs/FLUX.2-dev VAE choice is clarified. Verified against the live HF API. (#8)

Full Changelog: v0.2.0...v0.2.1

OxiBonsai 0.2.0 Release

Full Changelog: v0.1.5...v0.2.0

OxiBonsai 0.1.5 Release

Full Changelog: v0.1.4...v0.1.5

OxiBonsai 0.1.4 Release

Full Changelog: v0.1.3...v0.1.4

OxiBonsai 0.1.3 Release

Full Changelog: v0.1.2...v0.1.3

OxiBonsai 0.1.2 Release

Full Changelog: v0.1.1...v0.1.2

OxiBonsai 0.1.1 Release

Full Changelog: v0.1.0...v0.1.1

OxiBonsai 0.1.0 Release

Full Changelog: https://github.com/cool-japan/oxibonsai/commits/v0.1.0