Reasoning appears broken with --concurrent + WebUI on Qwen3.5-27B-8bit

[scouzi1966]

Summary

Reasoning (<think>) output from mlx-community/Qwen3.5-27B-8bit appears broken when the server runs in concurrent mode (--concurrent N) with the WebUI enabled (-w).

Environment

• Model: mlx-community/Qwen3.5-27B-8bit
• Flags: --concurrent <N> + -w (WebUI)
• Nightly: v0.9.10-628c2bb (release nightly-20260408-628c2bb)
• Platform: Apple Silicon, macOS 26+

Repro

MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
  afm mlx -m mlx-community/Qwen3.5-27B-8bit --concurrent 15 -w

Then submit a prompt via the browser WebUI.

Expected

Model emits <think>...</think> content which gets extracted into reasoning_content (streaming and non-streaming), with natural-language content following.

Observed

Reasoning output appears broken. (Exact symptoms to be filled in with a reproducer — e.g., missing <think> tags, reasoning_content empty when it should not be, malformed structure, or content/reasoning interleaving incorrectly.)

Suspected area

Concurrent + WebUI is a code path with overlapping state:

1. BatchScheduler per-slot token streaming — think-tag extraction is applied downstream of StreamChunk emission in MLXChatCompletionsController. Any per-slot boundary handling (think-buffer carry between chunks) would need to be independent per request.
2. WebUI frontend — the llama.cpp webui may strip or mishandle reasoning_content / <think> tags if they arrive via non-standard SSE fields.
3. Chat template — Qwen3.5-27B-8bit's chat_template.jinja <think> handling may interact with concurrent mode's prompt processing differently than serial mode.
4. Relation to --guided-json CLI flag is silently ignored, model produces unconstrained output #97 fix area — the batch completions controller now uses effectiveResponseFormat, and per-request think-extract state is initialized per slot; verify the think buffer isn't shared across slots.

Validation questions

• Does the same model + same prompt work correctly in serial mode (no --concurrent)?
• Does it work correctly in concurrent mode without -w (direct HTTP client)?
• Does the raw SSE stream contain reasoning_content deltas, or is the issue purely in the WebUI rendering layer?
• Does it reproduce across models that share the same <think> template (e.g., other Qwen3.5 variants)?

Related

• Gemma 4 streaming tool call type coercion (array/object remain strings) #87 — Gemma 4 streaming tool call type coercion (concurrent + streaming state)
• Prefix cache crash on second large prompt with Gemma 4 31B-8bit #94 — Radix cache SIGTRAP on wrapped RotatingKVCache (concurrent + per-slot state)

Known concurrent-mode regression to fix first

A separate TopPSampler 1D crash in concurrent mode (fixed locally, not yet published) may be masking this issue during reproduction — any request with top_p < 1 hits [squeeze] fatal error before reasoning output is evaluated. The WebUI default is top_p=0.95, which triggers that crash. Once the TopPSampler fix is published in the next nightly, re-test this issue to confirm the reasoning breakage is independent.

[scouzi1966]

Concrete symptom (from screenshot)

Prompt: why is the sky blue?

Assistant response (rendered in WebUI) shows the model's internal planning dump as the main message content:

Thinking Process:
1. Analyze the Request:
   - Question: "why is the sky blue?"
   - Intent: The user is asking for a scientific explanation of a common natural phenomenon.
   - Tone: Informative, clear, accessible.
2. Identify Key Concepts:
   - Sunlight (white light, spectrum of colors).
   - Earth's Atmosphere (gases, molecules like nitrogen and oxygen).
   - Scattering (specifically Rayleigh scattering).
   - Wavelengths (blue/violet vs. red/yellow).
   - Human eye sensitivity (why we see blue instead of violet).

No actual answer is shown — reasoning is rendered as content, and either the </think> boundary is never reached within the WebUI's view or the post-thinking content is being dropped.

Root cause (NOT concurrent-specific)

This affects all modes (serial and concurrent, streaming and non-streaming). The WebUI just surfaces it.

Chat template (Qwen3.5-27B-8bit/chat_template.jinja:149-152):

{%- if enable_thinking is defined and enable_thinking is false %}
    {{- '<think>\n\n</think>\n\n' }}
{%- else %}
    {{- '<think>\n' }}

The template pre-opens <think>\n at the end of the prompt. The model's generation begins already inside a think block — the stream never contains an opening <think> tag, only the eventual </think>.

AFM's extractor (MLXChatCompletionsController.swift:1344):

static func extractThinkTags(
    buffer: inout String,
    insideThinkBlock: inout Bool,   // ← always initialized to false at stream start
    startTag: String = "<think>",
    endTag: String = "</think>"
)

The state machine starts with insideThinkBlock = false and only flips true when it sees <think> in the stream. For Qwen3.5-27B, that tag is never emitted (it was in the prompt, not the generation). Consequences:

1. insideThinkBlock stays false forever
2. All the thinking tokens go into content instead of reasoning_content
3. When </think> finally arrives, the extractor is still in the "not inside" branch — the </think> is treated as a literal content string (no <think> to pair it with)
4. The real answer that follows may or may not be visible depending on how the WebUI renders the mess before it

Why Gemma 4 works but Qwen3.5 doesn't

MLXModelService.swift:1185-1194 has a special branch for Gemma 4's channel-based thinking:

if self.thinkStartTag == "<|channel>" {
    var kwargs = self.defaultChatTemplateKwargs ?? [:]
    if kwargs["enable_thinking"] == nil {
        kwargs["enable_thinking"] = true
        self.defaultChatTemplateKwargs = kwargs
    }
}

But there's no corresponding logic for <think>/</think> models that pre-open the tag in their template. Qwen3.5-27B, Qwen3, DeepSeek-R1, and similar reasoning-model templates all fall into this gap.

Fix direction (sketch — not implemented yet)

Two reasonable approaches:

Option A — model-family flag: when thinkStartTag == "<think>" is detected AND the tokenizer_config / chat_template indicates the template pre-opens it (look for '<think>\n' literal in the template, or use model_type match against a known list: qwen3_5, qwen3, deepseek_v3, etc.), pass this as chatPreOpensThinkTag: true into the per-request streaming state and initialize insideThinkBlock = true.

Option B — auto-detect from prompt: when rendering the prompt via the chat template, check whether the rendered prompt ends with <think>\n (or any detected thinkStartTag). If so, initialize the extractor with insideThinkBlock = true for that request. This is template-agnostic — it just observes what the template actually produced.

Option B is more general and has no hardcoded model list. It should also work for any future reasoning model that follows this pattern.

Concurrent mode is a red herring

I was wrong to file this as "concurrent + WebUI". Concurrent mode and WebUI are not causes — they're just the path the user happened to take. Retitling suggestion: "Reasoning extraction broken for chat templates that pre-open <think> tag (Qwen3.5, DeepSeek-R1 family)".

Scope of affected models

Any model whose chat template ends with a literal <think>\n (or equivalent) before the assistant turn. Likely candidates in the vesta-test-cache to verify:

• Qwen3.5-27B-8bit (confirmed — this issue)
• Qwen3.5-35B-A3B-4bit (needs check — same model_type qwen3_5)
• Qwen3-* (needs check)
• DeepSeek-R1-* (needs check)

If the nightly smart tests for Qwen3.5-35B-A3B-4bit passed with reasoning_content populated, that would falsify my hypothesis — but looking back at the test results, those tests use greedy/categorical sampling on short prompts, and may not have explicitly validated reasoning_content extraction separately from content. Worth verifying.