[Support Request] Support for Gemma 4 Assistant/Drafter models (Gemma4AssistantForCausalLM)

[abdo1954]

I am trying to use the newly released Google Gemma 4 MTP Drafter models (Assistant models) with llama.cpp for speculative decoding. Specifically, I am testing gemma-4-26B-A4B-it-assistant.

Currently, the convert_hf_to_gguf.py script does not recognize the architecture Gemma4AssistantForCausalLM.

Describe the solution you'd like
Support for converting and running Gemma 4 Assistant models in GGUF format. These models are crucial for enabling speculative decoding (using the --draft flag) to speed up inference for the larger Gemma 4 26B/31B models.

Describe alternatives you've considered
I attempted to manually bypass the architecture check by modifying config.json to gemma2, but the conversion failed with the following error due to missing tensor mappings:
ValueError: Can not map tensor 'model.layers.0.layer_scalar'

It seems these models introduced new scaling tensors (like layer_scalar) that are not yet mapped in the current Gemma2Model or GemmaModel classes in convert_hf_to_gguf.py.

Additional context

Model Link: google/gemma-4-26B-A4B-it-assistant

Environment: Windows 10, CUDA 13, Python 3.11

Goal: Use these models as drafters to achieve higher tokens-per-second on consumer hardware (RTX 5060 Ti).

Since Gemma 4 is a major release, having official support for its dedicated assistant models would significantly benefit the community.

[alexalmarch411-dev]

We are really looking forward to this support.

[k-inoway]

#22747
#22673

[DarkAndBlue]

Why was this closed? #22738

[feffy380]

They already gave a reason. The repo does not accept vibecoded garbage

[PavanKumarVattikuti]

found this one.
https://huggingface.co/collections/AtomicChat/gemma-4-assistant-gguf

[abdo1954]

yes <3 think U

[MoRanYue]

Gemma MTP Drafter is pretty useful, I hope this support could be merged as soon as possible.

[reffdev]

I have a fork with working Gemma 4 MTP speculative decoding (tested on E4B, ~70-87% acceptance depending on content, up to ~60% throughput improvement). Not PR-quality but functional, linking for reference: https://github.com/reffdev/llama.cpp/tree/gemma4-mtp

[hewabive]

I tested this on a small CPU-only machine, because I can realistically run only Gemma 4 E2B. I used reffdev/gemma4-mtp at 0fc1835 with one local build fix for the accept(...) signature. With unsloth Gemma 4 E2B Q3/Q4 target GGUF and google/gemma-4-E2B-it-assistant, the asistant converted to GGUF F16, and llama-server could load target + assistant with --spec-type draft-mtp and produce completions. So the converter/load/server smoke path is reproducible for E2B in this branch. On upstream master ,the same assistant GGUF fails before running with unknown model architecture: 'gemma4_assistant'.
I would not call the runtime path correct yet. With fixed seed, temp=0, return_tokens=true, and target-only output stable between upstream master and reffdev, baseline vs MTP still produced different token ids. This also stayed true with single thread and f32 KV cache. The different token was a close top-prob altermative ,so maybe it is numerical or batch-path sensitivity, but deterministic token-level equivalence is not confirmed in my test.

One more caveat: in the reffdev server path I tested, --spec-draft-n-max 1/2/3/5/8 and --spec-draft-p-min 1.0 did not change output or draft stats. I cheked the code and it looks like the Gemma4 MTP path uses slot capacity dp.n_max clamped to 5, and does not directly use the parsed params.speculative.draft.n_max or p_min in this path. So these flags seem ineffective for Gemma4 MTP server behavior in this experimental branch.

[d-shehu]

Is this the recommended approach? Still confused on MTP vs Spec decoding. I saw this detailed post of using speculative decoding but I'm unable to reproduce the numbers with some quick Q&A. As with Llama 3.3, etc. speculative decoding is actually slower.

Both draft and main model fit in VRAM

Ubuntu 24.04 + RADV Vulkan
Llama.cpp: b9222
2x R9700

./llama-cli 
--model models/models--unsloth--gemma-4-31B-it-GGUF/UD-Q8_K_XL/gemma-4-31B-it-UD-Q8_K_XL.gguf 
--parallel 1 
--spec-draft-n-max 8 
--spec-draft-n-min 1 
--model-draft models/models--unsloth--gemma-4-E2B-it-GGUF/UD-Q8_K_XL/gemma-4-E2B-it-UD-Q8_K_XL.gguf 
--device-draft Vulkan0,Vulkan1 
--ctx-size 262144 
--seed 3457
 -t 16 
 -dev Vulkan0,Vulkan1 
 --tensor-split 30,30 
 --no-warmup 
 -fit off 
 -ngl -1 
 --no-mmap 
 -fa auto 
 --cache-type-k q8_0 
 --cache-ty

1. Main Model: Unlsoth Gemma 4 31B IT
   File: gemma-4-31B-it-UD-Q8_K_XL.gguf

2. Draft Model: Unsloth Gemma 4 E2B IT
   File: gemma-4-E2B-it-UD-Q8_K_XL.gguf

https://www.reddit.com/r/LocalLLaMA/comments/1sjct6a/speculative_decoding_works_great_for_gemma_4_31b/

[am17an]

It's WIP #23398

[datlife]

super cool! Thank you!

[yarrrly]

Some numbers from a grammar-constrained workload, since most reports here are code or prose.

Setup: Gemma 4 E2B, Q4_0 QAT weights (Ollama's gemma4:e2b-it-qat, 4.6B including PLE) with the unsloth export of the official drafter (mtp-gemma-4-E2B-it-Q8_0.gguf, 98 MB). llama-server as bundled with Ollama 0.32.15, started with --spec-type draft-mtp --spec-draft-backend-sampling --spec-draft-n-max N --flash-attn on, ctx 32768, one request at a time. Apple M1 Pro 16 GB, Metal. Every request is a JSON-schema-constrained extraction (json_schema in the request, so a grammar), prompts of 1.7 to 2.4k tokens of mixed Russian and English, outputs of 0.9 to 1.6k tokens.

Acceptance from the server's draft-mtp statistics lines:

• n-max 3: 71.6 % of drafted tokens accepted (11,926 of 16,662), mean accepted length 3.15, per position 0.82 / 0.70 / 0.63. Temperature 1.0 and 0 gave the same rate (71.3 % over 50 calls at temperature 0 with repeat_penalty 1.1).
• n-max 5: 61.4 % (15,960 of 25,995), mean accepted length 4.07, but slower end to end than n-max 3: 20.0 vs 16.0 ms per output token. The extra verification work outweighs the longer drafts on this GPU.
• KV cache q4_0: acceptance unchanged at 70.1 % (9,560 of 13,635), and no speedup: 35.0 ms per output token with the drafter vs 32.2 without. With f16 KV the drafter gives 16.0 vs 19.5 ms (1.22x). So here the quantized cache does not reject drafts; it makes the verification pass expensive enough to cancel the gain. That is a different failure than the 0 % acceptance described in llama : add Gemma4 MTP #23398. Ollama passes --spec-draft-backend-sampling, which may be the difference.
• For scale, f16 vs q4_0 KV without any drafter on this model: 19.5 vs 32.2 ms per output token, a bigger effect than the drafter itself.

One caution when reading acceptance figures: a run that fell into a repetition loop at temperature 0 reported 91.7 % cumulative acceptance, because the drafter predicts repeated text perfectly. Check output length before trusting a high rate.

All timings are wall clock per output token and include a roughly 2k-token prefill.