Karen/gemmma multiturn template - #711
Merged
Merged
Conversation
Collaborator
Author
|
@jakmro here i fixed the chat template for lfm and qwen, fixed function call decoders for them, fixed multiturn gemma function call, and cleaned out some comments. Tested on gemma, lfm, qwen, needle for function calling |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the engine’s multi-turn chat + tool-calling support beyond Gemma to cover additional model families (notably Qwen and LFM2) by introducing model-specific tool serialization, prompt formatting, and tool-call extraction.
Changes:
- Add
chat_tools.hutilities for serializing tool definitions and extracting tool calls from Qwen/LFM2-style outputs. - Update
Tokenizerprompt formatting to emit Qwen/LFM2/Gemma4 family-specific templates (including tool call / tool response blocks). - Route tool-definition formatting in
prepare_prompt()based on tokenizer model family (Qwen/LFM2/Needle/Gemma).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
cactus-engine/src/utils.h |
Adds Qwen/LFM2 tool-call extraction during response parsing; improves Needle <tool_call> scanning robustness. |
cactus-engine/src/tokenizer.cpp |
Implements/updates Qwen + LFM2 prompt templates, tool call rendering, and Gemma4 multi-turn tool-response formatting. |
cactus-engine/src/model.cpp |
Removes several explanatory comments in embedding and KV-cache compression paths (behavior unchanged). |
cactus-engine/src/engine.h |
Adds tokenizer helpers to identify Qwen/LFM2 model families. |
cactus-engine/src/complete.cpp |
Chooses tool-definition serialization format based on tokenizer family (Needle/Qwen/LFM2/Gemma). |
cactus-engine/src/chat_tools.h |
New header providing tool JSON helpers, Qwen/LFM2 tool serialization, and tool-call extraction/parsing utilities. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+497
to
+502
| for (size_t k = 0; k < msg.tool_calls.size(); ++k) { | ||
| const auto& tc = msg.tool_calls[k]; | ||
| if ((k == 0 && !content.empty()) || k > 0) result += "\n"; | ||
| result += "<tool_call>\n{\"name\": \"" + tc.name + "\", \"arguments\": " + | ||
| chat_tools::respace_json(tc.arguments.empty() ? "{}" : tc.arguments) + "}\n</tool_call>"; | ||
| } |
Comment on lines
+511
to
+514
| if (add_generation_prompt) { | ||
| result += "<|im_start|>assistant\n"; | ||
| if (!enable_thinking_if_supported) result += "<think>\n\n</think>\n\n"; | ||
| } |
Comment on lines
2956
to
2958
| } | ||
| // run_audio_encoder_messages unloads the graph; restore so subsequent | ||
| // transcribe_* paths (which assume the encoder stays loaded) work. | ||
| load_component_graph(*audio_encoder_); | ||
| return embedding; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.