fix: graceful fallback when model has no chat_template (MedGemma)#271
fix: graceful fallback when model has no chat_template (MedGemma)#271jackneil wants to merge 1 commit intowaybarrios:mainfrom
Conversation
Models like MedGemma have apply_chat_template() as an inherited method but no chat_template configured, causing ValueError on every request. Now catches ValueError and falls back to plain-text prompt format in both BatchedEngine and SimpleEngine paths. Fixes: MedGemma crashes with "Cannot use apply_chat_template" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
One correctness concern before merge: the new fallback paths catch broad That is fine for the specific MedGemma case ( I think the fallback should be narrowed to the known missing-template case only — e.g. catch |
Thump604
left a comment
There was a problem hiding this comment.
I rechecked this against current main, and I do not think it is merge-ready in its current shape.
Two blockers:
-
The fallback is still too broad. Catching generic
TypeError/ValueErrorfromapply_chat_template()will silently mask unrelated template regressions. This should be narrowed to the known missing-template case only (theno chat_template configured/Cannot use apply_chat_template when no chat_template is setclass of failure). -
More importantly, this branch does not actually cover the real
SimpleEngine.chat()crash path on currentmain. The non-streaming simple-engine path still delegates straight intoself._model.chat(...)before any of this local fallback logic runs, so a model/processor withapply_chat_template()present but no configuredchat_templatecan still fail before reaching the new fallback code. The current diff mainly helps the local prompt-building/accounting paths, not the full execution path.
Given that the branch is also conflicting, I would rather see a small current-main replacement that fixes the real execution path and adds a regression against that path specifically.
Summary
apply_chat_template()inherited but nochat_templateconfiguredValueError: Cannot use apply_chat_template when no chat_template is setValueErrorand falls back to plain-text prompt format in both BatchedEngine and SimpleEngineTest plan
🤖 Generated with Claude Code