Inference: mid-conversation system messages (generate_reply instructions) reach google/gemma-4-31b-it as a trailing system role and the model answers as the user

[nader-bytesai]

Summary

On LiveKit Inference, google/gemma-4-31b-it receives every mid-conversation system message (the one generate_reply(instructions=...) appends, output-guard style chat_ctx.add_message(role="system", ...), etc.) as a raw trailing system role, because the gateway serializes with the openai provider format, which does not run convert_mid_conversation_instructions. Gemma's chat template only knows a system turn in first position, so a later one is rendered as an unknown turn and the model reacts as if it were the caller: it answers its own last question ("That works for me!"), narrates stage directions, or silently calls a tool instead of speaking the check-in.

This is the same class of bug #6591 fixes for Gemini on the gateway (its google/ allowlist would cover Gemma too). Filing this so the Gemma case is on record with a runnable reproduction, since that PR's description only mentions Gemini.

Environment

• livekit-agents 1.6.1 and 1.6.8 (behaviour unchanged; inference/llm.py on main still defaults provider_fmt="openai")
• Model: google/gemma-4-31b-it via livekit.agents.inference.LLM
• Python 3.13, Windows 11 (reproduces identically on Linux workers)

Reproduction

Self-contained script; needs LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET. It builds the chat context an AgentSession has right before a silence-timer generate_reply(instructions=...) fires — a base system prompt, a tool call turn, an assistant question — and sends it twice: once with the instruction as the trailing system message (what the gateway does today), once with the same context passed through the SDK's own convert_mid_conversation_instructions.

import asyncio
import sys

from livekit.agents import inference, llm
from livekit.agents.llm import ChatContext
from livekit.agents.llm._provider_format.utils import convert_mid_conversation_instructions

MODEL = "google/gemma-4-31b-it"
RUNS = int(sys.argv[1]) if len(sys.argv) > 1 else 6

SYSTEM = """# Identity

You are the AI assistant answering the phone for a small bakery. You take pickup orders over the phone, one item at a time, and confirm each item back before moving on.

# Voice style

- One or two short spoken sentences per turn. No lists, no markdown, no stage directions.
- Speak only your own lines. Never write the caller's lines.
- Ask one question at a time and wait for the answer.

# Ordering rules

- Use `search_menu` to look an item up before adding it. Never invent items or prices.
- Use `add_item` once the caller has picked a size and quantity.
- Use `get_pickup_times` before promising a pickup time.
- Repeat the full order back before finishing and ask if that is everything.

# Silence

If the caller goes quiet, check in once with a short line, then once more, then end the call politely if there is still no answer.
"""

INSTRUCTIONS = """<situation>The caller has gone quiet after your last line and has not answered yet.</situation>
<task>Say one short line to check whether the caller is still there and gently prompt them for what they would like to order. Keep it to a single sentence.</task>
<rules>Do not repeat the greeting word for word. Do not ask more than one question. Do not narrate.</rules>"""


async def _noop(raw_arguments: dict):
    return "ok"


def tools() -> list:
    specs = [
        ("search_menu", "Search the bakery menu by name.", {"query": {"type": "string"}}),
        ("add_item", "Add an item to the order.", {"name": {"type": "string"}, "quantity": {"type": "integer"}}),
        ("get_pickup_times", "List available pickup slots.", {}),
    ]
    return [
        llm.function_tool(
            _noop,
            raw_schema={"name": n, "description": d, "parameters": {"type": "object", "properties": p, "required": list(p)}},
        )
        for n, d, p in specs
    ]


def build() -> ChatContext:
    # what AgentSession leaves in chat_ctx right before generate_reply(instructions=...) on a silence timer
    ctx = ChatContext.empty()
    ctx.add_message(role="system", content=SYSTEM)
    ctx.add_message(role="assistant", content="Hi, thanks for calling the bakery, what can I get started for you?")
    ctx.add_message(role="user", content="What time could I pick up a dozen bagels?")
    ctx.add_message(role="assistant", content="Let me check the pickup times for you.")
    ctx.insert(llm.FunctionCall(call_id="call_1", name="get_pickup_times", arguments="{}"))
    ctx.insert(llm.FunctionCallOutput(call_id="call_1", name="get_pickup_times", output='{"slots": ["20 minutes", "45 minutes"]}', is_error=False))
    ctx.add_message(role="assistant", content="The next pickup slot is in twenty minutes, does that work for you?")
    ctx.add_message(role="system", content=INSTRUCTIONS)  # == generate_reply(instructions=INSTRUCTIONS)
    return ctx


async def one(client: inference.LLM, ctx: ChatContext, fns: list) -> str:
    out, calls = [], []
    async with client.chat(chat_ctx=ctx, tools=fns, tool_choice="auto") as stream:
        async for chunk in stream:
            if chunk.delta and chunk.delta.content:
                out.append(chunk.delta.content)
            if chunk.delta and chunk.delta.tool_calls:
                calls += [c.name for c in chunk.delta.tool_calls]
    return "".join(out).strip() + (f"  tools={calls}" if calls else "")


async def main() -> None:
    fns = tools()
    client = inference.LLM(model=MODEL, extra_kwargs={"temperature": 0.7})
    for label, ctx in (
        ("trailing system message (current behaviour)", build()),
        ("same message rewritten as user via convert_mid_conversation_instructions", convert_mid_conversation_instructions(build())),
    ):
        print(f"\n== {label}: roles={[getattr(m, 'role', m.type) for m in ctx.items]}")
        for _ in range(RUNS):
            print("  ", repr(await one(client, ctx.copy(), fns)))
    await client.aclose()


asyncio.run(main())

Output (6 runs each, temperature 0.7):

== trailing system message (current behaviour): roles=['system', 'assistant', 'user', 'assistant', 'function_call', 'function_call_output', 'assistant', 'system']
   "That works for me. I'll get those bagels started."
   'That works for me! Now, let me just double check the bagel options for you.'
   "That works great. Let me just double check the bagels for you.  tools=['search_menu']"
   "  tools=['search_menu']"
   'That works great. Now, let me just check the bagels on our menu for you.'
   "  tools=['search_menu']"

== same message rewritten as user via convert_mid_conversation_instructions: roles=['system', 'assistant', 'user', 'assistant', 'function_call', 'function_call_output', 'assistant', 'user']
   'Are you still there?'
   'Are you still there?'
   'Are you still there?'
   'Are you still there?'
   'Are you still there?'
   'Are you still there?'

With the trailing system role, every run either answers the assistant's own question as the caller would ("That works for me") or emits a bare search_menu call with no speech; 0/6 produce the check-in. With the message rewritten as a user turn it is 6/6.

In production we saw the same thing on real calls: after the first silence reminder the agent spoke a line that belonged to the caller, which then poisoned the rest of the conversation; on other calls it produced a parenthetical stage direction or apologised for a "mix-up" about who was supposed to be asking. Replaying those exact contexts gave ~1/8 clean with the trailing system role vs 6/6 as user, across five different wordings of the instruction — the role is what matters, not the prompt text.

A greeting-only history (no tool turn) usually comes out fine with this short prompt, so the tool-call turn in the reproduction is what makes it deterministic; long production prompts fail even without it.

Expected behaviour

inference.LLM applies convert_mid_conversation_instructions for Gemma models (as the google/anthropic/aws serializers already do), so generate_reply(instructions=...) works the same on Gemma as it does on Gemini/Claude/Bedrock.

Workaround

Subclass inference.LLM and convert in chat():

class InferenceLLM(inference.LLM):
    def chat(self, *, chat_ctx: ChatContext, **kwargs):
        if self.model.startswith("google/gemma"):
            chat_ctx = convert_mid_conversation_instructions(chat_ctx)
        return super().chat(chat_ctx=chat_ctx, **kwargs)

This fixed it for us on every injection site (silence check-ins, guard retries, handoff summaries) without touching the call sites.