Does smolagents offer structured outputs (constrained decoding not implicit)

[CakeCrusher]

@aymeric-roucher
I've been wondering what solutions HuggingFace offers with regards to structured outputs (SO) and have come short (the only thing I have found is dottxt/outlines, which is totally ok if HF intends to depend on them for SO).
This is particularly relevant for smolagents where SO compound in value.

I know this is not a very smolagents specific issue but nonetheless I think its highly consequential for smolagents.

Do you know what HF strategy towards structured outputs is? How are you guys thinking about it?

[NeuralNotwerk]

Smolagents has a grammar field through out all of the agentic workflows. As of yesterday, there weren't yet using it. I made my own "model" so I could use llama.cpp (llama-server) natively instead of through the only half functional litellm capability. When I made that "model", I added grammars to match the agentic parsing. It made even 1B parameter models like llama-3.2-1B functional (though not the best accuracy/correctness).

I still have absolutely NO idea why this isn't the standard approach with all agentic and tool calling behavior. Once you've used arbitrary grammars to enforce output structure, everything else comes down to basic prompting. It also makes it where you can flip one model out and throw in the next regardless of what kind of instruction fine tuning it has and prompt templating it expects - you effectively bake the 'compliance' into the grammar.

I think that most people, including some of the devs on this project, have no idea how powerful arbitrary grammars for creating structured outputs really are. The Ollama devs have dropped the ball repeatedly and won't pass the grammar through to llama.cpp which ollama is based on (and llama.cpp supports arbitrary grammars natively). Grammar constrained outputs are literally the only way to get agentic workflows reliable and efficient...and using SMALLER models.

Huge swing and a miss for this first set of releases not making grammars (or the lame, but still functional structured outputs) baked into this library. Also the random switch from basic python tuple to some insane hftoolcallingsomethingorother object was a pain in the functional breaking arse in a minor release.

I'm sure I'll get some people that will try to ignorantly fight this comment. They literally don't understand what grammars are and they think it's all part of the prompt or that it isn't absolute. When implemented correctly, it intercepts the token sampling process and only allows tokens that match the pattern to be selected. It can't be broken (unless you make a bad pattern, and that's on you).

[aymeric-roucher]

Hello @CakeCrusher, nice point!
What would you use structured outputs for? Generating correctly formed actions that can be parsed with 100% reliability? Or outputting a final output as the final answer? If it's the final answer, I'd suggest just doing a call to a structured LLM generation after the agent returns its answer, it will be more transparent and give you more freedom if you want to use a specific grammar. EDIT: see below

If it's to make tool calls more reliable, then this can be baked into your Model object! Most Model objects already support grammars, but I haven't had much feedback on that feature, please tell keep me informed if you try that!

@NeuralNotwerk good point about using smaller models with grammar, I agree that it can vastly improve reliability! Could you precise to what extent the current solutions don't work how you would want them to? (we can fix that in no time, especially if you open a PR with your changes)
Also regarding the switch to an object return type for Model objects, it makes their code much more simple to return a Message object (that can contain a tool call).

EDIT: We now have a much better way to enforce structured outputs from the agent! ✅

Just replace the agent's FinalAnswerTool with a custom version that has the input schema you want.

@tool
def final_answer(rationale: str, decision: Literal["BUY", "SELL", "PASS"]) -> EventDecisions:
    """
    This tool is used to validate and return the final event decision.
    
    This tool must be used only once. The rationale and decision must be provided in the same call.
    
    Args:
        rationale (str): The rationale for the decision.
        decision (Literal["BUY", "SELL", "PASS"]): The decision to make.
    """
    return ...

agent = CodeAgent(
    model=...
    tools=[..., final_answer]
)

We're extending this to support Pydantic schemas, in #1660.

[Hellisotherpeople]

@NeuralNotwerk is low key spitting FACTS about the power of constrained/structured generation with agents.

[NeuralNotwerk]

@aymeric-roucher - I'm using llama.cpp on my back end that allows for arbitrary grammars in an extended BNF (ebnf) form that is customized further into GBNF (ggml bnf). This gives me the best control over model output. For instance, with a .CodeAgent, I've got a grammar that defines the output as specified in the codeagent prompt template:

grammar = 'root ::= "Thoughts: " [a-zA-Z0-1. '\"\n]{0,500} "\nCode: \n" ("py"|"python") "\\n" .* "\\n<end_action>"'

I will probably get specific and instead of using the .*, I'll get a little more fancy to deny imports that aren't specifically listed in the BASE_BUILTIN_MODULES and only allow it function calls that are in the exposed tools or BASE_PYTHON_TOOLS.

The grammar for the tool calling agent is a little more complex as I've already gotten somewhat specific with it.
grammar = """
root ::= ws "Thought:" ws [a-zA-Z0-9 .]{0,500} ws "Action:\njson\\n" toolcall "\\n"
ws ::= (" " | "\n"){0,8}
toolcall ::= "{" ws "\"tool_name\": \"" toolnames "\"," ws "\"tool_arguments\": {" toolarguments (ws "," ws toolarguments)* "}" ws "}"
toolnames ::= ($%$TOOLS$%$)
toolarguments ::= ("\"" [a-zA-Z0-9_]* "\": " argvalues)
argvalues ::= object | array | string | number | ("true" | "false" | "null") ws
object ::= "{" ws ( string ":" ws argvalues ("," ws string ":" ws argvalues)* )? "}" ws
array ::= "[" ws ( argvalues ("," ws argvalues)* )? "]" ws
string ::= "\"" ([^"\\\\\\x7F\\x00-\\x1F] | "\\" (["\\bfnrt] | "u" [0-9a-fA-F]{4}) )* "\"" ws
number ::= ("-"? ([0-9] | [1-9] [0-9]{0,15})) ("." [0-9]+)? ([eE] [-+]? [0-9] [1-9]{0,15})? ws
"""
tools = ""
for tool in available_tools:
if len(tools) > 0:
tools += " | "
tools += f'"{tool.name}"'
grammar = grammar.replace("$%$TOOLS$%$", tools)

You can use the structured outputs capabilities in OpenAI, vLLM and Ollama in a similar way, but they only allow pure json constructed outputs which just makes the CodeAgent practically useless, though it would still work with tool calling agents. Fortunately, it does look like huggingface transformers library and some of the other components DO accept arbitrary grammars in the form of regex. I haven't used transformers in so long because it hadn't had the "guidance" feature when I initially found out how useful they were, only llama.cpp really supported it in terms of things I could run locally without modification. The same regex grammars that would work in transformers could likely be applied without modification to llama.cpp. They could even be applied in ollama if the devs would simply pass the argument through (as llama.cpp which it is based on already has it).

I would highly recommend making multiple paths for generating tool calling grammars. Generate both the OpenAI json structured outputs which should work on OpenAI, Ollama, and vLLM, and then also do an arbitrary grammar using regex for transformers and llama.cpp. The only caveat is the constraints would only be effective on tool calling for the back ends that only support json structured outputs where the regex could be more useful for other forms of constraint.

I can't understand why arbitrary grammars aren't the norm. I mean, I get it, things can get complex, but LLMs can be used to build the grammars too, haha. JSON structured outputs are a passable feature, but they hinder the more freeform complex workflows.

There's also times where I like to use grammars to constrain a model's output to yes/no only questions and follow decision trees. With an arbitrary grammar in llama.cpp of "root ::= "yes"|"no" and a decent prompt, these models become pretty powerful analysis widgets for automation. While this form of yes/no question could also work out in structured json outputs, JSON is pretty token heavy vs a single generated token representation of "yes" or "no". Considering most of my experimentation has been local and most of my applications are scaled (think FAANG companies...), single token constrained outputs make using LLMs more appropriate for business use and cost savings.

[Hellisotherpeople]

@NeuralNotwerk

vllm supports grammars/regex: https://docs.vllm.ai/en/latest/features/structured_outputs.html

[aymeric-roucher]

So I think we have a choice: do we think the grammar should be preset in the agent depending on its type (CodeAgent has one grammar, ToolCallingAgent has another, and they pass it when calling their model independently of which model it is, all models should accept the relevant grammars)

Or grammar handling is too model-specific, vLLM will accept completely different grammars than OpenAI API, thus we should leave grammar to the mode, and just decide to decide the grammar upon model initialization, like model = OpenAIServerModel(..., grammar=custom_grammar)? What do you think @NeuralNotwerk @Hellisotherpeople ?

[CakeCrusher]

It seems like the grammar is handled at the inference level is there a way I can use it with local inference?

[CakeCrusher]

This framework or at least CodeAgent is completely incompatible for consumer sized models, there is too many implicit based rules which make it impossible for any sufficiently small model to get anything done with the framework.

I think if anyone should develop something to support smol* models it should be HF.

[CakeCrusher]

@aymeric-roucher
Working on a solution here https://github.com/CakeCrusher/smolagents/tree/CakeCrusher/outlines-agent , will make a PR once its in a presentable condition.

At the moment this agent was able to run a dynamic workflow with 2 tools to completion using a meta-llama/Llama-3.2-3B-Instruct.

Here are a couple lessons and considerations moving forward:

1. The intended way of using smolagents using code as action, using CodeAgent, eliminates any consumer sized LLMs from being able to leverage this framework. Consequently, I reduced things to a much simpler json based execution using Outlines.
2. Planning has yet to be tested.
3. Current agent interface has been mutated to support outlines, not sure if I should create a new model class. if new model class is created, the model class should be able to accept json schema as an input
4. Code is a bit spaghetti.

[simeonwetzel]

Hello @CakeCrusher, nice point! What would you use structured outputs for? Generating correctly formed actions that can be parsed with 100% reliability? Or outputting a final output as the final answer? If it's the final answer, I'd suggest just doing a call to a structured LLM generation after the agent returns its answer, it will be more transparent and give you more freedom if you want to use a specific grammar.

If it's to make tool calls more reliable, then this can be baked into your Model object! Most Model objects already support grammars, but I haven't had much feedback on that feature, please tell keep me informed if you try that!

@NeuralNotwerk good point about using smaller models with grammar, I agree that it can vastly improve reliability! Could you precise to what extent the current solutions don't work how you would want them to? (we can fix that in no time, especially if you open a PR with your changes) Also regarding the switch to an object return type for Model objects, it makes their code much more simple to return a Message object (that can contain a tool call).

@aymeric-roucher A potential use case for structured outputs could be a RAG system, where you don't only want to get textual output (i.e. final_answer) but also the retrieved documents coming from a tool output in a structured, machine-readable way.
As far as I understood, the only way is to use a callback function with CodeAgent(..., step_callbacks=[] and parse the tool output from the AgentStep.observation or AgentStep.action_output using a regex, similar.
Did I miss something here?

[rht]

While @NeuralNotwerk made a compelling argument on #153 (comment) (in particular, that the token sampling interception makes it cheaper / reduces the number of shots), there is another solution: to validate using Pydantic.

At least using Pydantic is easier to write than rolling your own EBNF, in that the output is a Python dataclass, which makes it very reliable to compose with other agents or non-AI software components. Composing by parsing strings sounds more like a shell scripting mess than Rust safety. PydanticAI / Instructor nailed this properly, IMO.

[CakeCrusher]

@rht Outlines provides that solution,
ive created a fork that introduces an outlines agent that integrates with the current framework, waiting for approval though:
#487

You can refer to this to see how it works:
https://github.com/CakeCrusher/smolagents/blob/CakeCrusher/outlines-agent/tests/test_agents.py#L1021-L1054

NOTE: since the framework is opinionated on planning and execution of tools I had to horizontally abstract and create a different agent that inherits from the same class that CodeAgent does. In order to Incorporate Outlines into CodeAgent it will require a more expansive solution. For the moment `OutlinesAgent serves as a temporary fix.

@aymeric-roucher

[NeuralNotwerk]

While @NeuralNotwerk made a compelling argument on #153 (comment) (in particular, that the token sampling interception makes it cheaper / reduces the number of shots), there is another solution: to validate using Pydantic.

It's probably worth mentioning, if you are extracting text, you are already essentially doing some form of what an EBNF does. Most text parsing is still done with regex - if you can write the regex to parse it, you can literally turn around and use that regex to restrict it.

Maybe my grey beard is showing, but regex is just as natural to me as any other programming/scripting language. Wrapping things in the EBNF context is really the only change. It actually makes regex that would normally use lookaheads and lookbehinds much easier to read by simplifying the set of regex you'd need to use to accomplish the same goal.

At least using Pydantic is easier to write than rolling your own EBNF, in that the output is a Python dataclass, which makes it very reliable to compose with other agents or non-AI software components. Composing by parsing strings sounds more like a shell scripting mess than Rust safety. PydanticAI / Instructor nailed this properly, IMO.

Pydantic is only prompting wizardry which is fragile and easily influenced by attack (my day job is AI red teamer at a cloud company), there are no models that are not vulnerable to some form of prompt injection or jailbreak. This makes tool use and processing tool calls hazardous and expensive. Pydantic will try and fail and regenerate and fail and continue on it's path until you hit the upper limit of tokens or retries. You cannot get into this loop with grammars (assuming your grammars are appropriately implemented and defined).

There's no reason something like pydantic couldn't leverage grammars to make the library/framework even more effective - in fact there are a few issues where people have suggested it. Also, you can programmatically generate grammars in the same way smolagents (or pydantic) lets you throw in a function and it is parsed for creating the prompt to the agent - this is exactly the data you need to generate a grammar for a function (name, datatypes, enums, order of params, length restrictions, etc.).

I disagree that EBNFs are complex. It's just a new reality we need to move into. People are learning prompt engineering and tons of new AI related libraries, it's worth learning a method of constraining all of this output so we can move into better reliability and security with LLMs.

[rht]

@NeuralNotwerk I see, thank you for the insight.

There's no reason something like pydantic couldn't leverage grammars to make the library/framework even more effective - in fact there are a few issues where people have suggested it. Also, you can programmatically generate grammars in the same way smolagents (or pydantic) lets you throw in a function and it is parsed for creating the prompt to the agent - this is exactly the data you need to generate a grammar for a function (name, datatypes, enums, order of params, length restrictions, etc.).

I am interested in reading the discussion in the Pydantic issues you mentioned. If I understand correctly, the idea is to specify

from pydantic import BaseModel

class User(BaseModel):
    name: str
    age: int

and this will be automatically converted into an EBNF specification? I searched for "EBNF" in pydantic/pydantic and pydantic/pydantic-ai and found no mention of it.