Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/api/lemonade.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ The endpoint is available at:
| `prompt` | string | no | The prompt text to route. Defaults to `""`, which still exercises `min_chars` (0 chars) and any prompt-independent rules. |
| `has_images` | boolean | no | Simulate a request carrying image input. Default `false`. |
| `has_tools` | boolean | no | Simulate a request carrying tool definitions. Default `false`. |
| `turns` | integer | no | Simulate conversation depth for `min_turns`/`max_turns` conditions. Default `1`; a value below `1` (including `0`) floors to `1`, matching every real request. |
| `metadata` | object | no | String-valued metadata pairs matched by `metadata` conditions. |

### Example request
Expand Down Expand Up @@ -277,7 +278,7 @@ its own, only the synthesized `__route_0`, `__route_1`, … rules shown here.

| Status | Condition |
|--------|-----------|
| `400` | Body is not valid JSON, `policy` is missing or not an object, `prompt` is not a string, `has_images`/`has_tools` are not booleans, or `metadata` is not an object of string values. |
| `400` | Body is not valid JSON, `policy` is missing or not an object, `prompt` is not a string, `has_images`/`has_tools` are not booleans, `turns` is not a non-negative integer, or `metadata` is not an object of string values. |
| `400` | The policy document is invalid or internally inconsistent; the `error` field is prefixed with `Invalid routing policy:`. |

## `POST /v1/models/check-updates`
Expand Down
59 changes: 58 additions & 1 deletion docs/dev/router-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,9 @@ A `match` is a match-expression. Combine with the logical operators `any` (OR),
| `regex` | ECMAScript regex over the input. |
| `min_chars` / `max_chars` | Input length in UTF-8 bytes. |
| `min_total_chars` / `max_total_chars` | Length of **all** text in the request, in UTF-8 bytes. |
| `min_turns` / `max_turns` | Conversation depth (**whole-conversation scope**, like `total_chars` — not just the latest turn like `chars`) — count of `role:"user"` turns in `messages`/`input`. |
| `has_tools` / `has_images` | Boolean - request carries a non-empty `tools` array / image content parts. |
| `metadata` | `{ key, equals \| any \| exists }` over the request's OpenAI `metadata`. |
| `metadata` | `{ key, equals \| any \| exists \| gte \| lte }` over the request's OpenAI `metadata`. `gte`/`lte` parse the value as a number — useful for a harness-precomputed signal like a tool-error streak (see [trajectory-signal routing](#trajectory-signal-routing) below). |

The text conditions above - `keywords_any` / `keywords_all`, `regex`, and the
`chars` pair - see only the **routing input**: the last user message (or the
Expand Down Expand Up @@ -179,6 +180,62 @@ and every entry's `model` must be one of `components`:
> labels are the candidate models); a `type: "llm"` classifier only produces a
> label that rules combine with any other condition.

## Trajectory-signal routing

`min_turns`/`max_turns` and `metadata`'s `gte`/`lte` comparators exist to
route on *how a conversation is going*, not just the current request's text
— e.g. escalate to a bigger model once a coding agent has stalled or hit
repeated tool errors.

`min_turns`/`max_turns` are native: the engine counts `role:"user"` turns in
`messages`/`input` itself, no caller work required.

Anything harness-specific — a tool-call error streak, "no edits in the last N
turns", a result-pattern match — is intentionally **not** parsed by the
engine itself. Different agent harnesses format tool results differently and
name their own tools differently (there's no universal "error" field on an
OpenAI tool-result message), and baking harness assumptions into the router
would make it not-generic. Instead, the calling harness — which already
knows its own tool/error conventions — computes the signal itself and sends
it as a plain number in `metadata`:

```json
{
"messages": [...],
"metadata": { "tool_error_streak": "3" }
}
```

```json
"routing": {
"rules": [
{
"id": "escalate-after-stalls",
"match": {
"all": [
{ "min_turns": 4 },
{ "metadata": { "key": "tool_error_streak", "gte": 2 } }
]
},
"route_to": "Big-GGUF"
}
]
}
```

A missing or non-numeric `metadata` value never satisfies `gte`/`lte` (same
"absent counts as no match" posture as `equals`/`any`).

A single `metadata` leaf allows exactly one comparator, so a numeric range
needs **two** leaves under `all` rather than `{"gte": 1, "lte": 5}` on one:

```json
{ "all": [
{ "metadata": { "key": "tool_error_streak", "gte": 1 } },
{ "metadata": { "key": "tool_error_streak", "lte": 5 } }
] }
```

## Registering and invoking

Register the policy like any collection — `POST /v1/pull` with the policy JSON:
Expand Down
7 changes: 7 additions & 0 deletions src/cpp/include/lemon/routing_policy.h
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,13 @@ struct RouteContext {
std::size_t chars = 0; // UTF-8 byte count of `input`
std::size_t total_chars = 0; // UTF-8 byte count of all request text

// Conversation depth: the count of user-role turns in the request's
// `messages` (chat/completions) or `input` (Responses) array, at
// least 1 for any request that reaches the engine. Harness-agnostic
// (pure message count, no tool-call/error semantics) — see
// min_turns/max_turns.
std::size_t turn_count = 0;

// The caller's requested output-length ceiling (OpenAI `max_tokens` /
// `max_completion_tokens`), when present. This is a ceiling, not an
// estimate of actual completion length — nullopt when the caller
Expand Down
3 changes: 2 additions & 1 deletion src/cpp/resources/schemas/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,8 @@ v1** (pinned in the schema field descriptions):
| `min_score` / `max_score` | **inclusive** band (`>=` / `<=`); default `min_score: 0.5` when neither bound is given |
| `min_chars` / `max_chars` | input length in **UTF-8 bytes** (not code points) |
| `min_total_chars` / `max_total_chars` | length of **all text content in the request** in **UTF-8 bytes** — chat `messages` summed over every role, a legacy `prompt` (equal to `chars`), or a Responses `input` array summed over all items; non-text parts contribute nothing |
| `metadata` | reads a request `metadata` key; **case-sensitive** comparison, value decoded into a comma-split, trimmed **token set** (`equals` raw exact / `any` set-intersection / `exists` presence). A missing, empty, or **whitespace-only** value counts as absent (matches only `exists:false`) |
| `min_turns` / `max_turns` | conversation depth: count of `role:"user"` turns in `messages`/`input`; harness-agnostic (pure message count) |
| `metadata` | reads a request `metadata` key; **case-sensitive** comparison, value decoded into a comma-split, trimmed **token set** (`equals` raw exact / `any` set-intersection / `exists` presence). `gte`/`lte` parse the value as a number instead. A missing, empty, or **whitespace-only** value counts as absent (matches only `exists:false`, and never satisfies `gte`/`lte`); a non-numeric value likewise never satisfies `gte`/`lte` |
| multi-key leaf object | interpreted as implicit **`all`**; e.g. `{"keywords_any":[...],"max_chars":1000}` means both leaves must match |
| `on_error` (omitted) | default **`match_false`** (fail-open) |
| `routing.router` desugaring | expansion to one `llm` classifier + identity rules is deterministic and behavior-equivalent across versions |
Expand Down
24 changes: 22 additions & 2 deletions src/cpp/resources/schemas/route_policy.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -203,14 +203,24 @@
"type": "integer",
"minimum": 0
},
"min_turns": {
"description": "Inclusive lower bound on conversation depth: the count of role:\"user\" turns in the request's messages/input array (at least 1 for any request that reaches the engine). Harness-agnostic — pure message count, no tool-call or error semantics.",
"type": "integer",
"minimum": 0
},
"max_turns": {
"description": "Inclusive upper bound on conversation depth, same unit as min_turns.",
"type": "integer",
"minimum": 0
},
"has_tools": { "type": "boolean" },
"has_images": { "type": "boolean" },
"metadata": { "$ref": "#/$defs/metadata_match" }
},
"additionalProperties": false
},
"metadata_match": {
"description": "Deterministic leaf matching a caller-supplied OpenAI `metadata` key (e.g. task_class, consent), read verbatim from the request metadata map. Frozen v1 semantics: values are compared case-sensitively; a value is decoded into a token set by splitting on comma and trimming, so scalar and list-valued metadata match uniformly. A missing, empty, or whitespace-only value counts as absent (matches only `exists: false`). Exactly one comparator (equals / any / exists) must be present. A future comparator (regex, all, ...) ships as a new key, never a redefinition of these.",
"description": "Deterministic leaf matching a caller-supplied OpenAI `metadata` key (e.g. task_class, consent, or a harness-precomputed signal like tool_error_streak), read verbatim from the request metadata map. Frozen v1 semantics: string comparators (equals/any) are case-sensitive; a value is decoded into a token set by splitting on comma and trimming, so scalar and list-valued metadata match uniformly. A missing, empty, or whitespace-only value counts as absent (matches only `exists: false`, and never satisfies gte/lte). gte/lte parse the value as a number (leading/trailing ASCII whitespace tolerated, otherwise the entire value must be numeric); a non-numeric value never matches. Exactly one comparator (equals / any / exists / gte / lte) must be present. A future comparator ships as a new key, never a redefinition of these.",
"type": "object",
"required": ["key"],
"properties": {
Expand All @@ -232,12 +242,22 @@
"exists": {
"description": "exists:true matches when the key is present and non-empty; exists:false matches when it is absent or empty.",
"type": "boolean"
},
"gte": {
"description": "True if the metadata value parses as a number >= this threshold. A missing or non-numeric value never matches.",
"type": "number"
},
"lte": {
"description": "True if the metadata value parses as a number <= this threshold. A missing or non-numeric value never matches.",
"type": "number"
}
},
"oneOf": [
{ "required": ["equals"] },
{ "required": ["any"] },
{ "required": ["exists"] }
{ "required": ["exists"] },
{ "required": ["gte"] },
{ "required": ["lte"] }
],
"additionalProperties": false
}
Expand Down
2 changes: 1 addition & 1 deletion src/cpp/resources/schemas/schema-lock.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"route_policy.schema.json": {
"sha256": "b924faaad344dad8b87ca9dbcd2fbe7e94ee9299922f21e06f6b96edcb1dda3e",
"sha256": "7d1d209472f478a46a706314c0152449ad61712abd7a0f712ded6d2a74a9da62",
"released": false
},
"decision.schema.json": {
Expand Down
37 changes: 29 additions & 8 deletions src/cpp/server/routing_classifier_services.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -390,12 +390,17 @@ RouteContext build_route_context(const json& request_json, const std::string& mo

if (request_json.contains("messages") && request_json["messages"].is_array()) {
const auto& messages = request_json["messages"];
// Totalling every role's text is what makes `total_chars` a proxy for
// prefill size: the whole array is what the backend prefills, not just
// the routing turn. It cannot ride along with an image scan that stops
// at the first hit, so the flag is latched instead of breaking out.
// One forward pass counts user turns, detects images, and totals every
// role's text (the `total_chars` proxy for prefill size) together,
// rather than three separate walks that could drift apart. Totalling
// cannot ride along with an image scan that stops at the first hit, so
// has_images is latched instead of breaking out.
for (const auto& msg : messages) {
if (!msg.is_object() || !msg.contains("content")) continue;
if (!msg.is_object()) continue;
if (msg.value("role", std::string()) == "user") {
++ctx.params.turn_count;
}
if (!msg.contains("content")) continue;
const auto& content = msg["content"];
if (!ctx.params.has_images && content_has_image(content)) {
ctx.params.has_images = true;
Expand All @@ -412,6 +417,8 @@ RouteContext build_route_context(const json& request_json, const std::string& mo
}
} else if (request_json.contains("prompt")) {
const auto& prompt = request_json["prompt"];
// Legacy completions has no multi-turn concept: the whole prompt is one turn.
ctx.params.turn_count = 1;
if (prompt.is_string()) {
ctx.input = prompt.get<std::string>();
} else if (prompt.is_array()) {
Expand All @@ -428,12 +435,17 @@ RouteContext build_route_context(const json& request_json, const std::string& mo
const auto& input = request_json["input"];
if (input.is_string()) {
ctx.input = input.get<std::string>();
ctx.params.turn_count = 1;
ctx.params.total_chars = ctx.input.size();
} else if (input.is_array()) {
// Detect images anywhere in the input, mirroring how the chat path
// scans every message, and total every item's text in the same pass
// — role-tagged or bare, which is more than the routing input takes.
// One forward pass counts user turns, detects images anywhere in the
// input (mirroring how the chat path scans every message), and
// totals every item's text — role-tagged or bare, which is more
// than the routing input takes — rather than separate walks.
for (const auto& item : input) {
if (item.is_object() && item.value("role", std::string()) == "user") {
++ctx.params.turn_count;
}
if (item.is_string()) {
ctx.params.total_chars += item.get<std::string>().size();
continue;
Expand Down Expand Up @@ -479,6 +491,15 @@ RouteContext build_route_context(const json& request_json, const std::string& mo
}
}

// Any request that reaches the engine is at least one turn — including
// one with no messages/prompt/input field at all, or an empty
// messages/input array — so min_turns/max_turns' "at least 1" invariant
// holds unconditionally rather than only on the branches that happened
// to set turn_count already.
if (ctx.params.turn_count == 0) {
ctx.params.turn_count = 1;
}

ctx.params.chars = ctx.input.size();

if (request_json.contains("metadata") && request_json["metadata"].is_object()) {
Expand Down
Loading
Loading