Skip to content

feat(telemetry): emit the routing decision as span attributes - #3501

Open
Bekhouche wants to merge 1 commit into
lemonade-sdk:mainfrom
Bekhouche:feat/telemetry-route-attributes
Open

feat(telemetry): emit the routing decision as span attributes#3501
Bekhouche wants to merge 1 commit into
lemonade-sdk:mainfrom
Bekhouche:feat/telemetry-route-attributes

Conversation

@Bekhouche

Copy link
Copy Markdown
Collaborator

Summary

Mirror a collection.router decision onto the request's inference span. Spans
now carry llm.route.{collection,to,matched_rule,default_used} under the
existing llm.* namespace, plus the decision's estimated_cost (from #2763)
flattened into llm.route.cost_*. Previously the decision was visible only to
the client on x_lemonade_route, so a trace showed which model ran but not that
a policy picked it, or why.

The span is created inside Router, where the Decision is out of scope, so
the decision travels on a thread-local published by Server — the same pattern
as the existing trace-context fields, including the per-request clear that stops
a stale value leaking onto a later request on a reused worker. It is published
from route_collection_request, the one function every dispatch path already
goes through, so chat, completions and responses cannot drift apart.

Distinct from the routing telemetry in #2968, which counts decisions and
switches server-wide; this is per-request detail on the span itself.

Note the cost half stays empty for local models today: nothing populates cost
metadata for them, since USER_DEFINED_MODEL_PROPS drops cost_tier /
cost_input_per_million / cost_output_per_million on registration. The
routing fields are unaffected.

Scope

telemetry.h / telemetry.cpp — the thread-local, plus route_span_attributes
(payload -> attributes, a pure function) and apply_route_attributes.
server.cpp — per-request clear, and publishing the decision at the dispatch
choke point. router.cpp — one call in each of the five span sites (chat,
completions and responses, streaming and not). test_telemetry_helpers.cpp.

Testing

C++ build + full cpp-ci CTest suite, 69/69. Twelve new cases cover the
mapping: an empty, malformed or non-object payload emits nothing, a nested
member is skipped rather than stringified, and author-set estimated_cost keys
cannot shadow the decision's own.

Also confirmed on a live server rather than only in unit tests. Registered a
router collection over two local GGUF candidates, connected to the spans
WebSocket, and captured the emitted span for a routed request:

SPAN name=chat.completions
llm.backend            = llamacpp
gen_ai.request.model   = Qwen3-1.7B-GGUF
llm.route.collection   = user.Router-Demo
llm.route.matched_rule = code-to-bigger
llm.route.to           = Qwen3-1.7B-GGUF
llm.route.default_used = False

Documentation

No docs change.

Breaking Changes

None. Purely additive attributes on an existing span; a request that is not
routed emits nothing new.

AI-assisted contribution

Yes.

A collection.router decision was visible only to the client, on
x_lemonade_route. Inference spans now carry it too, under the existing llm.*
namespace: llm.route.{collection,to,matched_rule,default_used}, plus the
decision's estimated_cost flattened into llm.route.cost_* — so a trace shows
which candidate a policy picked and what that candidate was reported to cost.

The span is created inside Router, where the Decision is out of scope, so the
decision travels on a thread-local published by Server. That mirrors the
existing trace-context fields, including the per-request clear that keeps a
stale value from leaking onto a later request on a reused worker thread.
Published from route_collection_request, the one function every dispatch path
already goes through, so chat, completions and responses cannot drift apart.

Attribute mapping is a pure function so it is testable without a span: an
empty, malformed, or non-object payload emits nothing, a nested member is
skipped rather than stringified, and author-set estimated_cost keys cannot
shadow the decision's own.

Groundwork for showing routing and cost in the GUI3 Inspect view, which is fed
only by spans.
@github-actions github-actions Bot added the enhancement New feature or request label Sep 3, 2026
@Bekhouche
Bekhouche marked this pull request as ready for review September 3, 2026 17:28
@Bekhouche Bekhouche self-assigned this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant