feat(telemetry): emit the routing decision as span attributes - #3501
Open
Bekhouche wants to merge 1 commit into
Open
feat(telemetry): emit the routing decision as span attributes#3501Bekhouche wants to merge 1 commit into
Bekhouche wants to merge 1 commit into
Conversation
A collection.router decision was visible only to the client, on
x_lemonade_route. Inference spans now carry it too, under the existing llm.*
namespace: llm.route.{collection,to,matched_rule,default_used}, plus the
decision's estimated_cost flattened into llm.route.cost_* — so a trace shows
which candidate a policy picked and what that candidate was reported to cost.
The span is created inside Router, where the Decision is out of scope, so the
decision travels on a thread-local published by Server. That mirrors the
existing trace-context fields, including the per-request clear that keeps a
stale value from leaking onto a later request on a reused worker thread.
Published from route_collection_request, the one function every dispatch path
already goes through, so chat, completions and responses cannot drift apart.
Attribute mapping is a pure function so it is testable without a span: an
empty, malformed, or non-object payload emits nothing, a nested member is
skipped rather than stringified, and author-set estimated_cost keys cannot
shadow the decision's own.
Groundwork for showing routing and cost in the GUI3 Inspect view, which is fed
only by spans.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mirror a
collection.routerdecision onto the request's inference span. Spansnow carry
llm.route.{collection,to,matched_rule,default_used}under theexisting
llm.*namespace, plus the decision'sestimated_cost(from #2763)flattened into
llm.route.cost_*. Previously the decision was visible only tothe client on
x_lemonade_route, so a trace showed which model ran but not thata policy picked it, or why.
The span is created inside
Router, where theDecisionis out of scope, sothe decision travels on a thread-local published by
Server— the same patternas the existing trace-context fields, including the per-request clear that stops
a stale value leaking onto a later request on a reused worker. It is published
from
route_collection_request, the one function every dispatch path alreadygoes through, so chat, completions and responses cannot drift apart.
Distinct from the routing telemetry in #2968, which counts decisions and
switches server-wide; this is per-request detail on the span itself.
Note the cost half stays empty for local models today: nothing populates cost
metadata for them, since
USER_DEFINED_MODEL_PROPSdropscost_tier/cost_input_per_million/cost_output_per_millionon registration. Therouting fields are unaffected.
Scope
telemetry.h/telemetry.cpp— the thread-local, plusroute_span_attributes(payload -> attributes, a pure function) and
apply_route_attributes.server.cpp— per-request clear, and publishing the decision at the dispatchchoke point.
router.cpp— one call in each of the five span sites (chat,completions and responses, streaming and not).
test_telemetry_helpers.cpp.Testing
C++ build + full
cpp-ciCTest suite, 69/69. Twelve new cases cover themapping: an empty, malformed or non-object payload emits nothing, a nested
member is skipped rather than stringified, and author-set
estimated_costkeyscannot shadow the decision's own.
Also confirmed on a live server rather than only in unit tests. Registered a
router collection over two local GGUF candidates, connected to the spans
WebSocket, and captured the emitted span for a routed request:
Documentation
No docs change.
Breaking Changes
None. Purely additive attributes on an existing span; a request that is not
routed emits nothing new.
AI-assisted contribution
Yes.