Skip to content

feat: add LLM::Tracer::Multi - #277

Open
azmi2409 wants to merge 1 commit into
r-uby-dev:mainfrom
azmi2409:feat/273-tracer-multi
Open

azmi2409 wants to merge 1 commit into
r-uby-dev:mainfrom
azmi2409:feat/273-tracer-multi

Conversation

@azmi2409

Copy link
Copy Markdown
Contributor

Fixes #273

Adds LLM::Tracer::Multi, a tracer that fans out every tracing
callback to a set of tracers, so a request can (for example) be
logged to a file and exported to OpenTelemetry at the same time:

LLM::Tracer::Multi.new(
  LLM::Tracer.logger(llm, io: $stdout),
  LLM::Tracer.telemetry(llm)
)

Design notes:

  • Spans returned by on_request_start / on_tool_start are tracked
    per tracer in an LLM::Tracer::Multi::Spans wrapper, and each
    tracer is handed back the span it returned when the request or
    tool finishes, errors, or is interrupted.
  • A tracer that raises does not take the others down with it: the
    error is reported the way LLM::Tracer::Rescue reports a crashed
    tracer, and the remaining tracers still run. LLM::Interrupt
    still travels up the stack.
  • start_trace / stop_trace, set_finish_metadata_proc, and
    merge_extra fan out; spans aggregates every tracer's spans;
    flush! fans out.
  • The set tracer: :multi_tracer form from the issue already works:
    agent options resolve a symbol through a (possibly private) method
    via LLM.resolve_option, so no agent-side change was needed.

Tests: new spec/tracer/multi_spec.rb (15 examples, all green).
Full spec/tracer + spec/agent_spec.rb run: 310 examples,
0 failures.

@altruby altruby left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your contribution as always :)
This looks good, but I think we can make a few improvements.
What do you think? Cheers.

Comment thread lib/llm/tracer/multi.rb
@@ -0,0 +1,242 @@
# frozen_string_literal: true

module LLM

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an unwritten rule in llm.rb that we open - at most - two levels of nesting.
So for example - this should be:

class LLM::Tracer
  class Multi < self
  end
end

Comment thread lib/llm/tracer/multi.rb
# tracer, so the remaining tracers still run. An
# {LLM::Interrupt LLM::Interrupt} travels up the stack, like
# everywhere else in the gem.
def safely(tracer)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can drop this method.
There are two reasons or scenarios to think about, and we cover both of them without this code.

Scenario 1

The first scenario is where Multi raises an error. In that case Multi cannot forward the method call to the tracer(s) it wraps, and we never reach them. Since Multi inherits LLM::Tracer::Rescue - like all classes who inherit from LLM::Tracer - it means when a Multi callback does raise an error, we already catch it via the Rescue module it has inherited, and we would print something like this:

an llm.rb tracer has crashed
[tracer] LLM::Tracer::Multi
[class  ] NotImplementedError
# ...

Scenario 2

The second scenario is where the Multi tracer does not raise an error but one of the tracers that it wraps does. This is also covered by the Rescue module, and it covers every tracer under Multi's supervision. In this case, we would print something like:

an llm.rb tracer has crashed
[tracer] TracerBeingWrapped
[class  ] NotImplementedError
# ...

And the exception never reaches the Multi tracer. Does that make sense?

Comment thread lib/llm/tracer/multi.rb
self
end

##

@altruby altruby Oct 11, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spans and flush! methods are unique to the Telemetry tracer as far as I know.
We shouldn't implement them here. I think we should consider something like:

class Multi < self
 ##
 # Returns the tracer(s) under the supervision of Multi
 # @return [Array<LLM::Tracer>]
 attr_reader :tracers
end

Then if you need to access spans, or flush, you can do:

t = multi.tracers.grep(LLM::Tracer::Telemetry).first
t.spans
t.flush!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add LLM::Tracer::Multi

2 participants