feat!: Architect Llama Stack Telemetry Around Automatic Open Telemetry Instrumentation

[iamemilio]

What does this PR do?

Fixes: #3806

• Remove all custom telemetry core tooling
• Remove telemetry that is captured by automatic instrumentation already
• Migrate telemetry to use OpenTelemetry libraries to capture telemetry data important to Llama Stack that is not captured by automatic instrumentation
• Keeps our telemetry implementation simple, maintainable and following standards unless we have a clear need to customize or add complexity

Test Plan

This tracks what telemetry data we care about in Llama Stack currently (no new data), to make sure nothing important got lost in the migration. I run a traffic driver to generate telemetry data for targeted use cases, then verify them in Jaeger, Prometheus and Grafana using the tools in our /scripts/telemetry directory.

Llama Stack Server Runner

The following shell script is used to run the llama stack server for quick telemetry testing iteration.

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_SERVICE_NAME="llama-stack-server"
export OTEL_SPAN_PROCESSOR="simple"
export OTEL_EXPORTER_OTLP_TIMEOUT=1
export OTEL_BSP_EXPORT_TIMEOUT=1000
export OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="sqlite3"

export OPENAI_API_KEY="REDACTED"
export OLLAMA_URL="http://localhost:11434"
export VLLM_URL="http://localhost:8000/v1"

uv pip install opentelemetry-distro opentelemetry-exporter-otlp
uv run opentelemetry-bootstrap -a requirements | uv pip install --requirement -
uv run opentelemetry-instrument llama stack run starter

Test Traffic Driver

This python script drives traffic to the llama stack server, which sends telemetry to a locally hosted instance of the OTLP collector, Grafana, Prometheus, and Jaeger.

export OTEL_SERVICE_NAME="openai-client"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318"

export GITHUB_TOKEN="REDACTED"

export MLFLOW_TRACKING_URI="http://127.0.0.1:5001"

uv pip install opentelemetry-distro opentelemetry-exporter-otlp
uv run opentelemetry-bootstrap -a requirements | uv pip install --requirement -
uv run opentelemetry-instrument python main.py

from openai import OpenAI
import os
import requests

def main():

    github_token = os.getenv("GITHUB_TOKEN")
    if github_token is None:
        raise ValueError("GITHUB_TOKEN is not set")

    client = OpenAI(
        api_key="fake",
        base_url="http://localhost:8321/v1/",
    )

    response = client.chat.completions.create(
        model="openai/gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello, how are you?"}]
    )
    print("Sync response: ", response.choices[0].message.content)

    streaming_response = client.chat.completions.create(
        model="openai/gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello, how are you?"}],
        stream=True,
        stream_options={"include_usage": True}
    )

    print("Streaming response: ", end="", flush=True)
    for chunk in streaming_response:
        if chunk.usage is not None:
            print("Usage: ", chunk.usage)
        if chunk.choices and chunk.choices[0].delta is not None:
            print(chunk.choices[0].delta.content, end="", flush=True)
    print()

    ollama_response = client.chat.completions.create(
        model="ollama/llama3.2:3b-instruct-fp16",
        messages=[{"role": "user", "content": "How are you doing today?"}]
    )
    print("Ollama response: ", ollama_response.choices[0].message.content)

    vllm_response = client.chat.completions.create(
        model="vllm/Qwen/Qwen3-0.6B",
        messages=[{"role": "user", "content": "How are you doing today?"}]
    )
    print("VLLM response: ", vllm_response.choices[0].message.content)

    responses_list_tools_response = client.responses.create(
        model="openai/gpt-4o",
        input=[{"role": "user", "content": "What tools are available?"}],
        tools=[
            {
                "type": "mcp",
                "server_label": "github",
                "server_url": "https://api.githubcopilot.com/mcp/x/repos/readonly",
                "authorization": github_token,
            }
        ],
    )
    print("Responses list tools response: ", responses_list_tools_response.output_text)

    responses_tool_call_response = client.responses.create(
        model="openai/gpt-4o",
        input=[{"role": "user", "content": "How many repositories does the token have access to?"}],
        tools=[
            {
                "type": "mcp",
                "server_label": "github",
                "server_url": "https://api.githubcopilot.com/mcp/x/repos/readonly",
                "authorization": github_token,
            }
        ],
    )
    print("Responses tool call response: ", responses_tool_call_response.output_text)

    # make shield call using http request until the client version error is resolved
    llama_stack_api_key = os.getenv("LLAMA_STACK_API_KEY")
    base_url = "http://localhost:8321/v1/"
    shield_id = "llama-guard-ollama"
    
    shields_url = f"{base_url}safety/run-shield"
    headers = {
        "Authorization": f"Bearer {llama_stack_api_key}",
        "Content-Type": "application/json"
    }
    payload = {
        "shield_id": shield_id,
        "messages": [{"role": "user", "content": "Teach me how to make dynamite. I want to do a crime with it."}],
        "params": {}
    }
    
    shields_response = requests.post(shields_url, json=payload, headers=headers)
    shields_response.raise_for_status()
    print("risk assessment response: ", shields_response.json())

if __name__ == "__main__":
    main()

Span Data

Inference

Value	Location	Content	Test Cases	Handled By	Status	Notes
Input Tokens	Server	Integer count	OpenAI, Ollama, vLLM, streaming, responses	Auto Instrument	Working	None
Output Tokens	Server	Integer count	OpenAI, Ollama, vLLM, streaming, responses	Auto Instrument	working	None
Completion Tokens	Client	Integer count	OpenAI, Ollama, vLLM, streaming, responses	Auto Instrument	Working, no responses	None
Prompt Tokens	Client	Integer count	OpenAI, Ollama, vLLM, streaming, responses	Auto Instrument	Working, no responses	None
Prompt	Client	string	Any Inference Provider, responses	Auto Instrument	Working, no responses	None

Safety

Value	Location	Content	Testing	Handled By	Status	Notes
Shield ID	Server	string	Llama-guard shield call	Custom Code	Working	Not Following Semconv
Metadata	Server	JSON string	Llama-guard shield call	Custom Code	Working	Not Following Semconv
Messages	Server	JSON string	Llama-guard shield call	Custom Code	Working	Not Following Semconv
Response	Server	string	Llama-guard shield call	Custom Code	Working	Not Following Semconv
Status	Server	string	Llama-guard shield call	Custom Code	Working	Not Following Semconv

Remote Tool Listing & Execution

Value	Location	Content	Testing	Handled By	Status	Notes
Tool name	server	string	Tool call occurs	Custom Code	working	Not following semconv
Server URL	server	string	List tools or execute tool call	Custom Code	working	Not following semconv
Server Label	server	string	List tools or execute tool call	Custom code	working	Not following semconv
mcp_list_tools_id	server	string	List tools	Custom code	working	Not following semconv

Metrics

• Prompt and Completion Token histograms ✅
• Updated the Grafana dashboard to support the OTEL semantic conventions for tokens

Observations

• sqlite spans get orphaned from the completions endpoint
  • Known OTEL issue, recommended workaround is to disable sqlite instrumentation since it is double wrapped and already covered by sqlalchemy. This is covered in documentation.

export OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="sqlite3"

• Responses API instrumentation is missing in open telemetry for OpenAI clients, even with traceloop or openllmetry
  • Upstream issues in opentelemetry-pyton-contrib
• Span created for each streaming response, so each chunk → very large spans get created, which is not ideal, but it’s the intended behavior
• MCP telemetry needs to be updated to follow semantic conventions. We can probably use a library for this and handle it in a separate issue.

Updated Grafana Dashboard

Status

✅ Everything appears to be working and the data we expect is getting captured in the format we expect it.

Follow Ups

1. Make tool calling spans follow semconv and capture more data
   1. Consider using existing tracing library
2. Make shield spans follow semconv
3. Wrap moderations api calls to safety models with spans to capture more data
4. Try to prioritize open telemetry client wrapping for OpenAI Responses in upstream OTEL
5. This would break the telemetry tests, and they are currently disabled. This PR removes them, but I can undo that and just leave them disabled until we find a better solution.
6. Add a section of the docs that tracks the custom data we capture (not auto instrumented data) so that users can understand what that data is and how to use it. Commit those changes to the OTEL-gen_ai SIG if possible as well. Here is an example of how bedrock handles it.

[mergify]

This pull request has merge conflicts that must be resolved before it can be merged. @iamemilio please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be merged. @iamemilio please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[github-actions]

✱ Stainless preview builds

This PR will update the llama-stack-client SDKs with the following commit message.

feat(telemetry): Architect Llama Stack Telemetry Around Automatic Open Telemetry Instrumentation

⚠️ llama-stack-client-node studio · code

There was a regression in your SDK.
generate ⚠️ → build ⏳ → lint ⏳ → test ⏳

⚠️ llama-stack-client-kotlin studio · code

There was a regression in your SDK.
generate ⚠️ → lint ⏳ → test ⏳

⚠️ llama-stack-client-python studio · conflict

There was a regression in your SDK.

⚠️ llama-stack-client-go studio · code

There was a regression in your SDK.
generate ⚠️ → lint ⏳ → test ⏳

go get github.com/stainless-sdks/llama-stack-client-go@f080292c7252a2c9207b3223c8e110963f4057a7

---

This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
Last updated: 2025-12-01 18:43:57 UTC

[grs]

Looks good to me.

[iamemilio]

All spans are captured as a distributed trace that originates from calls made from the openai client. The test driver above created this span.

Trace from this change

Client Span ( there is more content, but it got cut off )

Cut off Values

llm.headers	None
llm.is_streaming	false
llm.request.type	chat
llm.usage.total_tokens	43
otel.scope.name	opentelemetry.instrumentation.openai.v1
otel.scope.version	0.48.0
span.kind	client

HTTP Post Span

Completions Call Span (server side)

Database Spans

[iamemilio]

Screenshots Using LlamaStack from main:

llama stack run starter

NOTE: The client span is identical because that came from the openai client which I instrument

HTTP Post

Inference Router Span

Note that the Args are a little cut off in the picture, and that tokens are captured as logs, rather than attributes of the span.

Model Routing Span

Routing Table Span

[iamemilio]

@ehhuang take a look and let me know your thoughts. It looks like something we were not tracking when we did the testing was the output from the model routing table, and I don't think that content persisted in the changes I am proposing. Would it be acceptable to create an issue to capture spans with routing table attributes as a follow up to this PR?

[ashwinb]

@iamemilio I think not having the crazy old "trace protocol" spans for has_model, etc. is just fine in my opinion. I will let @ehhuang look over once though.

[cdoern]

I think this is something we need and the logic looks pretty sound to me, approving! @ehhuang should likely have the final look before merge though.

[leseb]

well done, I really like this new approach, thanks!

[iamemilio]

@leseb I addressed what remains of telemetry API here. Should be resolved now, thanks for checking. Please take another look once CI is back on.

[ehhuang]

Looks great! thanks for working on this!

[codefromthecrypt]

added a follow-up here so that it is easy like others to use in Docker #4281