RFC: Vendor-neutral gen_ai.* semantic convention for LLM observability — feedback from Phoenix/Arize team welcome

[sauravGit]

Hi Phoenix / Arize community,

I'm proposing a vendor-neutral, OpenTelemetry-compatible semantic convention and SDK layer for standardizing LLM observability metrics, and I'd love input from practitioners and maintainers who are building and using LLM evaluation and observability tools in production.

The problem: Teams instrumenting LLM applications today face a fragmented metric landscape. Every tool — including Phoenix, Langfuse, OpenLLMetry, and others — uses slightly different names and schemas for the same signals (latency, token counts, cost, error rates). There's no shared language at the metric layer.

The proposal: A canonical gen_ai.* metric schema built on top of OpenTelemetry GenAI semantic conventions:

• gen_ai.latency, gen_ai.time_to_first_token (histograms)
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens (counters)
• gen_ai.usage.cost (counter)
• gen_ai.error_rate, gen_ai.retry_rate (derived)
• Required attributes: gen_ai.system, gen_ai.request.model, gen_ai.operation.name
• Interoperability rules: backends that emit these attributes in any format can map to the canonical schema

Why Phoenix specifically: Arize Phoenix is one of the most respected open-source LLM tracing and evaluation platforms. The team and community here have deep experience with what metrics practitioners actually need in production — especially around quality signals like groundedness, relevance, and retrieval. I'd like to understand:

1. What metric names does Phoenix currently store/export for latency, TTFT, and token usage?
2. Are there quality/evaluation metrics (hallucination score, retrieval precision, etc.) that should be in a gen_ai.eval.* extension namespace alongside core KPIs?
3. Would Phoenix be interested in formally supporting this convention for its OTEL export layer?

Links:

• RFC + full spec: https://github.com/sauravGit/open-llm-observability/blob/main/RFC.md
• GitHub Discussion: RFC: Universal LLM Observability Semantic Convention (v0.4) sauravGit/open-llm-observability#1
• OTel spec discussion: RFC feedback request: Vendor-neutral LLM observability semantic convention (gen_ai.* metrics) — looking for input from GenAI SIG maintainers open-telemetry/opentelemetry-specification#5069
• Python SDK scaffold: https://github.com/sauravGit/open-llm-observability/tree/main/sdk/python

This is v0.1 — explicitly designed to evolve based on community input. The goal is a shared language for LLM telemetry, not another vendor tool.

[sauravGit]

Update: RFC v0.2 now available — with Phoenix-specific migration table and gen_ai.eval.* extension

Since posting this, the RFC has been significantly expanded. Key additions most relevant to Phoenix/Arize:

Corrected canonical names:

• gen_ai.client.operation.duration — Histogram, s
• gen_ai.client.time_to_first_token — Histogram, s
• gen_ai.usage.input_tokens — Counter, {token}
• gen_ai.usage.output_tokens — Counter, {token}
• gen_ai.usage.cost — Counter, usd

Migration mapping table for Phoenix specifically:

Phoenix attribute	Canonical name
llm.latency_ms	gen_ai.client.operation.duration (convert ms→s)
llm.token_count.prompt	gen_ai.usage.input_tokens
llm.token_count.completion	gen_ai.usage.output_tokens

gen_ai.eval.* Extension Pack (directly relevant to Phoenix):

Metric	Type	Description
gen_ai.eval.groundedness	Histogram	Groundedness score [0,1]
gen_ai.eval.faithfulness	Histogram	Faithfulness score [0,1]
gen_ai.eval.relevance	Histogram	Context relevance score [0,1]
gen_ai.eval.toxicity	Histogram	Toxicity score [0,1]

This directly addresses question #2 about quality/evaluation metrics. The eval extension pack is designed to be optional and versioned independently from core.

Also new:

• gen_ai.rag.* extension pack with retrieval latency, context recall, and context token ratio metrics
• An OTel maintainer (@trask) engaged and pointed to semantic-conventions-genai as the right upstream venue
• PromQL dashboard templates including Tier 3 RAG-specific panels
• Versioning policy, working SDK examples, migration guide

Full RFC v0.2: https://github.com/sauravGit/open-llm-observability/blob/main/RFC.md

Would still love Phoenix team input on whether gen_ai.eval.* naming aligns with how Phoenix exports evaluation scores today.

[sauravGit]

RFC v0.3 update — three key alignment fixes based on OTel upstream research:

1. Reframed as normalization/migration layer on top of existing OTel GenAI + OpenInference conventions — not a new parallel standard
2. OpenInference / Arize Phoenix now explicitly in the fragmentation table with correct mappings: llm.latency → gen_ai.client.operation.duration, llm.token_count.prompt → gen_ai.usage.input_tokens, llm.token_count.completion → gen_ai.usage.output_tokens, llm.cost.total → gen_ai.usage.cost (extension pack). Also added open question feat(2): react inside of notebook #5: should OpenInference migration mappings be normative (MUST) or informative (SHOULD)?
3. gen_ai.usage.cost moved to optional Cost extension pack — not Core REQUIRED

Formal upstream proposal is open at semantic-conventions-genai #101.

Full RFC v0.3: https://github.com/sauravGit/open-llm-observability/blob/main/RFC.md

Would still love Phoenix team input on whether gen_ai.eval.* naming aligns with how Phoenix exports evaluation scores today, and whether OpenInference mappings should be normative in the spec.

[musaabhasan]

The normalization-layer framing is the right direction. I would be careful to separate raw telemetry from derived metrics in the convention.

For production debugging, the schema needs enough structure to reconstruct the path of a response, not only summarize it. I would expect separate span/event concepts for:

• request/generation: model, provider, prompt/template version, parameters, latency, token usage, cost
• retrieval: index name, retriever version, query rewrite version, returned source IDs, scores, filters, top_k
• tool call: tool name, argument schema version, policy decision, approval state, downstream status
• evaluation: evaluator name/version, dataset version, label source, score, threshold, pass/fail reason

That makes trace_id and span_id more important than a flat metric name. For example, hallucination or faithfulness scores only become actionable when linked back to the retrieved sources and the exact prompt/model version that produced the answer.

I would also avoid treating error_rate as a first-class emitted metric if possible. Emit counted events with normalized error type and status, then derive rates downstream. It keeps the convention portable across Phoenix, OpenInference, OTel collectors, and vendor dashboards without forcing every SDK to compute windows the same way.

[sauravGit]

Thanks @musaabhasan, this is exactly the kind of Phoenix/OpenInference feedback I was hoping for.

I agree that the convention needs to preserve the response path, not just summarize it. I’ll revise the RFC so the Phoenix/OpenInference mapping separates:

• request/generation spans and related events
• retrieval spans/events
• tool-call spans/events
• evaluation spans/events
• low-cardinality metrics derived from those signals

I also agree on error_rate. A portable convention should probably emit normalized error/status events or counts and let collectors or dashboards derive rates over windows. I’ll change the open question from “Gauge vs counter pair” toward “emit error/status events or counts, derive error rate downstream,” unless the SIG has a strong preference otherwise.

For evals, I’ll make trace_id and span_id linkage explicit so scores like faithfulness, hallucination, and relevance can be traced back to retrieved sources, prompt/model version, and evaluator version.

[sauravGit]

Thanks also @musaabhasan. I agree the mapping should include more than field names. I’ll update the RFC mapping tables to include:

• source field
• canonical field
• unit conversion
• expected cardinality
• signal type: span attribute, metric, event/log, or attachment
• lossiness notes
• whether the field belongs in core or an optional extension pack

I’ll also keep evals first-class, linked by trace/span IDs rather than forcing all quality data into flat metrics.