chore(llmobs): [MLOB-3828] link input and output guardrail spans for openai agents #14488
Conversation
|
|
ncybul
added
the
changelog/no-changelog
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 269 ± 3 ms. The average import time from base is: 272 ± 2 ms. The import time difference between this PR and base is: -2.6 ± 0.1 ms. Import time breakdownThe following import paths have grown:
|
ncybul
changed the title
ncybul
changed the title
Performance SLOsCandidate: nicole-cybul/link-guardrail-spans (034d897) 🔵 No Baseline Data (24 suites)🔵 coreapiscenario - 12/12 (2 unstable)🔵 No baseline data available for this suite
|
| @@ -219,8 +221,6 @@ def __init__( | |||
| self._annotations: List[Tuple[str, str, Dict[str, Any]]] = [] | |||
There was a problem hiding this comment.
Do we need to remove self._link_tracker too? https://github.com/DataDog/dd-trace-py/pull/14488/files#diff-3ee322478554cf5c3393bc0bcbbff273887d06dc2e7c568341747de44c9b80e7L218
There was a problem hiding this comment.
Whoops, I already merged this but no. I repurposed LinkTracker to contain the logic for span linking both LLMs / Tools and LLMs / Guardrails.
…openai agents (#14488) This PR implements span linking for Open AI Agents guardrail spans. For input guardrails, a span link is created from the guardrail span to the first LLM span which is a sibling of the input guardrail (share the same llmobs parent). This implementation prevents a link from the guardrail span to the openai_agents.response span from being created while still linking the guardrail to the Triage Agent (LLM) span. <img width="708" height="264" alt="image" src="https://github.com/user-attachments/assets/35df597a-9d27-464e-b302-9b241e2b4f4d" /> For output guardrails, a span link is created from the last finished LLM span (for a given agent run) to the output guardrail span. <img width="2766" height="178" alt="image" src="https://github.com/user-attachments/assets/35773ad9-d512-4be5-8db9-8a559ab29047" /> With this implementation, we get a trace like the following which better represents the guardrail's functionality of checking the user input to the first agent and the final output of the last agent. <img width="488" height="1296" alt="image" src="https://github.com/user-attachments/assets/4aac3564-d9bb-491e-9155-5847cec1f8ce" /> This PR also removes the experimental `_record_object` method from the LLMObs service since it is not being used anywhere. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
This PR implements span linking for Open AI Agents guardrail spans.
For input guardrails, a span link is created from the guardrail span to the first LLM span which is a sibling of the input guardrail (share the same llmobs parent). This implementation prevents a link from the guardrail span to the openai_agents.response span from being created while still linking the guardrail to the Triage Agent (LLM) span.
For output guardrails, a span link is created from the last finished LLM span (for a given agent run) to the output guardrail span.
With this implementation, we get a trace like the following which better represents the guardrail's functionality of checking the user input to the first agent and the final output of the last agent.
This PR also removes the experimental
_record_objectmethod from the LLMObs service since it is not being used anywhere.Checklist
Reviewer Checklist