You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR fixes a bug where audio IDs were being incorrectly concatenated when streaming audio chunks from OpenAI's gpt-4o-audio-preview model, causing "Assistant audio not found" errors.
Problem
When streaming audio chunks, the _mergeDicts function in langchain-core/src/messages/base.ts was concatenating audio IDs, transforming:
This resulted in invalid audio IDs that caused subsequent API requests to fail.
Solution
Added a targeted fix to preserve the first non-empty value for audio ID fields instead of concatenating them:
// Special case: preserve the first non-empty value for audio.id field// This prevents audio ID duplication when streaming audio chunksif(key==="id"&&merged[key]&&merged[key].startsWith("audio_")){continue;}
Testing
Added comprehensive unit tests for audio ID preservation
Verified other audio fields still concatenate correctly
Ensured non-audio ID fields maintain existing behavior
All existing message tests pass (51 total)
Future Considerations
While this addresses the immediate audio ID issue, similar fields may need consideration:
When streaming audio chunks from OpenAI's gpt-4o-audio-preview model,
audio IDs were being incorrectly concatenated in the _mergeDicts function,
causing "Assistant audio not found" errors on subsequent requests.
The audio ID "audio_6871323ab2fc8191b2cf516af96cd851" would become
"audio_6871323ab2fc8191b2cf516af96cd851audio_6871323ab2fc8191b2cf516af96cd851"
when merging message chunks.
Changes:
- Add special case in _mergeDicts to preserve first non-empty audio ID
- Only affects string fields with key "id" that start with "audio_"
- Other audio fields (transcript, data) continue to concatenate normally
- Add comprehensive unit tests to prevent regressions
Fixes audio streaming functionality while maintaining backward compatibility
for all other message field concatenation behavior.
Will call out that there is a similar issue when using the image generation tool with openai. Off the top of my head we retain each partial_image object for streaming responses, but the intended behavior is to keep the latest one instead of retaining all of them. Similar semantics here where we only want to retain one audio id.
I'm also not sure how much of a fan I am of baking in the assumption that ids prefixed with audio_ should always be collated this way. This is how OpenAI views the world, but whos to say anthropic and google will in the near future? There's some work to be done in terms of beefing up the base chunk class that can reconcile with these aggregation considerations.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes a bug where audio IDs were being incorrectly concatenated when streaming audio chunks from OpenAI's
gpt-4o-audio-previewmodel, causing "Assistant audio not found" errors.Problem
When streaming audio chunks, the
_mergeDictsfunction inlangchain-core/src/messages/base.tswas concatenating audio IDs, transforming:"audio_6871323ab2fc8191b2cf516af96cd851"→"audio_6871323ab2fc8191b2cf516af96cd851audio_6871323ab2fc8191b2cf516af96cd851"This resulted in invalid audio IDs that caused subsequent API requests to fail.
Solution
Added a targeted fix to preserve the first non-empty value for audio ID fields instead of concatenating them:
Testing
Future Considerations
While this addresses the immediate audio ID issue, similar fields may need consideration:
request_id,response_id,conversation_id)_at,_time,timestamp)_id,uuid,guid)index,position,sequence)status,state,phase)Fixes #8487