feat(provider/openai):Set the annotations from the Responses API to doStream #10253

tsuzaki430 · 2025-11-15T08:43:07Z

Background

In OpenAI, after obtaining a file_id through Code Interpreter, the results included in the text of a TextUIPart depend on the attached annotations.
Because of this, we needed to adjust the implementation so that annotations could be properly attached at the end of the doStream process.

Summary

We have now made it possible for doStream—just like doGenerate—to set Response API annotations into providerMetadata.

By using an ongoingAnnotations variable in doStream, we accumulate the annotation array until processing is completed at the final text-end event.

As a result, in next-openai, useChat can utilize both annotations and Streamdown’s Markdown transformation to produce clickable text content.
This allows us to significantly improve the user experience.

Below is a comparison of plain text vs. markdown rendering (Streamdown).
When the container_file_citation annotation is applied, users can click to download the file directly — improving both functionality and UX.

Manual Verification

In our CI/CD tests, we verified that annotations added via doStream were correctly applied as expected, and updated the implementation accordingly.
packages/openai/src/responses/openai-responses-language-model.test.ts
packages/azure/src/azure-openai-provider.test.ts
After running the Code Interpreter via the streamText function, we confirmed that the file-container_file_citation annotation was successfully received.
examples/ai-core/src/stream-text/openai-responses-code-interpreter.ts
examples/ai-core/src/stream-text/azure-responses-code-interpreter.ts
We updated the Next.js sample so that the result returned by useChat can embed file-download links directly into the text output.
http://localhost:3000/test-openai-code-interpreter

Checklist

Tests have been added / updated (for bug fixes / features)
Documentation has been added / updated (for bug fixes / features)
A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
I have reviewed this pull request (self-review)

Future Work

#10255

Related Issues

#10255

examples/next-openai/components/tool/openai-responses-text.tsx

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>

tsuzaki430 · 2025-11-15T13:21:57Z

I have reviewed this pull request (self-review)

jephal · 2025-11-17T13:28:46Z

@tsuzaki430 can confirm this works well for azure. Tested the script in azure-responses-code-interpreter.ts

Merging this would allow for streamdown rendering of code-interpreter download links for streaming as talked about in the community session (with the start and end index) @gr2m

tsuzaki430 · 2025-11-17T13:40:30Z

@jephal

Thank you for test and show us the result.
Yes , this feature provides us the url link using useChat and streamdown😊

rahulbhadja · 2025-11-18T17:42:15Z

I also reviewed this as well, great work @tsuzaki430

…0370) ## Background Web Search Preview has become available on Microsoft Azure OpenAI. We enabled the use of `web_search_preview` from `@ai-sdk/openai` within `@ai-sdk/azure`. [https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/web-search?view=foundry-classic](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/web-search?view=foundry-classic) ## Summary Using Next.js, it is now possible to perform web searches with both `useChat` and `streamText`. * We enabled Azure’s webSearchPreview tool by routing through the OpenAI internal implementation. * In `doGenerate`, the Zod type definition for `url_citation` was updated to support annotations. * In `doStream`, there were some missing pieces when switching between `web_search` and `web_search_preview`. * add description `web_search_preview` in azure provider web page. ## Manual Verification * Added verification for `web_search_preview` in Azure’s CI/CD tests. `packages/azure/src/__fixtures__/azure-web-search-preview-tool.1.chunks.txt` `packages/azure/src/__fixtures__/azure-web-search-preview-tool.1.json` * Added tests for `generateText` and `streamText` in `examples/ai-core`. `generateText` correctly includes the `url_citation` annotation, while `streamText` currently does not output it. This is expected to be resolved once PR #10253 is merged. `examples/ai-core/src/generate-text/azure-responses-web-search-preview.ts' `examples/ai-core/src/stream-text/azure-responses-web-search-preview.ts' * Added a working Next.js example demonstrating web search in `examples/next-openai`. `http://localhost:3000/test-azure-web-search-preview` `examples/next-openai/app/test-azure-web-search-preview/page.tsx` <img width="908" height="1032" alt="image" src="https://github.com/user-attachments/assets/b51bf4b8-ffb2-4106-9859-a3b017128aae" /> ## Future Work Since `streamText` does not currently output annotations, we need to confirm that this is resolved once PR #10253 is merged. ## Related Issues #10253 --------- Co-authored-by: tsuzaki430 <[email protected]>

gr2m

Thank you Azure team 💙

vercel · 2025-11-20T18:30:12Z

packages/openai/src/responses/openai-responses-language-model.ts

+                  openai: {
+                    itemId: value.item.id,
+                    ...(ongoingAnnotations.length > 0 && {
+                      annotations: ongoingAnnotations,


Suggested change

annotations: ongoingAnnotations,

annotations: [...ongoingAnnotations],

The ongoingAnnotations array is passed by reference (not copied) when creating the text-end event. This causes a bug where if another message arrives and clears the array before the consumer processes the text-end event, the annotations will appear empty.

View Details

Analysis

Array reference causes annotations to be lost in text-end event

What fails: In OpenAIResponsesLanguageModel.doStream(), when multiple messages are streamed, annotations from the first message become empty in the text-end event if a second message arrives before the event is consumed.

How to reproduce:

Stream a request with multiple messages where the first message contains annotations. The second message will arrive and clear the `ongoingAnnotations` array before the first message's `text-end` event is fully processed, causing the array reference to point to an empty array. This occurs in the TransformStream's synchronous transform() method where: 1. Message 1's text-end event is created with direct reference to ongoingAnnotations (line 1474) 2. Message 2 arrives and immediately clears the array with splice() (line 984) 3. The queued text-end event now references an empty array

Result: The text-end event's providerMetadata.openai.annotations field is empty even though annotations were present in the stream.

Expected: Each text-end event should contain the annotations that were present for that specific message, regardless of when subsequent messages arrive.

Fix: Create a copy of the array when passing it to the event using the spread operator: annotations: [...ongoingAnnotations] instead of annotations: ongoingAnnotations (line 1474 of openai-responses-language-model.ts)

vercel-ai-sdk · 2025-11-20T18:31:52Z

⚠️ Backport to release-v5.0 created but has conflicts: #10419

…s API to doStream (#10419) This is an automated backport of #10253 to the release-v5.0 branch. --------- Co-authored-by: tsuzaki430 <[email protected]> Co-authored-by: Gregor Martynus <[email protected]>

lgrammel · 2025-11-25T10:14:34Z

packages/openai/src/responses/openai-responses-language-model.ts

    > = {};

+    // set annotations in 'text-end' part providerMetadata.
+    const ongoingAnnotations: Array<


this is potentially buggy. it needs to be indexed by the text part id to avoid conflicts. this may not be an issue now but could lead to bugs in the future.

lgrammel · 2025-11-25T10:14:53Z

packages/openai/src/responses/openai-responses-language-model.ts

              }
+            } else if (
+              isResponseOutputItemDoneChunk(value) &&
+              value.item.type === 'message'


reverted the if-else split in #10586

lgrammel · 2025-11-25T10:18:20Z

How are file annotations related to the text parts? I feel this PR can potentially be problematic because we set a precedent of adding unrelated information to text parts, and it will become part of the stored history so we have to maintain backwards compat forever. I would have preferred to wait until we properly implement citation support.

tsuzaki430 · 2025-11-25T10:59:58Z

@lgrammel

When I originally wrote the code, text-end was executed before the annotations were captured, so I wasn’t able to retrieve them.
With #10586, I understand that this workaround is no longer necessary.
I apologize for the odd implementation and any inconvenience it may have caused.

## Background #10253 split the if-else statement unnecessarily. ## Summary Revert the if-else split. ## Verification Refactoring. Unit tests passing without changes.

Set the annotations from the Responses API to doStream.

d9607d1

vercel-ai-sdk bot added the ai/provider label Nov 15, 2025

link using annotation on next.js example

90931db

tsuzaki430 marked this pull request as ready for review November 15, 2025 12:25

vercel bot reviewed Nov 15, 2025

View reviewed changes

examples/next-openai/components/tool/openai-responses-text.tsx Outdated Show resolved Hide resolved

Update examples/next-openai/components/tool/openai-responses-text.tsx

6f67b1c

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>

tsuzaki430 mentioned this pull request Nov 15, 2025

provider/openai,provider/azure : Allow clients to download CodeInterpreter output files #10255

Open

1 task

tsuzaki430 mentioned this pull request Nov 20, 2025

feat(provider/azure): Responses API web search preview is enabled #10370

Merged

4 tasks

gr2m approved these changes Nov 20, 2025

View reviewed changes

gr2m added provider/openai provider/azure feature New feature or request backport labels Nov 20, 2025

Merge branch 'main' into tsuz/openai-responses-annotations-dostream

b797419

gr2m enabled auto-merge (squash) November 20, 2025 18:08

gr2m added 2 commits November 20, 2025 10:16

test: update snapshots

f5c76ec

test: update snapshots

1278add

vercel bot reviewed Nov 20, 2025

View reviewed changes

gr2m merged commit 423ba08 into vercel:main Nov 20, 2025
17 of 18 checks passed

vercel-ai-sdk bot pushed a commit that referenced this pull request Nov 20, 2025

Backport conflicts for PR #10253 to release-v5.0

4fbc5d7

vercel-ai-sdk bot mentioned this pull request Nov 20, 2025

Backport: feat(provider/openai):Set the annotations from the Responses API to doStream #10419

Merged

vercel-ai-sdk bot removed the backport label Nov 20, 2025

tsuzaki430 deleted the tsuz/openai-responses-annotations-dostream branch November 21, 2025 12:14

lgrammel mentioned this pull request Nov 25, 2025

refactoring(provider/openai): simplify code #10586

Merged

lgrammel reviewed Nov 25, 2025

View reviewed changes

	annotations: ongoingAnnotations,
	annotations: [...ongoingAnnotations],

feat(provider/openai):Set the annotations from the Responses API to doStream #10253

feat(provider/openai):Set the annotations from the Responses API to doStream #10253

Uh oh!

Conversation

tsuzaki430 commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Summary

Manual Verification

Checklist

Future Work

Related Issues

Uh oh!

Uh oh!

tsuzaki430 commented Nov 15, 2025

Uh oh!

jephal commented Nov 17, 2025

Uh oh!

tsuzaki430 commented Nov 17, 2025

Uh oh!

rahulbhadja commented Nov 18, 2025

Uh oh!

gr2m left a comment

Choose a reason for hiding this comment

Uh oh!

vercel bot Nov 20, 2025

Choose a reason for hiding this comment

Analysis

Array reference causes annotations to be lost in text-end event

Uh oh!

Uh oh!

vercel-ai-sdk bot commented Nov 20, 2025

Uh oh!

lgrammel Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

lgrammel Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

lgrammel commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tsuzaki430 commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tsuzaki430 commented Nov 15, 2025 •

edited

Loading

lgrammel commented Nov 25, 2025 •

edited

Loading