Fix AttributeError in streaming response cleanup

[r-bit-rry]

This PR fixes issue #3185
The code calls await event_gen.aclose() but OpenAI's AsyncStream doesn't have an aclose() method - it has close() (which is async).
when clients cancel streaming requests, the server tries to clean up with:

await event_gen.aclose()  # ❌ AsyncStream doesn't have aclose()!

But AsyncStream has never had a public aclose() method. The error message literally tells us:

AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
                                                                            ^^^^^^^^

Verification

• Reproduction script reproduce_issue_3185.sh can be used to verify the fix.
• Manual checks, validation against original OpenAI library code

[mattf]

@r-bit-rry a chain of hasattr like this suggests we've done something wrong in the design. have we or can we just call close?

[r-bit-rry]

@r-bit-rry a chain of hasattr like this suggests we've done something wrong in the design. have we or can we just call close?

It really comes down to what we want to support, since this was never strictly typed, I'm assuming there are other objects that can be generated by the sse_generator.
I'd go even a step further to ask why do we even assume close method exists?

and on a more serious note @mattf
PEP 525 defines aclose() for async generators
OpenAI and Anthropic SDKs deviate from the standard and use close() instead (while underneath the hood they both call httpx.Response.aclose() )
We cannot control third-party SDK design choices

so its our decision if we want to enforce certain typings, and act upon, or let this pattern "catch all"

[mattf]

as propose, the hasattr chain will cover up an api contract bug somewhere in the system.

an AsyncStream is making it to a place where only AsyncIterators should be.

i did a little sleuthing and i think there is a bug in at least _maybe_overwrite_id. there are multiple provider impls, so there may be others.

will you find the places where the api contract is being violated and patch them?

also, will you create a regression test that at least tests the openai mixin provider?

[r-bit-rry]

@mattf sure thing, I'll start working on those

[r-bit-rry]

@mattf We're facing two options in order to avoid hasattr chain as I see it when treating the AsyncStream object:
option 1: wrapping

async def wrap_async_stream(stream): 
    async for item in stream: yield item

this is explicit and simple but carries a small overhead per chunk

option 2: adapter pattern

class AsyncStreamAdapter(AsyncIterator[T]):
    def __init__(self, stream): self._stream = stream
    async def aclose(self): await self._stream.close()  # Delegate close→aclose

direct delegation with no re-yielding and a more explicit intent

regarding locations of violations, where we will need patching, these are the places I was able to spot:

1. _maybe_overwrite_id(): Location: src/llama_stack/providers/utils/inference/openai_mixin.py:251
   when overwrite_completion_id=False and stream=True

returned AsyncStream (has close()) instead of AsyncIterator (has aclose())

2. PassthroughInferenceAdapter - openai_chat_completion()
   Location: src/llama_stack/providers/remote/inference/passthrough/passthrough.py:123-136

Returned raw client.chat.completions.create() response

3. PassthroughInferenceAdapter - openai_completion()
   Location: src/llama_stack/providers/remote/inference/passthrough/passthrough.py:108-121

Returned raw client.completions.create() response

4. LiteLLMOpenAIMixin - openai_chat_completion()
   Location: src/llama_stack/providers/utils/inference/litellm_openai_mixin.py:222-278

Returned raw litellm.acompletion() result

5. LiteLLMOpenAIMixin - openai_completion()
   Location: src/llama_stack/providers/utils/inference/litellm_openai_mixin.py:179-220
   Returned raw litellm.atext_completion() result

[mattf]

@r-bit-rry great finds! it looks like we're violating the api contract and using # ignore to cover it up...resulting in the bug. what fix do you suggest?

[r-bit-rry]

@mattf I suggest using the first option, with direct declaration of the types, and changing the API contract for openai_completion, removing the #type: ignore comments.
since I'm already touching all the places where the return value of the method is handled, I think fixing the contract there is the most responsible way of handling this.

[mattf]

@r-bit-rry sounds good!

[r-bit-rry]

@mattf as I'm digging, I'm finding new stuff, apparently we have a discrepancy between official and internal documentation of openai_completion and the implementation in place:
Request schema from inference.py:871:
class OpenAICompletionRequestWithExtraBody(BaseModel, extra="allow"):
# ...
stream: bool | None = None # ← Streaming IS a parameter!

however the documentation both on our end and on openai official end does not support streaming to begin with (docs/docs/contributing/new_api_provider.mdx:40-43:

• openai_completion(): Legacy text completion API with full parameter support
• openai_chat_completion(): Chat completion API supporting streaming, tools, and function calling

OpenAI documentation:
Schema: CreateCompletionResponse
"Represents a completion response from the API. Note: both the streamed and non-streamed response objects share the same shape (unlike the chat endpoint)."
so only our documentation is lacking. I've taken this into the changes I've comitted

[mattf]

can we get rid of the type ignores now?

[mattf]

since the async iterator business is an impl detail, what about mentioning SSE events instead?

[r-bit-rry]

1. I'm not sure I can dismiss async iterator as impl detail, as we decided that the contract has changed, its an expected type to be returned, I don't mind for documentation/comment purpose considering SSE events even though those are not typed.
2. Remaining # type: ignore comments are only for external library issues:
   # type: ignore[arg-type] - LiteLLM streaming types don't match OpenAI
   # type: ignore[return-value] - External libs lack type stubs

[mattf]

@r-bit-rry this text gets published in the OpenAPI spec, which is language / implementation agnostic.

[mattf]

can we get rid of the type ignores now?

Remaining type: ignore comments are only for external library issues:
type: ignore[arg-type] - LiteLLM streaming types don't match OpenAI
type: ignore[return-value] - External libs lack type stubs

what about passthrough?