[Platform] Realtime / client-to-server flow

[valtzu]

At least Gemini & OpenAI have support for client-to-server flow where the client/browser connects directly to the platform via WebSocket.
This solution scales much better than streaming with php and it also provides smaller latency, which is especially important with voice chat.

In such flow, the server fetches an ephemeral access token (which by default can only be used once) using the static api key and provides that to the client, then the client uses that to establish the websocket connection.

When fetching the ephemeral access token, you can also provide many other settings, including which model to use, tools, system prompt etc. In this flow you'd need to expose an endpoint to call tools and then make the frontend call the endpoint when receiving a toolcall message from the websocket connection.

---

An example request to get ephemeral access token from Gemini Live API (there are no official docs for this)

POST https://generativelanguage.googleapis.com/v1alpha/auth_tokens?key=xx

{
    "uses": 1,
    "expireTime": "2025-08-02T00:00:00Z",
    "bidiGenerateContentSetup": {
        "model": "models/gemini-2.0-flash-live-001",
        "generationConfig": {
            "temperature": 0.7,
            "responseModalities": [
                "TEXT"
            ]
        }
    }
}

For OpenAI: https://platform.openai.com/docs/api-reference/realtime-sessions

---

Do you see a place for this within symfony/ai?

[chr-hertel]

Yes, some webrtc realtime stuff is def also on my list - are you working on that already?

[valtzu]

I don't have any ideas on how to organize the code, thus, not planning to work on this in near future. But I'll play with Gemini Live API to see if there are any gotchas.

[valtzu]

For what it's worth, I did manage to do a PoC with Gemini Live API with realtime bidirectional audio & the official @google/genai js sdk by following several examples online. It requires a lot of javascript! (even without audio visualization). The audio is choppy sometimes & cuts off abruptly, not sure if it's due to a bug in the code or due to using deprecated script processor node for audio which runs on the main thread.

Here's the code: aea1e6d

The same GEMINI_API_KEY works with live api

---

Note that this demo uses literally nothing from symfony/ai and therefore is not a candidate for a demo as-is – more just a use-case demonstration to discuss about the possible implementation in symfony/ai.

[chr-hertel]

Nice one, I won't make it in the next days but those two things would be my starting point in understanding what we could do:

• https://github.com/openai/openai-realtime-agents
• https://fastrtc.org/cookbook/

But yes, will be quite some javascript involved