Add support for OpenAI Connectors and MCP Registries

[jaideepr97]

🚀 Describe the new functionality needed

Related discussion: #3926

Overview

This document proposes a multi-phase evolution of how the Llama Stack instance manages and exposes MCP connectors to agents and models via the responses API

• Phase 1 introduces a static configuration model, where connectors are manually defined.
• Phase 2 introduces a dynamic registry-based model, enabling discovery of connectors from one or more MCP registries while preserving backward compatibility with static entries.
• Phase 3 introduces a dedicated API for CRUD operations on connectors as well as MCP Registries

Client side usage

Client side usage would be similar to how it is documented in https://platform.openai.com/docs/guides/tools-connectors-mcp#quickstart

from openai import OpenAI
# Initialize OpenAI client
client = OpenAI(base_url="http://localhost:8321/v1", api_key="fake")

resp = client.responses.create(
    model="Qwen/Qwen3-32B",
    tools=[
        {
            "type": "mcp",
            "server_label": "Dropbox",
            "connector_id": <connector_id>,
            "authorization": "<oauth access token>",
            "require_approval": "never",
        },
    ],
    input="Summarize the Q2 earnings report.",
)

print(resp.output_text)

Static Connector Configuration

In this mode, the Llama Stack instance administrator must manually define the allowed MCP connectors in the run.yaml
EDIT: this section has been split into its own issue #4186

Example config

apis: ...
providers:
  agents:
  - provider_id: meta-reference
    provider_type: inline::meta-reference
    config:
      persistence:
        agent_state:
          namespace: agents
          backend: kv_default
        responses:
          table_name: responses
          backend: sql_default
          max_write_queue_size: 10000
          num_writers: 4
      connectors:
        - connector_id: connector_github
          url: https://api.github.com/mcp
        - connector_id: connector_gitlab
          url: https://gitlab.com/mcp
        - connector_id: connector_slack
          url: https://slack.com/mcp
...

Operational notes

• Any new connector addition requires updating run.yaml and restarting the server.
• The server loads these connectors into memory at startup and exposes them to agents.
• This configuration model is suitable for small or tightly controlled deployments where server persistence is not required
• Connector names must be exposed to users via off band (like documentation) means if they are unable to view the server configuration

Dynamic Registry Integration

In dynamic mode, the administrator may configure one or more MCP registries.
The server will query these registries at runtime to discover and construct connector strings automatically.
Each registry must follow the official Anthropic MCP Registry API definition as described at:
Official MCP Registry Reference

Example configuration

apis: ...
providers:
  agents:
  - provider_id: meta-reference
    provider_type: inline::meta-reference
    config:
      persistence:
        agent_state:
          namespace: agents
          backend: kv_default
        responses:
          table_name: responses
          backend: sql_default
          max_write_queue_size: 10000
          num_writers: 4
      connectors:
      - name: connector_github
        url: https://api.github.com/mcp
      mcp_registries:
      - name: registry_internal
        id: internal
        url: https://registry.internal.dev/
      - name: registry_public
        id: public
        url: https://registry.public.dev/
...

Each dynamically discovered server produces a unique connector_id in the form mcp::<registry_id>::<server_name>
This convention ensures uniqueness across registries and provides users with a stable, self-describing identifier. Administrators must ensure all registry_id values are unique within the run.yaml.

Operational notes

• Any new registry addition requires updating run.yaml and restarting the server.
• Users should be made aware of this pattern (via documentation) so they can construct the connector string for themselves based on how the registries are configured and the server information in the registry.

Connector/Registry Management API

Allow runtime inspection and modification (CRUD) of connectors and registries via a new API + provider (Full spec to be fleshed out, but would retain MCP registries as a first class concept and be built around that)

• Allow registration/removal/update of connectors without server restart
• Allow registration/removal/update of MCP registries without server restart
• Allow listing of constructed connector_ids

💡 Why is this needed? What if we don't build it?

Background and motivation

Currently llama stack’s OpenAI Responses implementation does not cover connectors. This requires users to be aware of each remote MCP server’s URLs to be able to use them through their queries. There are a number of benefits to being able to abstract away that piece of information from the users, such as not having to worry about updating applications when a server URL/transport protocol changes and being able to separate the user plane from the platform where the server itself maybe running (potentially useful in enterprise settings)

This feature would allow llama stack to cater towards an MCP as a service paradigm that is likely around the corner. If we don't build this feature, llama stack could be less favorably positioned in the market when such use cases become more and more common. Anthropic's MCP Registry API is well positioned to become the industry standard, llama stack should give its users the ability to integrate with it.

Other thoughts

Alternative approaches:

Lead with API design if community need for it is more urgent

Issue tracking

Based on discussions, this work has been broken down into the following sub-issues for tracking purposes:

Static connector configuration

• [] Introduce connector <-> MCP URL mapping #4186
  • feat(api): Implement connector support via static configuration  #4263

Dynamic registry integration

• [] Allow configuration of MCP registries for dynamic connector support #4238

Connector Registry API Management

• Deprecate toolgroups and tool_runtime APIs #4233
  • feat(api)!: deprecate toolgroup and tool_runtime apis #4249
• [] Introduce a new API for connector discovery #4235
  • feat(api): add readonly connectors API #4258
• [] Add /admin level Create/Update/Delete endpoints for the connectors API #4239
  • blocked by Hardening Stack registry & admin APIs for multi-tenant SaaS #4163

[jwm4]

I think all three of these "phases" meet a distinct user need, so I am inclined to see these more as complementary capabilities rather than steps toward a final goal. With that said, the Static Connector Configuration seems a lot simpler to implement and to use than the other two, so I think that there is a good case for doing that one first. The other two could come later to meet the needs of more advanced users.

[jaideepr97]

I think all three of these "phases" meet a distinct user need, so I am inclined to see these more as complementary capabilities rather than steps toward a final goal. With that said, the Static Connector Configuration seems a lot simpler to implement and to use than the other two, so I think that there is a good case for doing that one first. The other two could come later to meet the needs of more advanced users.

makes sense, the phases are just a way to break the work down into logical units, they can be implemented in any order or all together for that matter, as long as there is a will to go down this direction

[ashwinb]

I will look into the details soon and comment. One quick question: what would be the use case for CRUD mutations?

[jaideepr97]

I will look into the details soon and comment. One quick question: what would be the use case for CRUD mutations?

Thanks for looking @ashwinb!
The CRUD abilities become important in multi-tenant environments, where a single llama stack instance might be shared between applications across multiple namespaces
Each tenant might need to register their own set of registries/connectors that will require on the fly validation/authorization against those registries.

There would also be cases where registry instances are created and updated by a separate team, and llama stack will need to be watching them. The CRUD API would allow for automated pipelines to register these new registry instances/connectors automatically with llama stack whenever they are created or updated.

The CRUD API would also allow llama stack to support short term clients who need to register new connectors/registries, run some requests and then un-register those connectors/registries.

The API would also allow users to query connector_ids directly from llama stack rather than having to go through documentation

[mattf]

I think all three of these "phases" meet a distinct user need, so I am inclined to see these more as complementary capabilities rather than steps toward a final goal. With that said, the Static Connector Configuration seems a lot simpler to implement and to use than the other two, so I think that there is a good case for doing that one first. The other two could come later to meet the needs of more advanced users.

makes sense, the phases are just a way to break the work down into logical units, they can be implemented in any order or all together for that matter, as long as there is a will to go down this direction

@jaideepr97 this is all good stuff. please separate out the request for a connector_id -> url map. user can readily use that shorthand. it also allows the deployer to create a stable name for what may be a shifting url, or, like in the opanai case, front a custom mcp impl for an important data source that doesn't have an mcp url.

[ashwinb]

@jaideepr97 This is a very useful primitive to add to the Stack, thank you for the proposal. Some thoughts on this:

• We do have a notion of toolgroups today in the Stack and we allow registering a "MCP" toolgroup today. Those can then be used in the tool-runtime APIs by specifying a toolgroup_id. This is very similar in spirit and functionality but of course does not contain the dynamic aspect of pulling from remote registries.

• Of course toolgroups (as they exist today in the Stack) are more complex given the history of other built-in tools we supported earlier and as such they haven't seen much usage or iteration (we think). The term is also kind of opaque and off-putting somewhat.

• I think we should kill the non-MCP tools portion of toolgroups and then rebrand toolgroups as connectors. Would you be open to that as the first step of this process? This will also provide you with the CRUD you want for connectors. Is this reasonable?

• As noted in Allow provider connections to be dynamically managed #3809, the CRUD mutations will be moved off to the /admin set of APIs also, but that is slightly orthogonal.

[jaideepr97]

I think all three of these "phases" meet a distinct user need, so I am inclined to see these more as complementary capabilities rather than steps toward a final goal. With that said, the Static Connector Configuration seems a lot simpler to implement and to use than the other two, so I think that there is a good case for doing that one first. The other two could come later to meet the needs of more advanced users.

makes sense, the phases are just a way to break the work down into logical units, they can be implemented in any order or all together for that matter, as long as there is a will to go down this direction

@jaideepr97 this is all good stuff. please separate out the request for a connector_id -> url map. user can readily use that shorthand. it also allows the deployer to create a stable name for what may be a shifting url, or, like in the opanai case, front a custom mcp impl for an important data source that doesn't have an mcp url.

@mattf created #4186

[jaideepr97]

@jaideepr97 This is a very useful primitive to add to the Stack, thank you for the proposal. Some thoughts on this:

• We do have a notion of toolgroups today in the Stack and we allow registering a "MCP" toolgroup today. Those can then be used in the tool-runtime APIs by specifying a toolgroup_id. This is very similar in spirit and functionality but of course does not contain the dynamic aspect of pulling from remote registries.
• Of course toolgroups (as they exist today in the Stack) are more complex given the history of other built-in tools we supported earlier and as such they haven't seen much usage or iteration (we think). The term is also kind of opaque and off-putting somewhat.
• I think we should kill the non-MCP tools portion of toolgroups and then rebrand toolgroups as connectors. Would you be open to that as the first step of this process? This will also provide you with the CRUD you want for connectors. Is this reasonable?
• As noted in Allow provider connections to be dynamically managed #3809, the CRUD mutations will be moved off to the /admin set of APIs also, but that is slightly orthogonal.

thanks for the suggestion @ashwinb !
I think that seems reasonable as a first step if we can add fields for registry URLs and ids down the line to the API, and the /get endpoint is able to return all the computed connectors as described.
I'm curious about what you have in mind for the tool_runtime API post this rebrand? Seems like a fair bit of overlap with the responses api once we get rid of non-MCP tools IIUC

[ashwinb]

@jaideepr97 what would you propose there? There are two methods there:

• list all tools
• invoke tool

Why would one use them from the client? This starts feeling like a MCP server interface itself to me. I'd therefore recast those APIs into a "expose llama stack itself as a wrapped MCP". This thought has been expressed by various folks over time also.

[jaideepr97]

@jaideepr97 what would you propose there? There are two methods there:

• list all tools
• invoke tool

Why would one use them from the client? This starts feeling like a MCP server interface itself to me. I'd therefore recast those APIs into a "expose llama stack itself as a wrapped MCP". This thought has been expressed by various folks over time also.

Yeah I agree that api would end up just being a way to query info from the mcp servers directly if desired. Is there a strong enough use case for something like this? I'm not sure. Maybe for listing tools so that folks could have more informed tool_choice constraints etc if they know exactly what the server offers. But we should probably not allow invocation of MCP tools directly from here

All this to say IMO after the rebrand the tool_runtime API becomes less useful for most use cases and could probably be deprioritized/deprecated unless we really want to pursue the MCP wrapper angle you outlined

The documentation suggests that this API also currently supports some RAG operations, could those be handled elsewhere?

[ashwinb]

I agree. I am on board to fully deprecate the tool_runtime API.