feat(azure_ai): add Responses API support (incl. Azure Government)#33393
feat(azure_ai): add Responses API support (incl. Azure Government)#33393grahamprimm wants to merge 1 commit into
Conversation
|
|
Greptile SummaryAdds
Confidence Score: 3/5The core routing for the primary The tool-flattening transform that litellm/llms/azure_ai/responses/transformation.py — specifically whether
|
| Filename | Overview |
|---|---|
| litellm/llms/azure_ai/responses/transformation.py | New AzureAIResponsesAPIConfig: URL routing and auth logic work for the primary services.ai.azure.com use case, but the _is_azure_foundry_host helper is broader than the existing chat helper, causing auth inconsistency for cognitiveservices.azure.com, a wrong path for openai.azure.com, a query-string false-positive in the /projects/ check, and a likely missing tool-flattening transform. |
| litellm/utils.py | Adds a single elif branch to route AZURE_AI provider to the new AzureAIResponsesAPIConfig; minimal and consistent with existing provider dispatch pattern. |
| litellm/init.py | Adds TYPE_CHECKING import for AzureAIResponsesAPIConfig, mirroring the existing pattern for AzureOpenAIResponsesAPIConfig. |
| litellm/_lazy_imports_registry.py | Registers AzureAIResponsesAPIConfig in the lazy-import name list and module map, following the existing convention exactly. |
| tests/test_litellm/llms/azure_ai/responses/test_azure_ai_responses.py | 24 unit tests covering provider registration, URL construction for commercial and government hosts, project-path precedence, api-version propagation, auth-header selection, and missing api_base. All purely unit/mock — no network calls. Tests do not cover tool-payload transformation or the cognitiveservices.azure.com auth discrepancy. |
Reviews (1): Last reviewed commit: "feat(azure_ai): add Responses API suppor..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| _AZURE_FOUNDRY_HOST_SUFFIXES = ( | ||
| # Azure commercial | ||
| ".services.ai.azure.com", | ||
| ".openai.azure.com", | ||
| ".cognitiveservices.azure.com", | ||
| # Azure US Government | ||
| ".services.ai.azure.us", | ||
| ".openai.azure.us", | ||
| ".cognitiveservices.azure.us", | ||
| ) | ||
|
|
||
|
|
||
| def _is_azure_foundry_host(api_base: Optional[str]) -> bool: | ||
| """ | ||
| Return True when ``api_base`` points at an Azure Foundry / Azure OpenAI | ||
| resource (commercial or government), which expect the ``api-key`` header. | ||
| """ | ||
| if not api_base: | ||
| return False | ||
| host = urlparse(api_base).hostname | ||
| return bool(host) and host.endswith(_AZURE_FOUNDRY_HOST_SUFFIXES) |
There was a problem hiding this comment.
_should_use_api_key_header
The existing AzureAIStudioConfig._should_use_api_key_header (used by the chat transformation) only matches .services.ai.azure.com and .openai.azure.com; it does NOT match .cognitiveservices.azure.com. The new _is_azure_foundry_host adds .cognitiveservices.azure.com (and its .azure.us counterpart), so for that host a chat request gets Authorization: Bearer <key> while a responses request gets api-key: <key>. If a user supplies a cognitiveservices.azure.com endpoint with the azure_ai provider today, the two code paths already disagree on which auth header to use, creating confusing debugging experience.
| # project URLs are also on Azure Foundry hosts. Checking the host first | ||
| # would send project endpoints to /models/responses instead of | ||
| # /openai/v1/responses. | ||
| if "/projects/" in api_base: |
There was a problem hiding this comment.
/projects/ check matches query-string parameters
"/projects/" in api_base runs against the full URL string, including any query-string portion. If api_base is, for example, https://res.services.ai.azure.com?context=/projects/foo, the check triggers and the URL is built as .../openai/v1/responses instead of the intended /models/responses. Checking against only the path component (e.g., via urlparse(api_base).path) avoids the false positive.
| elif _is_azure_foundry_host(api_base): | ||
| new_url = _add_path_to_api_base( | ||
| api_base=api_base, ending_path="/models/responses" | ||
| ) | ||
| else: | ||
| new_url = _add_path_to_api_base( | ||
| api_base=api_base, ending_path="/v1/responses" |
There was a problem hiding this comment.
openai.azure.com routed to /models/responses instead of /openai/responses
_is_azure_foundry_host matches .openai.azure.com, so a caller who passes an openai.azure.com endpoint with the azure_ai provider is sent to <base>/models/responses. That path is a Foundry Serverless API path. The correct path for standard Azure OpenAI resources (openai.azure.com) is /openai/responses (as used by the azure provider). Existing azure_ai chat behavior is also inconsistent here: it sends openai.azure.com to /chat/completions (not /models/chat/completions). While mixing azure_ai + openai.azure.com is arguably a misconfiguration, including openai.azure.com in _AZURE_FOUNDRY_HOST_SUFFIXES silently routes such users to an almost certainly incorrect responses endpoint.
| class AzureAIResponsesAPIConfig(OpenAIResponsesAPIConfig): | ||
| """ | ||
| Configuration for the Azure AI Foundry Responses API. | ||
|
|
||
| Extends the OpenAI Responses config because Azure AI Foundry endpoints follow | ||
| the OpenAI `/responses` spec on the wire. The differences handled here are: | ||
|
|
||
| - Auth: the ``api-key`` header for Azure Foundry / Azure OpenAI hosts | ||
| (commercial ``*.azure.com`` and government ``*.azure.us``), falling back to | ||
| ``Authorization: Bearer`` and then to Azure AD token auth. | ||
| - URL construction: the correct `/responses` path per endpoint shape, including | ||
| Azure Government hosts. | ||
| """ | ||
|
|
||
| @property | ||
| def custom_llm_provider(self) -> LlmProviders: | ||
| return LlmProviders.AZURE_AI | ||
|
|
||
| def validate_environment( | ||
| self, | ||
| headers: dict, | ||
| model: str, | ||
| litellm_params: Optional[GenericLiteLLMParams], | ||
| ) -> dict: | ||
| litellm_params = litellm_params or GenericLiteLLMParams() | ||
| api_key = AzureFoundryModelInfo.get_api_key(api_key=litellm_params.api_key) | ||
| api_base = AzureFoundryModelInfo.get_api_base(api_base=litellm_params.api_base) | ||
|
|
||
| if api_key: | ||
| if api_base and _is_azure_foundry_host(api_base): | ||
| headers["api-key"] = api_key | ||
| else: | ||
| headers["Authorization"] = f"Bearer {api_key}" | ||
| else: | ||
| # Fall back to Azure AD token-based auth (entra id / managed identity). | ||
| headers = BaseAzureLLM._base_validate_azure_environment( | ||
| headers=headers, litellm_params=litellm_params | ||
| ) | ||
|
|
||
| headers.setdefault("Content-Type", "application/json") | ||
| return headers |
There was a problem hiding this comment.
AzureOpenAIResponsesAPIConfig.transform_responses_api_request overrides the base to flatten tool objects (moving nested function.* fields to the top level) because Azure's Responses API rejects the OpenAI-standard {type, function: {name, description, parameters}} shape. If Azure AI Foundry's /responses endpoint has the same requirement (likely, given it follows the same spec), not overriding transform_responses_api_request here means function-tool calls will arrive with the nested function key and be rejected. Consider confirming the expected wire format and, if needed, inheriting the flattening logic from AzureOpenAIResponsesAPIConfig or duplicating it here.
grahamprimm
force-pushed
the
feat/azure_ai-responses-api
branch
from
8da9de8 to
faea11c
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The `azure_ai` (Azure AI Foundry) provider has no Responses API config, so
`litellm.responses(...)` — and `/chat/completions` requests bridged to the
Responses API via `model_info: {mode: responses}` — never reach a real upstream
`/responses` endpoint. They fall back to the chat-completions bridge instead.
This matters for newer Azure AI Foundry model deployments that only accept
function tools together with reasoning on the Responses API and reject the
combination on `/chat/completions` (the backend returns "Please use
/v1/responses instead"). Without an `azure_ai` Responses config there is no
way to reach that endpoint through the provider.
Add `AzureAIResponsesAPIConfig` (extends `OpenAIResponsesAPIConfig`):
- URL construction for project-based endpoints (`/openai/v1/responses`),
Azure Foundry / Azure OpenAI hosts (`/models/responses`), and generic
OpenAI-compatible bases (`/v1/responses`); the `/projects/` branch is checked
before the host branch since project URLs are also on Foundry hosts.
- Azure auth: `api-key` header for Foundry/OpenAI hosts, `Authorization: Bearer`
otherwise, falling back to Azure AD token auth.
- **Azure Government support**: recognizes `.azure.us` hosts
(`services.ai.azure.us`, `openai.azure.us`, `cognitiveservices.azure.us`)
alongside their commercial `.azure.com` equivalents, sharing one code path.
- `api-version` query-param propagation.
Register the config in `ProviderConfigManager`, `__init__.py`, and the lazy
imports registry (mirroring the existing `azure` Responses wiring).
Builds on the approach of the earlier (unmerged) PR BerriAI#25462 and issue BerriAI#25407,
extended with Azure Government domains and a unified host-matching helper.
Adds 24 unit tests covering URL construction and auth-header selection for
commercial and government hosts, project vs. non-project endpoints, the
project-before-host precedence, api-version propagation, and the missing-
api_base error.
grahamprimm
force-pushed
the
feat/azure_ai-responses-api
branch
from
faea11c to
fa7f53a
Compare
|
Thanks for the review — addressed the substantive points in the latest commit: Tool-flattening (the important one). You're right that Azure's Responses API needs function tools flattened (params at the top level, not nested under
Host-suffix breadth. The intent is to deliberately cover Foundry/Azure OpenAI hosts across clouds — including All 26 unit tests pass; ruff format/check clean. |
|
Heads-up on the one red check ( Both jobs run This PR touches only Everything owned by this PR is green: |
2 participants
Title
feat(azure_ai): add Responses API support (incl. Azure Government)
Summary
Adds a Responses API config for the
azure_ai(Azure AI Foundry) provider, with Azure Government (.azure.us) support.Today
azure_aihas no Responses API implementation. As a result,litellm.responses(...)— and/chat/completionsrequests bridged to the Responses API viamodel_info: {mode: responses}— never reach a real upstream/responsesendpoint forazure_ai; they fall back to the chat-completions bridge instead.This revives the approach of the earlier, unmerged #25462 (fixes #25407) and extends it with Azure Government domain support and a unified host-matching helper.
Why this matters (Responses API context)
Newer Azure AI Foundry model deployments reject function tools + reasoning in the same request on
/chat/completions, returning an error that explicitly says to use/v1/responsesinstead:This isn't arbitrary. Reasoning models emit hidden reasoning state that must be carried across each tool-call round-trip within a turn. The Responses API represents this as typed reasoning items (with ids / encrypted content) that are replayed alongside the paired
function_call, so the model resumes its prior chain-of-thought after a tool result. Chat Completions has no schema slot to carry that state between the tool call and the tool result, so these deployments refuse the combination there.Because
azure_aihas no Responses config, there is currently no way to reach the endpoint that does accept reasoning + tools for these models through the provider — even settingmodel_info: {mode: responses}just falls back to the chat bridge. This PR closes that gap.What's added
AzureAIResponsesAPIConfig(extendsOpenAIResponsesAPIConfig):api_basecontains/projects/) →/openai/v1/responses/models/responses/v1/responses/projects/branch is checked before the host branch, since project URLs are also on Foundry hosts.api-keyheader for Foundry / Azure OpenAI hosts,Authorization: Bearerotherwise, falling back to Azure AD token auth..azure.ushosts (services.ai.azure.us,openai.azure.us,cognitiveservices.azure.us) alongside their commercial.azure.comequivalents, via a single shared host-suffix helper so both clouds take the same code path.api-versionquery-param propagation.Registered in
ProviderConfigManager._get_python_responses_api_config,litellm/__init__.py, and the lazy imports registry, mirroring the existingazureResponses wiring.Example
Testing
Adds
tests/test_litellm/llms/azure_ai/responses/test_azure_ai_responses.py— 24 tests covering:AzureAIResponsesAPIConfig/projects/precedence over the host checkapi-versionpropagationapi-keyvsBearerauth-header selection across commercial and.azure.ushostsapi_baseraisesValueErrorAll 24 pass. Existing
azure_aitests are unaffected (two unrelated pre-existing failures in that suite reproduce on the base branch without this change).Related