Deprecate non-OpenAI inference endpoint

[reluctantfuturist]

🚀 Describe the new functionality needed

Deprecate non-OpenAI inference endpoint

💡 Why is this needed? What if we don't build it?

Converging on a single API standard

Other thoughts

No response

[mattf]

THIS IS A DRAFT

Proposal: Simply remove non-OpenAI compatible endpoints for 0.2.11 REJECTED during 5 June 2025 community meeting.

Plan

• Mark endpoints as deprecated: docs in llama-stack
• Mark APIs as deprecated: warnings in llama-stack-client-python -- mark inference.{embeddings,chat_completion,completion} deprecated llama-stack-client-python#248
• Migrate llama-stack-apps to use OpenAI compat endpoints before removing endpoints
• Release 0.3 of llama-stack and llama-stack-client-python w/ endpoints removed (current version is 0.2.10)
• Migrate internal uses of endpoints: Agent -- Move Agents provider from /v1/inference/chat-completion to /v1/chat/completions (or /v1/responses) #2530, Update implementation for Agents API to use newer inference APIs #3067
• Migrate internal uses of endpoints: RAG
• Migrate internal uses of endpoints: ReActAgent?

Inference APIs

API	Legacy	Replacement
chat completions	/v1/inference/chat-completion	/v1/chat/completions
completions	/v1/inference/completion	/v1/completions
embeddings	/v1/inference/embeddings	/v1/embeddings
batch chat completions	/v1/inference/batch-chat-completion
batch completions	/v1/inference/batch-completion

llama-stack-apps uses

app	completions	chat completions	embeddings
DocQA	none	none	none
AgentStore	none	none	none
Agent Examples	none	none	none
Android Demo App	yes	yes	yes
Client Tools	none	none	none
Interior Design Assistant	none	none	none
iOSCalendarAssistant	none	none	none
iOSQuickDemo	client -> lib	none	none
Cybersecurity Demo	none	none	none
LlamaGuard	none	none	none
LlamaGuardMM	none	none	none

Inference providers

provider	chat	completions	embeddings	notes
anthropic	yes	yes	yes	pending via OpenAIMixin
bedrock	yes	yes	no	via OpenAIChatCompletionToLlamaStackMixin and OpenAICompletionToLlamaStackMixin
cerebas	yes	yes	yes	via OpenAIChatCompletionToLlamaStackMixin and OpenAICompletionToLlamaStackMixin
databricks	yes	yes	no	via OpenAIChatCompletionToLlamaStackMixin and OpenAICompletionToLlamaStackMixin
fireworks	yes	yes	yes	custom
gemini	yes	yes	yes	via LiteLLMOpenAIMixin
groq	yes	yes	yes	via LiteLLMOpenAIMixin
llama	yes	yes	yes	via OpenAIMixin
nvidia	yes	yes	no	via OpenAIMixin
ollama	yes	yes	yes	pending via OpenAIMixin
openai	yes	yes	yes	via OpenAIMixin
sambanova	yes	yes	yes	via LiteLLMOpenAIMixin
tgi	yes	yes	no	via OpenAIChatCompletionToLlamaStackMixin and OpenAICompletionToLlamaStackMixin
together	yes	yes	yes	custom
vllm	yes	yes	no	custom
watsonx	yes	yes	no	custom

Internal uses

chat_completion

All internal uses.

Notable

• inline meta-reference agent
• inline llama-guard safety
• inline llm-as-judge scoring
• inline rag tool
• inline meta-reference inference

Questions

• What about Batch endpoints?
• Keep endpoint names? Currently /v1/openai/v1/{chat/completions,completions,embeddings} -- CHANGE THEM to /v1/...