From 39cca20ee82ba2756b8475d55a05b48ad868130c Mon Sep 17 00:00:00 2001 From: Kyle Carberry Date: Wed, 19 Nov 2025 10:17:30 -0500 Subject: [PATCH 1/3] =?UTF-8?q?=F0=9F=A4=96=20feat:=20add=20Cloudflare=20A?= =?UTF-8?q?I=20Gateway=20support?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add support for routing AI requests through Cloudflare AI Gateway for caching, analytics, rate limiting, and cost control. Implementation: - Added cloudflareGateway config to ProviderConfig type with accountId and gatewayName fields - Updated aiService.ts to construct gateway URL when configured - Updated workspaceTitleGenerator.ts to use gateway baseURL - Gateway URL construction: https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayName}/{provider} - cloudflareGateway takes precedence over baseUrl when both present - Added helper function buildCloudflareGatewayURL() for URL construction - Updated providers.jsonc example comments with gateway configuration Documentation: - Created comprehensive docs/cloudflare-ai-gateway.md with setup, configuration, troubleshooting, and benefits - Added to docs/SUMMARY.md under Advanced section Configuration Example: { "anthropic": { "apiKey": "sk-ant-...", "cloudflareGateway": { "accountId": "your-account-id", "gatewayName": "your-gateway-name" } } } All providers (Anthropic, OpenAI, Google, etc.) are supported. Gateway is optional - omit config for direct connections. _Generated with `mux`_ --- bun.lock | 1 + docs/SUMMARY.md | 1 + docs/cloudflare-ai-gateway.md | 166 +++++++++++++++++++ src/node/config.ts | 25 +++ src/node/services/aiService.ts | 47 +++++- src/node/services/workspaceTitleGenerator.ts | 56 +++++-- 6 files changed, 279 insertions(+), 17 deletions(-) create mode 100644 docs/cloudflare-ai-gateway.md diff --git a/bun.lock b/bun.lock index ec7e3fa97..9b210b9f5 100644 --- a/bun.lock +++ b/bun.lock @@ -1,5 +1,6 @@ { "lockfileVersion": 1, + "configVersion": 0, "workspaces": { "": { "name": "@coder/cmux", diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 7a7677d31..c26628cf2 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -26,6 +26,7 @@ - [Prompting Tips](./prompting-tips.md) - [System Prompt](./system-prompt.md) +- [Cloudflare AI Gateway](./cloudflare-ai-gateway.md) - [Telemetry](./telemetry.md) # Development diff --git a/docs/cloudflare-ai-gateway.md b/docs/cloudflare-ai-gateway.md new file mode 100644 index 000000000..e716f82a6 --- /dev/null +++ b/docs/cloudflare-ai-gateway.md @@ -0,0 +1,166 @@ +# Cloudflare AI Gateway + +Cloudflare AI Gateway provides a unified interface for accessing AI models from various providers with built-in features like caching, rate limiting, analytics, and cost control. + +## What is Cloudflare AI Gateway? + +Cloudflare AI Gateway sits between your application and AI model providers, offering: + +- **Request management**: Rate limiting, request queueing, and load balancing +- **Caching**: Cache AI responses to reduce costs and improve latency +- **Analytics**: Track usage, costs, and performance across all providers +- **Cost control**: Set spending limits and monitor API usage +- **Observability**: Detailed logs of all AI requests and responses + +## Setup + +### 1. Create a Cloudflare AI Gateway + +1. Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up) (free tier available) +2. Navigate to the AI Gateway section in your Cloudflare dashboard +3. Click "Create Gateway" and give it a name (e.g., `my-mux-gateway`) +4. Note your **Account ID** (found in the dashboard URL or account settings) +5. Note your **Gateway Name** (the name you just created) + +### 2. Configure mux + +Open or create `~/.mux/providers.jsonc` and add the `cloudflareGateway` configuration to any provider: + +```jsonc +{ + "anthropic": { + "apiKey": "sk-ant-...", + "cloudflareGateway": { + "accountId": "your-cloudflare-account-id", + "gatewayName": "my-mux-gateway" + } + }, + "openai": { + "apiKey": "sk-...", + "cloudflareGateway": { + "accountId": "your-cloudflare-account-id", + "gatewayName": "my-mux-gateway" + } + } +} +``` + +### 3. Verify Setup + +When you send a message, mux will automatically route requests through Cloudflare AI Gateway. You should see a log message in the console: + +``` +Using Cloudflare AI Gateway { provider: 'anthropic', accountId: '...', gatewayName: '...' } +``` + +All AI requests will now appear in your Cloudflare AI Gateway dashboard with detailed analytics. + +## Configuration Options + +### Per-Provider Configuration + +You can configure different gateways for different providers: + +```jsonc +{ + "anthropic": { + "apiKey": "sk-ant-...", + "cloudflareGateway": { + "accountId": "account-id-1", + "gatewayName": "production-gateway" + } + }, + "openai": { + "apiKey": "sk-...", + "cloudflareGateway": { + "accountId": "account-id-2", + "gatewayName": "testing-gateway" + } + }, + // Ollama without gateway (direct connection) + "ollama": { + "baseUrl": "http://localhost:11434/api" + } +} +``` + +### Mixing Gateway and Direct Connections + +You can use Cloudflare AI Gateway for some providers and direct connections for others. Simply omit the `cloudflareGateway` configuration for providers you want to connect directly. + +## How It Works + +When `cloudflareGateway` is configured: + +1. mux constructs the Cloudflare AI Gateway URL: + ``` + https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayName}/{provider} + ``` + +2. This URL is passed to the Vercel AI SDK as the `baseURL` parameter + +3. All API requests are routed through Cloudflare's infrastructure + +4. Your provider API key is still required and sent with each request + +5. Cloudflare logs, caches, and manages the requests according to your gateway settings + +## Benefits + +### Cost Optimization + +- **Caching**: Identical requests can be served from cache, avoiding provider API calls +- **Rate limiting**: Prevent unexpected bill spikes from runaway requests +- **Usage tracking**: Monitor spending across all providers in one dashboard + +### Performance + +- **Global edge network**: Requests are routed through Cloudflare's global network +- **Intelligent caching**: Reduce latency for repeated queries +- **Load balancing**: Distribute requests efficiently + +### Observability + +- **Request logs**: See every AI request and response +- **Analytics**: Track usage patterns, error rates, and performance +- **Cost breakdown**: Understand spending by model, project, or time period + +## Troubleshooting + +### Gateway Not Working + +1. **Verify account ID and gateway name**: Check your Cloudflare dashboard +2. **Check API key**: Ensure your provider API key is still valid +3. **Review Cloudflare logs**: Check the AI Gateway dashboard for error messages +4. **Test direct connection**: Temporarily remove `cloudflareGateway` to verify basic connectivity + +### Performance Issues + +1. **Check cache settings**: Cloudflare might be caching responses you don't want cached +2. **Review rate limits**: Your gateway might be throttling requests +3. **Check regional routing**: Ensure Cloudflare edge locations are optimal for your region + +### Missing Analytics + +1. **Verify gateway is active**: Check Cloudflare dashboard shows "Active" status +2. **Wait for propagation**: Analytics may take a few minutes to appear +3. **Check configuration**: Ensure `accountId` and `gatewayName` match exactly + +## Supported Providers + +Cloudflare AI Gateway supports all major AI providers that mux integrates with: + +- Anthropic (Claude) +- OpenAI (GPT, o1, o3) +- Google (Gemini) +- Any provider using OpenAI-compatible APIs + +## Advanced Configuration + +For advanced Cloudflare AI Gateway features (custom caching, rate limiting, etc.), configure them in your Cloudflare dashboard. The mux integration automatically uses whatever settings you've configured for your gateway. + +## Learn More + +- [Cloudflare AI Gateway Documentation](https://developers.cloudflare.com/ai-gateway/) +- [Vercel AI SDK Integration Guide](https://developers.cloudflare.com/ai-gateway/integrations/vercel-ai-sdk/) +- [Cloudflare AI Gateway Dashboard](https://dash.cloudflare.com/) diff --git a/src/node/config.ts b/src/node/config.ts index a0f92fcc8..aa4f15e27 100644 --- a/src/node/config.ts +++ b/src/node/config.ts @@ -16,6 +16,13 @@ export type { Workspace, ProjectConfig, ProjectsConfig }; export interface ProviderConfig { apiKey?: string; baseUrl?: string; + /** Cloudflare AI Gateway configuration */ + cloudflareGateway?: { + /** Cloudflare account ID */ + accountId: string; + /** Gateway name (created in Cloudflare dashboard) */ + gatewayName: string; + }; [key: string]: unknown; } @@ -465,6 +472,24 @@ export class Config { // "baseUrl": "http://localhost:11434/api" // Optional - only needed for remote/custom URL // } // } +// +// Cloudflare AI Gateway example: +// { +// "anthropic": { +// "apiKey": "sk-ant-...", +// "cloudflareGateway": { +// "accountId": "your-cloudflare-account-id", +// "gatewayName": "your-gateway-name" +// } +// }, +// "openai": { +// "apiKey": "sk-...", +// "cloudflareGateway": { +// "accountId": "your-cloudflare-account-id", +// "gatewayName": "your-gateway-name" +// } +// } +// } ${jsonString}`; fs.writeFileSync(this.providersFile, contentWithComments); diff --git a/src/node/services/aiService.ts b/src/node/services/aiService.ts index 166ac28ba..5958552b7 100644 --- a/src/node/services/aiService.ts +++ b/src/node/services/aiService.ts @@ -132,6 +132,21 @@ function parseModelString(modelString: string): [string, string] { return [providerName, modelId]; } +/** + * Construct Cloudflare AI Gateway URL for a provider + * @param accountId - Cloudflare account ID + * @param gatewayName - Gateway name from Cloudflare dashboard + * @param provider - Provider name (openai, anthropic, google, etc.) + * @returns Gateway URL for the provider + */ +function buildCloudflareGatewayURL( + accountId: string, + gatewayName: string, + provider: string +): string { + return `https://gateway.ai.cloudflare.com/v1/${accountId}/${gatewayName}/${provider}`; +} + export class AIService extends EventEmitter { private readonly streamManager: StreamManager; private readonly historyService: HistoryService; @@ -280,10 +295,34 @@ export class AIService extends EventEmitter { let providerConfig = providersConfig?.[providerName] ?? {}; // Map baseUrl to baseURL if present (SDK expects baseURL) - const { baseUrl, ...configWithoutBaseUrl } = providerConfig; - providerConfig = baseUrl - ? { ...configWithoutBaseUrl, baseURL: baseUrl } - : configWithoutBaseUrl; + const { baseUrl, cloudflareGateway, ...configWithoutBaseUrl } = providerConfig; + + // If Cloudflare AI Gateway is configured, use it as baseURL (takes precedence over baseUrl) + if (cloudflareGateway && typeof cloudflareGateway === "object") { + const { accountId, gatewayName } = cloudflareGateway as { + accountId?: string; + gatewayName?: string; + }; + if (accountId && gatewayName) { + providerConfig = { + ...configWithoutBaseUrl, + baseURL: buildCloudflareGatewayURL(accountId, gatewayName, providerName), + }; + log.info("Using Cloudflare AI Gateway", { + provider: providerName, + accountId, + gatewayName, + }); + } else { + providerConfig = baseUrl + ? { ...configWithoutBaseUrl, baseURL: baseUrl } + : configWithoutBaseUrl; + } + } else { + providerConfig = baseUrl + ? { ...configWithoutBaseUrl, baseURL: baseUrl } + : configWithoutBaseUrl; + } // Handle Anthropic provider if (providerName === "anthropic") { diff --git a/src/node/services/workspaceTitleGenerator.ts b/src/node/services/workspaceTitleGenerator.ts index f680e9665..ce73c55aa 100644 --- a/src/node/services/workspaceTitleGenerator.ts +++ b/src/node/services/workspaceTitleGenerator.ts @@ -1,11 +1,45 @@ import { generateObject, type LanguageModel } from "ai"; import { z } from "zod"; -import type { Config } from "@/node/config"; +import type { Config, ProviderConfig } from "@/node/config"; import { log } from "./log"; import { createAnthropic } from "@ai-sdk/anthropic"; import { createOpenAI } from "@ai-sdk/openai"; import { MODEL_NAMES } from "@/common/constants/knownModels"; +/** + * Build configuration with Cloudflare AI Gateway support + * @param providerConfig - Provider configuration from providers.jsonc + * @param providerName - Provider name (openai, anthropic, etc.) + * @returns Configuration object with baseURL set if gateway is configured + */ +function buildProviderConfig( + providerConfig: ProviderConfig, + providerName: string +): Record { + const { cloudflareGateway, baseUrl, ...rest } = providerConfig; + + // If Cloudflare AI Gateway is configured, use it as baseURL + if (cloudflareGateway && typeof cloudflareGateway === "object") { + const { accountId, gatewayName } = cloudflareGateway as { + accountId?: string; + gatewayName?: string; + }; + if (accountId && gatewayName) { + return { + ...rest, + baseURL: `https://gateway.ai.cloudflare.com/v1/${accountId}/${gatewayName}/${providerName}`, + }; + } + } + + // Otherwise use baseUrl if provided (mapped to baseURL) + if (baseUrl) { + return { ...rest, baseURL: baseUrl }; + } + + return rest; +} + const workspaceNameSchema = z.object({ name: z .string() @@ -70,17 +104,15 @@ function getModelForTitleGeneration(modelString: string, config: Config): Langua try { // Try Anthropic Haiku first (fastest/cheapest) if (providersConfig.anthropic?.apiKey) { - const provider = createAnthropic({ - apiKey: String(providersConfig.anthropic.apiKey), - }); + const config = buildProviderConfig(providersConfig.anthropic, "anthropic"); + const provider = createAnthropic(config); return provider(MODEL_NAMES.anthropic.HAIKU); } // Try OpenAI GPT-5-mini second if (providersConfig.openai?.apiKey) { - const provider = createOpenAI({ - apiKey: String(providersConfig.openai.apiKey), - }); + const config = buildProviderConfig(providersConfig.openai, "openai"); + const provider = createOpenAI(config); return provider(MODEL_NAMES.openai.GPT_MINI); } @@ -92,16 +124,14 @@ function getModelForTitleGeneration(modelString: string, config: Config): Langua } if (providerName === "anthropic" && providersConfig.anthropic?.apiKey) { - const provider = createAnthropic({ - apiKey: String(providersConfig.anthropic.apiKey), - }); + const config = buildProviderConfig(providersConfig.anthropic, "anthropic"); + const provider = createAnthropic(config); return provider(modelId); } if (providerName === "openai" && providersConfig.openai?.apiKey) { - const provider = createOpenAI({ - apiKey: String(providersConfig.openai.apiKey), - }); + const config = buildProviderConfig(providersConfig.openai, "openai"); + const provider = createOpenAI(config); return provider(modelId); } From 1d24fad1d71ce1fff76dd117f9c393ef7d4c86d9 Mon Sep 17 00:00:00 2001 From: Kyle Carberry Date: Wed, 19 Nov 2025 10:21:28 -0500 Subject: [PATCH 2/3] =?UTF-8?q?=F0=9F=A4=96=20fix:=20format=20cloudflare-a?= =?UTF-8?q?i-gateway.md=20with=20prettier?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _Generated with `mux`_ --- docs/cloudflare-ai-gateway.md | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/cloudflare-ai-gateway.md b/docs/cloudflare-ai-gateway.md index e716f82a6..6cdec63d0 100644 --- a/docs/cloudflare-ai-gateway.md +++ b/docs/cloudflare-ai-gateway.md @@ -32,16 +32,16 @@ Open or create `~/.mux/providers.jsonc` and add the `cloudflareGateway` configur "apiKey": "sk-ant-...", "cloudflareGateway": { "accountId": "your-cloudflare-account-id", - "gatewayName": "my-mux-gateway" - } + "gatewayName": "my-mux-gateway", + }, }, "openai": { "apiKey": "sk-...", "cloudflareGateway": { "accountId": "your-cloudflare-account-id", - "gatewayName": "my-mux-gateway" - } - } + "gatewayName": "my-mux-gateway", + }, + }, } ``` @@ -67,20 +67,20 @@ You can configure different gateways for different providers: "apiKey": "sk-ant-...", "cloudflareGateway": { "accountId": "account-id-1", - "gatewayName": "production-gateway" - } + "gatewayName": "production-gateway", + }, }, "openai": { "apiKey": "sk-...", "cloudflareGateway": { "accountId": "account-id-2", - "gatewayName": "testing-gateway" - } + "gatewayName": "testing-gateway", + }, }, // Ollama without gateway (direct connection) "ollama": { - "baseUrl": "http://localhost:11434/api" - } + "baseUrl": "http://localhost:11434/api", + }, } ``` @@ -93,6 +93,7 @@ You can use Cloudflare AI Gateway for some providers and direct connections for When `cloudflareGateway` is configured: 1. mux constructs the Cloudflare AI Gateway URL: + ``` https://gateway.ai.cloudflare.com/v1/{accountId}/{gatewayName}/{provider} ``` From 99d6735ea57a8d055148761b09e08b3f5c3ff6f5 Mon Sep 17 00:00:00 2001 From: Kyle Carberry Date: Wed, 19 Nov 2025 10:22:39 -0500 Subject: [PATCH 3/3] =?UTF-8?q?=F0=9F=A4=96=20fix:=20preserve=20OpenAI=20t?= =?UTF-8?q?runcation=20patch=20with=20Cloudflare=20gateway=20URLs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated regex to match /responses anywhere in path, not just /v1/responses. This ensures truncation: auto is applied when routing through Cloudflare gateway which uses paths like /v1/{account}/{gateway}/openai/responses. Without this fix, large messages routed through Cloudflare gateway would fail with 413 errors due to missing automatic truncation. _Generated with `mux`_ --- src/node/services/aiService.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/node/services/aiService.ts b/src/node/services/aiService.ts index 5958552b7..6a363ccbd 100644 --- a/src/node/services/aiService.ts +++ b/src/node/services/aiService.ts @@ -389,7 +389,10 @@ export class AIService extends EventEmitter { })(); const method = (init?.method ?? "GET").toUpperCase(); - const isOpenAIResponses = /\/v1\/responses(\?|$)/.test(urlString); + // Match /responses anywhere in path (supports both direct and gateway URLs) + // Direct: /v1/responses + // Gateway: /v1/{account}/{gateway}/openai/responses + const isOpenAIResponses = /\/responses(\?|$)/.test(urlString); const body = init?.body; if (