Skip to content

feat(ask_sb): OpenAI compatible language models #424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 5, 2025

Conversation

brendan-kellam
Copy link
Contributor

@brendan-kellam brendan-kellam commented Aug 1, 2025

This PR adds support for any OpenAI compatible language model provider, such as Ollama and llama.cpp.

Summary by CodeRabbit

  • New Features

    • Added support for configuring and using any OpenAI-compatible language model, including self-hosted models such as Ollama and llama.cpp, by specifying a custom base URL and API key.
    • Updated chat interface to display a distinct icon for OpenAI-compatible providers.
  • Documentation

    • Expanded documentation to include setup instructions, example configurations, and troubleshooting for OpenAI-compatible providers.
    • Updated links to the latest Vercel AI SDK documentation.
    • Added a detailed schema reference section for language model configuration.

Copy link

coderabbitai bot commented Aug 1, 2025

Walkthrough

Support for OpenAI-compatible language model providers has been added across schema definitions, TypeScript types, documentation, and UI. This enables configuring self-hosted models compatible with the OpenAI Chat Completions API. Updates include new schema entries, TypeScript interfaces, documentation, dependency additions, and UI logic for provider handling and display.

Changes

Cohort / File(s) Change Summary
Documentation & Schema Reference
docs/docs/configuration/language-model-providers.mdx
Updated documentation to introduce the "OpenAI Compatible" provider, added usage instructions, troubleshooting, schema reference section, and updated Vercel AI SDK links.
Schema Definitions (Markdown)
docs/snippets/schemas/v3/index.schema.mdx, docs/snippets/schemas/v3/languageModel.schema.mdx
Added OpenAICompatibleLanguageModel definition and included it as a valid configuration option for language models.
Schema Definitions (JSON)
schemas/v3/languageModel.json
Added OpenAICompatibleLanguageModel object schema and updated oneOf array to include it.
Schema Implementation (TypeScript)
packages/schemas/src/v3/index.schema.ts, packages/schemas/src/v3/languageModel.schema.ts
Added new schema definition and validation logic for OpenAICompatibleLanguageModel and included it in the language model options.
TypeScript Types
packages/schemas/src/v3/index.type.ts, packages/schemas/src/v3/languageModel.type.ts
Introduced OpenAICompatibleLanguageModel interface and extended the LanguageModel union type to include it.
Web Package Dependency
packages/web/package.json
Added new dependency: @ai-sdk/openai-compatible.
Web API Logic
packages/web/src/app/api/(server)/chat/route.ts
Added support for the "openai-compatible" provider in the AI SDK model selection logic, including import and instantiation of the compatible client.
UI Model Provider Logo
packages/web/src/features/chat/components/chatBox/modelProviderLogo.tsx
Updated provider logo logic to support "openai-compatible" with an icon, adjusted rendering logic, and simplified class names for other providers.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant WebUI
    participant API_Server
    participant OpenAICompatibleClient
    participant SelfHostedModel

    User->>WebUI: Selects "OpenAI Compatible" provider
    WebUI->>API_Server: Sends chat request with provider config
    API_Server->>OpenAICompatibleClient: Instantiates client with baseUrl, model, token
    OpenAICompatibleClient->>SelfHostedModel: Sends Chat Completions API request
    SelfHostedModel-->>OpenAICompatibleClient: Returns response
    OpenAICompatibleClient-->>API_Server: Returns model response
    API_Server-->>WebUI: Returns chat response
    WebUI-->>User: Displays response with provider icon
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~15–20 minutes

Suggested reviewers

  • msukkari

Poem

A rabbit hopped in code so neat,
Now self-hosted models you can greet!
With schemas, docs, and icons new,
OpenAI-compatible hops into view.
From config to chat, the flow is clear—
More choices for all, let’s give a cheer!
🐇✨

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cb871c8 and 631e87e.

📒 Files selected for processing (7)
  • docs/snippets/schemas/v3/index.schema.mdx (2 hunks)
  • docs/snippets/schemas/v3/languageModel.schema.mdx (2 hunks)
  • packages/schemas/src/v3/index.schema.ts (2 hunks)
  • packages/schemas/src/v3/index.type.ts (2 hunks)
  • packages/schemas/src/v3/languageModel.schema.ts (2 hunks)
  • packages/schemas/src/v3/languageModel.type.ts (2 hunks)
  • schemas/v3/languageModel.json (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (7)
  • packages/schemas/src/v3/languageModel.type.ts
  • packages/schemas/src/v3/index.type.ts
  • docs/snippets/schemas/v3/languageModel.schema.mdx
  • packages/schemas/src/v3/index.schema.ts
  • schemas/v3/languageModel.json
  • packages/schemas/src/v3/languageModel.schema.ts
  • docs/snippets/schemas/v3/index.schema.mdx
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: build
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch bkellam/openai_compatible_model

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@TheDropZone
Copy link

I was just digging in to why my locally hosted openAI model wasn't working with sourcebot and realized this project didn't have openai-compatible-providers setup through the Vercel API. Props for putting this together! Excited for the add

This comment has been minimized.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
docs/snippets/schemas/v3/index.schema.mdx (1)

2637-2642: Duplicate inconsistency for baseUrl in the oneOf branch

The same “Optional base URL.” wording is repeated here while baseUrl remains mandatory (Lines 2644-2648). Please sync the description with the actual requirement to avoid schema drift.

🧹 Nitpick comments (1)
packages/schemas/src/v3/index.schema.ts (1)

2590-2649: Consider using schema references to reduce duplication.

The schema definition is correct and follows the established pattern in this file. However, the complete duplication of the schema definition (both in definitions and oneOf) could lead to maintenance challenges.

Consider using JSON Schema $ref to reference the definition:

-          {
-            "type": "object",
-            "properties": {
-              "provider": {
-                "const": "openai-compatible",
-                "description": "OpenAI Compatible Configuration"
-              },
-              "model": {
-                "type": "string",
-                "description": "The name of the language model."
-              },
-              // ... rest of properties
-            },
-            "required": [
-              "provider",
-              "model",
-              "baseUrl"
-            ],
-            "additionalProperties": false
-          }
+          {
+            "$ref": "#/definitions/OpenAICompatibleLanguageModel"
+          }

This would maintain consistency with the definitions section while reducing duplication.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b46fb53 and 621081c.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (11)
  • docs/docs/configuration/language-model-providers.mdx (13 hunks)
  • docs/snippets/schemas/v3/index.schema.mdx (2 hunks)
  • docs/snippets/schemas/v3/languageModel.schema.mdx (2 hunks)
  • packages/schemas/src/v3/index.schema.ts (2 hunks)
  • packages/schemas/src/v3/index.type.ts (2 hunks)
  • packages/schemas/src/v3/languageModel.schema.ts (2 hunks)
  • packages/schemas/src/v3/languageModel.type.ts (2 hunks)
  • packages/web/package.json (1 hunks)
  • packages/web/src/app/api/(server)/chat/route.ts (2 hunks)
  • packages/web/src/features/chat/components/chatBox/modelProviderLogo.tsx (4 hunks)
  • schemas/v3/languageModel.json (2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*

📄 CodeRabbit Inference Engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

  • packages/web/package.json
  • packages/schemas/src/v3/index.type.ts
  • docs/docs/configuration/language-model-providers.mdx
  • packages/schemas/src/v3/languageModel.schema.ts
  • packages/web/src/app/api/(server)/chat/route.ts
  • packages/web/src/features/chat/components/chatBox/modelProviderLogo.tsx
  • packages/schemas/src/v3/languageModel.type.ts
  • schemas/v3/languageModel.json
  • docs/snippets/schemas/v3/languageModel.schema.mdx
  • packages/schemas/src/v3/index.schema.ts
  • docs/snippets/schemas/v3/index.schema.mdx
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: build
🔇 Additional comments (23)
packages/web/package.json (1)

23-23: LGTM!

The addition of @ai-sdk/openai-compatible dependency is correctly placed in alphabetical order and uses an appropriate version constraint consistent with other AI SDK packages.

packages/web/src/app/api/(server)/chat/route.ts (2)

20-20: LGTM!

The import for createOpenAICompatible is correctly added and follows the existing import pattern for AI SDK providers.


448-460: Well-implemented OpenAI-compatible provider support.

The implementation correctly:

  • Uses required baseUrl from config (consistent with schema)
  • Falls back to modelId when displayName is not provided
  • Handles optional token authentication properly
  • Returns the chat model instance consistently with other providers

The simpler approach without provider options or headers is appropriate for a generic compatibility provider.

packages/schemas/src/v3/index.type.ts (2)

24-24: LGTM!

The OpenAICompatibleLanguageModel is correctly added to the LanguageModel union type, maintaining alphabetical order with other provider types.


795-828: Well-defined interface for OpenAI-compatible provider.

The OpenAICompatibleLanguageModel interface correctly:

  • Defines the provider as a literal type "openai-compatible"
  • Requires baseUrl field (appropriate for self-hosted models)
  • Follows established patterns for token authentication with secret/env union
  • Includes clear documentation for the token's Authorization header behavior

The interface structure is consistent with other language model providers in the codebase.

schemas/v3/languageModel.json (2)

354-386: Well-structured schema definition for OpenAI-compatible provider.

The OpenAICompatibleLanguageModel schema correctly:

  • Defines required properties (provider, model, baseUrl) appropriate for self-hosted models
  • Uses proper URL format validation with regex pattern consistent with other providers
  • References shared Token definition for authentication
  • Includes clear description of token behavior for Authorization header
  • Enforces strict validation with additionalProperties: false

484-486: LGTM!

The new OpenAICompatibleLanguageModel reference is correctly added to the oneOf array, enabling proper schema validation for the new provider type.

docs/docs/configuration/language-model-providers.mdx (4)

6-10: Excellent documentation addition.

The import statement and introductory note effectively guide users to the new OpenAI-compatible provider option for self-hosting models.


257-283: Comprehensive OpenAI Compatible provider documentation.

The new section effectively:

  • Clearly explains the purpose and compatibility scope (Ollama, llama.cpp)
  • Provides a well-structured example configuration
  • Links to relevant Vercel AI SDK documentation
  • Includes practical troubleshooting guidance for llama.cpp users

The documentation will help users successfully configure self-hosted OpenAI-compatible models.


327-334: Valuable schema reference addition.

The new schema reference section with accordion provides users easy access to the complete JSON schema definition, enhancing the documentation's completeness.


54-307: Consistent link updates.

All Vercel AI SDK documentation links have been properly updated from the old v5.ai-sdk.dev domain to the current ai-sdk.dev domain, ensuring users access up-to-date documentation.

packages/schemas/src/v3/languageModel.type.ts (2)

13-13: LGTM! Proper integration of the new provider type.

The addition of OpenAICompatibleLanguageModel to the union type is correctly placed alphabetically and follows the established pattern.


367-400: Well-designed interface for OpenAI-compatible providers.

The OpenAICompatibleLanguageModel interface is well-structured with a key architectural decision: making baseUrl required rather than optional. This is appropriate since OpenAI-compatible providers (like Ollama, llama.cpp) require specifying the self-hosted endpoint. The token documentation clearly explains the Authorization header behavior.

docs/snippets/schemas/v3/languageModel.schema.mdx (2)

626-685: Comprehensive schema definition for OpenAI-compatible providers.

The JSON schema properly defines all required and optional properties with appropriate validation rules. The required baseUrl field (line 682) correctly enforces that self-hosted endpoints must be specified, which is essential for OpenAI-compatible providers.


1429-1488: Consistent schema validation in oneOf array.

The schema definition in the oneOf array correctly mirrors the definitions section, ensuring proper validation. The required fields (lines 1482-1486) are consistent with the interface requirements.

packages/web/src/features/chat/components/chatBox/modelProviderLogo.tsx (5)

16-16: Appropriate icon import for the new provider.

The Box icon from lucide-react is a suitable choice for representing OpenAI-compatible providers, providing a generic yet recognizable symbol for self-hosted models.


27-27: Well-typed interface extension.

The updated return type properly includes the Icon property with LucideIcon type, maintaining type safety while supporting both image and icon-based logos.


32-75: Good refactoring of logo class handling.

Removing the explicit width/height classes from individual provider cases improves maintainability since these dimensions are now consistently applied in the render logic (lines 89, 96).


76-80: Thoughtful provider representation.

The openai-compatible case uses the Box icon with muted styling, which appropriately represents the generic/self-hosted nature of OpenAI-compatible providers while maintaining visual consistency.


84-100: Clean conditional rendering logic.

The render logic elegantly handles both image-based and icon-based logos with consistent sizing and styling. The fallback to null ensures graceful handling of undefined providers.

packages/schemas/src/v3/index.schema.ts (1)

1787-1846: LGTM! Well-structured schema definition.

The OpenAICompatibleLanguageModel schema is properly structured and consistent with other language model definitions. The required baseUrl field appropriately distinguishes this from the standard OpenAI provider, and the token handling follows established patterns.

packages/schemas/src/v3/languageModel.schema.ts (2)

625-684: LGTM! Well-designed schema for OpenAI-compatible providers.

The schema definition correctly implements support for OpenAI-compatible providers with appropriate required fields. Making baseUrl required is the right choice for self-hosted models, and the token description clearly explains the Authorization header behavior.


1428-1487: LGTM! Consistent oneOf entry implementation.

The oneOf entry correctly mirrors the OpenAI-compatible provider definition and follows the established pattern used throughout the schema file.

@brendan-kellam brendan-kellam merged commit 65d3cd9 into main Aug 5, 2025
7 checks passed
@brendan-kellam brendan-kellam deleted the bkellam/openai_compatible_model branch August 5, 2025 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants