-
Notifications
You must be signed in to change notification settings - Fork 512
Open
Labels
Description
Feature Request
Add LM Studio as a local model provider alongside Ollama
Motivation
LM Studio demonstrates significant performance advantages over Ollama with the same models:
- 26-30% higher tokens/second on identical hardware
- Provides detailed performance metrics (tokens/sec, TTFT) in API responses
- GPU offloading enables partial acceleration when models exceed VRAM
- OpenAI-compatible API at
http://localhost:1234/v1/*
For local agent deployments, this performance difference is crucial for real-time responsiveness.
Proposal
Add rig::providers::lmstudio
module implementing the CompletionModel
trait:
// rig-core/src/providers/lmstudio/mod.rs
pub struct Client {
base_url: String,
http_client: reqwest::Client,
}
impl Client {
pub fn from_default() -> Self {
Self::new("http://localhost:1234/v1")
}
}
The provider would:
- Use OpenAI-compatible endpoints (already proven in Rig)
- Support streaming responses
- Expose LM Studio's performance metrics in responses