Skip to content

GETTING_STARTED.md

Latest commit

 

History

History
291 lines (220 loc) · 7.07 KB

GETTING_STARTED.md

File metadata and controls

291 lines (220 loc) · 7.07 KB

Getting Started with LLM Middleware

This guide will help you get up and running with LLM Middleware in your project.

Prerequisites

  • Node.js 18+ installed
  • TypeScript 4.9+ knowledge
  • LLM Provider configured (e.g., Ollama server for Ollama provider)

Installation

npm install llm-middleware

Quick Setup

1. Environment Configuration

Create a .env file in your project root:

# Server Configuration
PORT=3000
NODE_ENV=development
LOG_LEVEL=info

# LLM Provider Configuration
# IMPORTANT: MODEL1_NAME is REQUIRED
MODEL1_NAME=phi3:mini               # Required: Your model name (e.g., phi3:mini, llama3:8b, gemma2:2b)
MODEL1_URL=http://localhost:11434  # Optional: Defaults to localhost:11434 (Ollama default)
MODEL1_TOKEN=your-token-here        # Optional: For authenticated providers

2. Basic Use Case Implementation

Create your first AI use case:

import {
  BaseAIUseCase,
  BaseAIRequest,
  BaseAIResult,
  LLMProvider  // Import for provider selection
} from 'llm-middleware';

// Define your interfaces
interface ChatRequest extends BaseAIRequest<string> {
  message: string;
}

interface ChatResult extends BaseAIResult {
  response: string;
}

// Implement your use case (uses Ollama by default)
class SimpleChatUseCase extends BaseAIUseCase<string, ChatRequest, ChatResult> {
  protected readonly systemMessage = \`
    You are a helpful AI assistant.
    Provide clear and concise responses.
  \`;

  // Required: Return the user template function
  protected getUserTemplate(): (formattedPrompt: string) => string {
    return (message) => message;  // For simple chat, return message as-is
  }

  protected formatUserMessage(prompt: any): string {
    return typeof prompt === 'string' ? prompt : prompt.message;
  }

  protected createResult(
    content: string,
    usedPrompt: string,
    thinking?: string
  ): ChatResult {
    return {
      generatedContent: content,
      model: this.modelConfig.name,
      usedPrompt,
      thinking,
      response: content
    };
  }
}

// Use a different provider (e.g., Anthropic Claude)
class AnthropicChatUseCase extends SimpleChatUseCase {
  protected getProvider(): LLMProvider {
    return LLMProvider.ANTHROPIC;  // Override to use Claude
  }
}

Provider Selection: Override getProvider() to use different LLM providers (Ollama, Anthropic, OpenAI, Google). Default is Ollama.

3. Controller Implementation

Create a controller to handle HTTP requests:

import { BaseController } from 'llm-middleware';
import { RequestWithUser } from 'llm-middleware';
import { Response } from 'express';

class ChatController extends BaseController {
  private chatUseCase = new SimpleChatUseCase();

  public async chat(req: RequestWithUser, res: Response): Promise<void> {
    await this.handleRequest(req, res, async () => {
      const { message } = req.body;
      
      if (!message) {
        throw new Error('Message is required');
      }

      const result = await this.chatUseCase.execute({
        prompt: message,
        authToken: req.headers.authorization
      });
      
      return {
        response: result.response,
        model: result.model
      };
    });
  }
}

4. Express Server Setup

Set up your Express server:

import express from 'express';
import cors from 'cors';
import { appConfig, logger } from 'llm-middleware';

const app = express();
const chatController = new ChatController();

// Middleware
app.use(cors());
app.use(express.json());

// Client info middleware
app.use((req: any, res, next) => {
  req.clientInfo = {
    platform: req.headers['user-agent'] || 'unknown',
    ip: req.ip || 'unknown',
    timestamp: new Date().toISOString()
  };
  next();
});

// Routes
app.post('/api/chat', (req: any, res) => {
  chatController.chat(req, res);
});

// Start server
const port = appConfig.server.port;
app.listen(port, () => {
  logger.info(\`Server running on port \${port}\`, {
    context: 'Server'
  });
});

Testing Your Setup

1. Start Your LLM Provider

Make sure your LLM provider is running (example for Ollama):

ollama serve

2. Start Your Server

npm run dev

3. Test the API

curl -X POST http://localhost:3000/api/chat \\
  -H "Content-Type: application/json" \\
  -d '{"message": "Hello, how are you?"}'

Expected response:

{
  "success": true,
  "data": {
    "response": "Hello! I'm doing well, thank you for asking...",
    "model": "mistral:latest"
  },
  "timestamp": "2025-09-07T21:00:00.000Z"
}

Configuration Options

Model Configuration

You can configure multiple models:

import { getModelConfig } from 'llm-middleware';

// Get specific model config (throws error if MODEL1_NAME not set)
const model1 = getModelConfig('MODEL1');
console.log(model1.name);     // Value from MODEL1_NAME env variable
console.log(model1.baseUrl);  // Value from MODEL1_URL or default localhost

// Use in your use case
class MyUseCase extends BaseAIUseCase<MyRequest, MyResult> {
  // Override default model (optional - MODEL1 is now default)
  protected get modelConfigKey(): ModelConfigKey {
    return 'MODEL1'; // MODEL1 is the only supported model
  }
}

Logging Configuration

Control logging behavior:

import { logger } from 'llm-middleware';

// Different log levels
await logger.debug('Debug message');
await logger.info('Info message');
await logger.warn('Warning message');
await logger.error('Error message');

// With context and metadata
await logger.info('Operation completed', {
  context: 'MyService',
  metadata: {
    userId: 123,
    duration: 150
  }
});

Next Steps

Common Issues

LLM Provider Connection Error

If you get connection errors (Ollama example):

  1. Check if your provider is running: ollama list (for Ollama)
  2. Verify the URL in your .env file
  3. Test direct connection: curl http://localhost:11434/api/tags (for Ollama)

TypeScript Errors

Make sure you have the correct types installed:

npm install @types/node @types/express

Model Not Found

If you get "model not found" errors (Ollama example):

  1. Check available models: ollama list (for Ollama)
  2. Pull a model: ollama pull phi3:mini (or any model you prefer)
  3. Set MODEL1_NAME in your .env file to match the model name

Getting Help