Skip to content

LOCAL_GPU_IMPLEMENTATION.md

Latest commit

 

History

History
312 lines (238 loc) · 8.69 KB

LOCAL_GPU_IMPLEMENTATION.md

File metadata and controls

312 lines (238 loc) · 8.69 KB
Raw

🚀 Local GPU Mode Implementation (Ollama + faster-whisper)

Start Date: 2026-02-02
Hardware: RTX 6000 Blackwell PRO (96GB VRAM)
Status: ✅ Complete - Ready for Testing

📋 Implementation Checklist

Phase 1: Configuration

  • 1.1 Update .env.example with new variables ✅
  • 1.2 Update requirements.txt with new dependencies ✅

Phase 2: Whisper Implementation (Local Transcription)

  • 2.1 Create local_whisper.py for faster-whisper ✅
  • 2.2 Modify whisper_transcription.py for hybrid mode ✅

Phase 3: VLM Implementation (Ollama Vision)

  • 3.1 Create local_vlm.py for Ollama communication ✅
  • 3.2 Modify sop_analyzer.py for hybrid mode ✅

Phase 4: Integration

  • 4.1 Modify main.py for automatic mode detection ✅
  • 4.2 Modify webapp/app.py for web interface ✅

Phase 5: Testing (AWAITING USER)

  • 5.1 Install Ollama and download model
  • 5.2 Install Python dependencies
  • 5.3 Test local Whisper
  • 5.4 Test Ollama VLM
  • 5.5 End-to-end test of the entire pipeline

Phase 6: Documentation

  • 6.1 This document - LOCAL_GPU_IMPLEMENTATION.md ✅
  • 6.2 Update main README.md (after testing)

🔧 WHAT YOU NEED TO DO (Prerequisites)

Step 1: Install Ollama

# Download installer from:
# https://ollama.com/download/windows

# Verify after installation:
ollama --version

Step 2: Download Vision model

# For your 96GB VRAM - best quality (90B parameters):
ollama pull llama3.2-vision:90b

# Alternatives (smaller, faster):
# ollama pull qwen2.5-vl:72b
# ollama pull llava:34b

# Verify model is downloaded:
ollama list

Step 3: Start Ollama server

# Ollama must run in the background:
ollama serve

# Or start Ollama via GUI (runs automatically after installation)

Step 4: Install Python dependencies

# Activate venv:
.\venv\Scripts\activate

# Install new dependencies:
pip install -r requirements.txt

Step 5: Create .env file

# Copy template:
copy .env.example .env

# Edit .env file - set these values:
# Main switch - LOCAL = GPU, API = Cloud
AI_MODE=LOCAL

# Ollama configuration
OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=llama3.2-vision:90b

# Whisper configuration
WHISPER_MODEL=large-v3
WHISPER_COMPUTE_TYPE=float16

Step 6: Test

# Test Ollama connection:
python local_vlm.py

# Test entire pipeline:
python main.py "path/to/test_video.mp4" -o test_output.pdf

📊 Progress Log

Time Action Status
12:49 Created tracking document ✅
12:50 Updated .env.example ✅
12:50 Updated requirements.txt ✅
12:51 Created local_whisper.py ✅
12:52 Created local_vlm.py ✅
12:53 Modified whisper_transcription.py ✅
12:54 Modified sop_analyzer.py ✅
12:55 Modified main.py ✅
12:56 Modified webapp/app.py ✅
--- IMPLEMENTATION COMPLETE ✅

📁 New/Modified Files

File Type Description Status
local_whisper.py NEW Local Whisper via faster-whisper ✅
local_vlm.py NEW Ollama VLM client ✅
sop_analyzer.py MODIFIED Hybrid mode (API/Local) ✅
whisper_transcription.py MODIFIED Hybrid mode (API/Local) ✅
main.py MODIFIED Mode detection + prerequisites ✅
.env.example MODIFIED New variables for LOCAL mode ✅
requirements.txt MODIFIED faster-whisper, httpx ✅
webapp/app.py MODIFIED Web interface with hybrid mode ✅

🎯 Target Architecture

┌─────────────────────────────────────────────────────────────┐
│                        main.py                               │
│              (automatic AI_MODE detection)                   │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│              AI_MODE = ? (from .env)                         │
└──────────┬──────────────────────────────┬───────────────────┘
           │                              │
           ▼                              ▼
┌──────────────────────┐      ┌──────────────────────┐
│   AI_MODE = "API"    │      │  AI_MODE = "LOCAL"   │
├──────────────────────┤      ├──────────────────────┤
│ • Gemini API         │      │ • Ollama VLM         │
│ • Groq Whisper       │      │ • faster-whisper     │
│ • Cloud processing   │      │ • RTX 6000 GPU       │
│ • Pay per use        │      │ • Zero cost          │
│ • Requires API keys  │      │ • Requires Ollama    │
└──────────────────────┘      └──────────────────────┘

🚀 How to Run

Local Mode (GPU) - RECOMMENDED FOR YOU:

# 1. Ensure Ollama is running:
ollama serve

# 2. Set AI_MODE=LOCAL in .env

# 3. CLI version:
python main.py "path/to/video.mp4" -o output.pdf

# 4. Or Web version:
cd webapp
python app.py
# Open http://localhost:5000

API Mode (Cloud):

# 1. Set in .env:
#    AI_MODE=API
#    GOOGLE_API_KEY=your_key
#    GROQ_API_KEY=your_key

# 2. Run:
python main.py "path/to/video.mp4" -o output.pdf

⚙️ Complete .env Configuration

# ============================================================
# AI MODE SELECTION
# ============================================================
# Options: "API" (cloud) or "LOCAL" (GPU)
AI_MODE=LOCAL

# ============================================================
# LOCAL MODE SETTINGS (used when AI_MODE=LOCAL)
# ============================================================

# Ollama server address
OLLAMA_HOST=http://localhost:11434

# Ollama Vision model
# For 96GB VRAM: llama3.2-vision:90b (best)
# Alternatives: qwen2.5-vl:72b, llava:34b
OLLAMA_MODEL=llama3.2-vision:90b

# Local Whisper model size
# Options: tiny, base, small, medium, large-v3
WHISPER_MODEL=large-v3

# Whisper compute type
# float16 = best quality, int8 = faster
WHISPER_COMPUTE_TYPE=float16

# ============================================================
# API MODE SETTINGS (used when AI_MODE=API)
# ============================================================

# Google Gemini API Key
GOOGLE_API_KEY=your_key_here

# Groq API Key (for Whisper)
GROQ_API_KEY=your_key_here

# ============================================================
# FLASK WEB APP
# ============================================================
SECRET_KEY=your_secret_key_here

🔍 Troubleshooting

"Ollama not responding"

# Start Ollama server:
ollama serve

# Or check if it's running:
curl http://localhost:11434/api/tags

"Model not found"

# Download model:
ollama pull llama3.2-vision:90b

# Verify available models:
ollama list

"CUDA not available"

# Verify CUDA installation:
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

# If it returns False, install PyTorch with CUDA:
pip install torch --index-url https://download.pytorch.org/whl/cu121

"faster-whisper import error"

pip install faster-whisper

"httpx import error"

pip install httpx

📈 Expected Performance

Operation API Mode LOCAL Mode (96GB VRAM)
Whisper Transcription (4min video) ~30s ~5-8s
VLM Analysis (20 frames) ~75s ~20-40s
PDF Generation ~5s ~5s
Total ~2 min ~30-60s
Cost $0.01-0.05/video $0

✅ DONE!

Implementation is complete. Now follow the steps in the "WHAT YOU NEED TO DO" section above.

After installing Ollama and downloading the model, you can test with:

python main.py "your_video.mp4" -o test.pdf

If you encounter issues, refer to the Troubleshooting section or ask.