DocuFine AI is an AI engine that watches your screen recordings and listens to your voice-over to automatically generate documentation.
Visuals over Text: unlike generic RAG or transcription tools that just dump blocks of text, DocuFine AI understands that documentation is visual. It identifies the exact moment of interaction and generates annotated screenshots automatically—giving users "glanceable" context that pure text cannot match.
Upload a raw video file (from OBS, QuickTime, Zoom, etc.) and DocuFine AI uses Google Gemini 3 Flash to understand exactly what you're doing—then crops screenshots, highlights buttons, and writes crystal-clear instructions to create a professional HTML or MP4 video guide.
TL;DR: Record once with your voice → DocuFine AI turns it into a beautiful step-by-step guide. Zero manual editing.
docufine-ai-github.mp4
For Document Creators (Technical Writers, Support Engineers, & Help Desk)
DocuFine AI eliminates the manual overhead of traditional documentation. There are no browser extensions to install or complex recording modes to master. Simply capture your workflow using your preferred screen recorder and narrate as you go. The AI manages the "Visual Tax" by automatically cropping, zooming, and highlighting the precise UI elements during each interaction.
For End Users: The "Anti-Video" Experience
Users hate scrubbing through a 5-minute video just to find a 10-second answer. DocuFine AI synthesizes raw recordings into precision visual guides where they can skim, find the exact step they need, and get back to work—no play button required.
This project showcases the power of Multimodal AI—combining vision, voice, and temporal understanding to decode software interfaces.
1. Understanding Context (Vision + Voice)
The AI doesn't just "see" pixels—it watches and listens simultaneously. It uses your narration to understand why you're performing a task, and the video to pinpoint exactly where the action happens.
2. Pinpoint Accuracy & Smart Zoom
DocuFine AI knows exactly where every button, input field, and menu lives on your screen. It automatically decides when to show the full page for context, and when to zoom into dense UIs so users don't have to squint.
3. Capturing the "Perfect Moment"
Instead of random screenshots every 5 seconds, the AI analyzes the video to find the exact split-second an action occurs—ignoring mouse jitters and wait times, capturing only clean, decisive moments.
- 🎯 AI-Powered Visual Grounding: Automatically identifies key moments and interactive elements in your video
- 📍 Precision Annotation: Uses RapidOCR and fuzzy matching to place visual "spotlights" exactly on the buttons you interact with
- 📝 Voice-to-Text Integration: Extracts your narration and adds it as contextual notes to each step
- 🎨 Multiple Output Formats: Export as interactive HTML guides or MP4 videos
- ⚡ Smart Video Processing: Automatically creates lightweight proxies for large files to optimize AI analysis
- 🔄 Regenerate Logic: Quickly re-run visual analysis on existing videos to refine guide quality
- 🎭 Modern UI: Sleek glassmorphism dashboard with real-time job progress tracking
- AI Engine: Google Gemini 3 Flash (Multimodal Vision/Audio/Text)
- Visual Processing: RapidOCR, Pillow, FFmpeg
- Backend: Python 3.10+ / Flask
- Frontend: Tailwind CSS, Jinja2
- Fuzzy Matching: RapidFuzz for OCR text alignment
- Python 3.10+
- FFmpeg: Must be installed and available on your system path
- Google Gemini Access: Choose one authentication method:
- Option 1 (Recommended): Google AI Studio API Key (Free tier available)
- Option 2: Vertex AI with GCP Service Account (For enterprise deployments)
Note: This project currently works with Google's Gemini 3 Flash model only. Interested in testing with other AI providers like Anthropic, openRouter , openAI ...etc open an issue and let us know!
- Standard Usage (HTML Guides): Works efficiently on minimal hardware (e.g., free-tier VMs).
- Video Export (MP4 Guides): Requires more memory for rendering—we recommend at least 2GB RAM if you plan to generate video outputs.
-
Clone the repository:
git clone https://github.com/your-username/docufine-ai.git cd docufine-ai -
Create a virtual environment:
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
-
Install dependencies:
pip install -r src/requirements.txt
-
Configure environment variables:
Copy the sample environment file and edit it with your actual values:
cp src/.env.sample src/.env
Google AI Studio (Recommended) Ensure
GEMINI_API_KEYandFLASK_SECRET_KEYare set insrc/.env.Vertex AI (Enterprise) If using Vertex AI, configure the
GCP_SERVICE_ACCOUNT_JSONorGOOGLE_APPLICATION_CREDENTIALSsections in the same file.
cd src
python app.pyOpen your browser to http://localhost:5000 and start creating guides!
Build and run using the provided Dockerfile:
docker build -t docufine-ai .
docker run -p 5000:5000 --env-file src/.env docufine-aiFor the best documentation quality, follow these recording tips:
- Narrate as You Go: Since the AI listens to you, explaining why you're clicking a button helps it generate better descriptions
- Resolution Matters: Use 1080p recordings for optimal text recognition (720p minimum)
- Steady Mouse Movement: Move your cursor deliberately to the target—this helps the vision model track your focus
- The One-Second Rule: Pause briefly after every click or menu expansion to let animations complete
- Keep Text Visible: Don't rest your cursor directly on top of labels you want highlighted
💡 Pro-Tip: If the AI missed a step or didn't quite catch the right moment on the first try, don't worry! Just click the Regenerate button on the job status page to re-run the visual analysis.
💡 Pro-Tip: Want to change how the AI writes your guides? You can tweak the base prompt in src/prompts/system_prompt.txt to adjust the tone, detail level, or formatting to match your brand!
Contributions are welcome! Feel free to open issues or submit pull requests.
Love DocuFine AI? Give us a ⭐ on GitHub — it helps others discover the project!
MIT License - see LICENSE for details.