AI Math Tutor

A voice-first AI math tutor that helps students understand homework in real time by reading the whiteboard, talking with the student, and writing step-by-step explanations directly on a shared canvas.

The Problem

Every student eventually gets stuck on homework. For math, the problem is especially painful: a student may understand the class material generally, but still freeze when a specific equation, fraction, or written step does not make sense.

When that happens, the options are limited:

• Ask a parent or friend, if someone is available and knows the topic.
• Search online and hope the explanation matches the exact problem.
• Use a chatbot, which often gives a text answer disconnected from the student's actual work.
• Hire a human tutor, which can be very expensive and is not always available at the exact moment the student needs help.

One-on-one tutoring is valuable because a real tutor can look at the student's work, ask questions, explain out loud, write the next step, correct mistakes, and adapt the pace. But private tutoring is costly and difficult to scale. This creates a large opportunity for AI tutoring products that can provide immediate, personalized, visual, and conversational help.

The Idea

AI Math Tutor explores what a real-time AI math tutor could feel like.

Instead of only answering in a chat box, the AI tutor shares a whiteboard with the student. The student can upload a homework image, write on the board, or ask a question by voice. The AI joins through LiveKit, listens to the student, reads the current tldraw canvas, and writes structured solution steps, checks, hints, and corrections back onto the same board.

The goal is not just to produce final answers. The goal is to make math help feel closer to a live tutoring session:

1. The student shows the problem.
2. The tutor sees the current work.
3. The student explains where they are stuck.
4. The tutor talks through the reasoning.
5. The tutor writes the next step on the board.
6. The student continues from there.

This kind of voice + canvas interaction can make AI learning products much more useful for students who need step-by-step guidance, not just answers.

Live Application

https://math-tutor.io

Recommended flow:

1. Open the application on desktop or tablet.
2. Upload a homework image or choose a sample equation.
3. Enter the tutoring board.
4. Click Start tutoring.
5. Ask the AI tutor to explain the problem or write the steps.

What It Does

• Uploads a homework photo and places it onto a tldraw whiteboard.
• Provides ready-made equation samples for quick testing.
• Starts a LiveKit-powered voice session between the student and AI tutor.
• Shows a live transcript beside the whiteboard.
• Sends structured canvas context and screenshot data to the agent.
• Lets the AI write formulas, hints, checks, and annotations on the canvas.
• Keeps AI annotations separate from student work so the board remains understandable.
• Focuses the tutoring loop around the student's actual homework, not a disconnected chat thread.

Why This Matters

Math tutoring is a large and persistent education need because homework support is time-sensitive. A student usually needs help while they are actively working, not days later. Human tutors provide high value, but the cost and availability limits are real.

AI Math Tutor is an attempt to lower that barrier. If an AI tutor can talk naturally, see the student's board, and write the explanation in the same space where the student is working, it can become a much more practical learning companion:

• Students get help when they are stuck.
• Parents get a more affordable homework support option.
• Tutors and teachers can use AI as a guided practice layer.
• Education products can move beyond static answer checking into interactive instruction.

The current implementation focuses on equation-solving workflows because they are a clear, visual, step-by-step tutoring scenario. The same architecture can later expand into word problems, geometry, functions, graphing, and broader STEM tutoring.

Product Flow

Homepage
  |
  | Upload homework image or choose a sample equation
  v
Tutor board
  |
  | Student writes, speaks, or asks for help
  v
LiveKit voice session
  |
  | Agent reads canvas context and screenshot
  v
AI reasoning
  |
  | Agent sends structured canvas actions
  v
tldraw whiteboard updates with steps, hints, and checks

Architecture

Student browser
  |
  | Next.js UI
  | tldraw whiteboard
  | Live transcript
  | Canvas RPC handlers
  v
apps/web
  |
  | LiveKit room
  | Voice tracks
  | RPC bridge
  v
LiveKit Cloud
  |
  | Agent joins the same room
  | STT -> LLM -> tools -> TTS
  v
apps/agent
  |
  | Gemini reasoning and visual understanding
  | Canvas reading and writing tools
  v
Spoken tutor response + whiteboard actions

The web app owns the visible classroom. The agent owns the tutoring behavior. LiveKit connects them through real-time audio, transcript events, and RPC calls that allow the agent to inspect and modify the canvas.

Repository Structure

ai-math-tutor/
  README.md
  LICENSE
  docs/
    images/
      screenshot-01-home-hero.png
      screenshot-02-ai-tutoring.png
      screenshot-03-math-samples.png
  apps/
    web/
      app/
        page.tsx
        tutor/page.tsx
        api/token/route.ts
      components/
        math-tutor/
          homework-upload-page.tsx
          math-tutor-workspace.tsx
          math-tutor-logo.tsx
          mobile-desktop-tip.tsx
        agents-ui/
        ai-elements/
        ui/
      lib/
        math-tutor/
          canvas-protocol.ts
          canvas-context.ts
          board-understanding.ts
          canvas-events.ts
          tutor-turn-state.ts
          tutor-action-layout.ts
          apply-tutor-actions.ts
          session-health.ts
      hooks/
      public/
      package.json
    agent/
      src/
        main.ts
        agent.ts
        tutor.ts
        pipeline.ts
        canvas_bridge.ts
        canvas_protocol.ts
        canvas_vision.ts
        canvas_solution.ts
        text_input.ts
        greeting.ts
        observability.ts
      livekit.toml
      Dockerfile
      package.json

Core Code Guide

apps/web/app/page.tsx

The public entry page. It renders the homework upload experience and sample equation picker.

apps/web/components/math-tutor/homework-upload-page.tsx

Handles the first step of the student journey:

• drag-and-drop homework upload
• clipboard paste
• image validation and optional cropping
• sample equation selection
• saving the selected homework image into session storage before opening the tutor board

apps/web/app/tutor/page.tsx

The route for the live tutoring classroom. It loads the main workspace component.

apps/web/components/math-tutor/math-tutor-workspace.tsx

The main classroom UI. This is one of the most important files in the project.

It connects:

• the tldraw whiteboard
• the LiveKit session
• the transcript panel
• the start/end call controls
• the text fallback input
• the canvas RPC methods used by the agent

It also loads uploaded homework images onto the board, keeps student work separate from AI annotations, and exposes the browser-side functions that let the agent read and write canvas state.

apps/web/lib/math-tutor/canvas-protocol.ts

Defines the shared structure for canvas data and AI actions. This is the contract between the browser and the agent.

Examples of supported actions:

• write_text
• write_formula
• hint_card
• mark_wrong
• mark_correct
• draw_arrow
• clear_ai_annotations

The protocol uses Zod validation so model outputs must be normalized before affecting the whiteboard.

apps/web/lib/math-tutor/canvas-context.ts

Prepares the canvas context that is sent to the agent. It keeps payload size under control while preserving the important board information.

apps/web/lib/math-tutor/board-understanding.ts

Turns raw tldraw shapes into a higher-level view of the board. It helps identify likely problem statements, student steps, and the active area of work.

apps/web/lib/math-tutor/tutor-action-layout.ts

Decides where AI-generated steps and hints should appear on the canvas. This is important because AI writing should not cover the student's homework or previous work.

apps/web/lib/math-tutor/apply-tutor-actions.ts

Converts structured AI actions into real tldraw shapes. For example, it can create text, render formulas, draw arrows, add hint cards, and clear previous AI annotations.

apps/web/lib/math-tutor/session-health.ts

Tracks whether the voice tutoring session is connected, stalled, failed, or waiting for the agent. This helps the UI explain what is happening instead of silently getting stuck.

apps/agent/src/main.ts

Starts the LiveKit Agent worker.

apps/agent/src/pipeline.ts

Builds the speech pipeline:

• speech-to-text
• Gemini reasoning
• text-to-speech

This is where the real-time voice tutor is assembled.

apps/agent/src/agent.ts

Defines the AI agent behavior and tool wiring. It connects the tutor instructions with the canvas tools.

apps/agent/src/tutor.ts

Contains the tutoring prompt and teaching behavior. This is where the project describes how the tutor should guide students instead of immediately dumping answers.

apps/agent/src/canvas_bridge.ts

The agent-side bridge to the browser canvas. It calls LiveKit RPC methods exposed by the web app to:

• read canvas context
• request screenshot chunks
• apply AI canvas actions
• clear AI annotations

apps/agent/src/canvas_vision.ts

Uses Gemini visual understanding to interpret screenshots of the canvas, especially when the student writes by hand or uploads a homework image.

apps/agent/src/canvas_solution.ts

Generates structured solution steps and canvas actions for math problems. It includes deterministic equation-solving paths for supported cases and Gemini-backed generation for broader inputs.

apps/agent/src/text_input.ts

Handles typed fallback questions from the UI. This keeps the tutor usable even if the user does not start voice immediately.

apps/agent/src/observability.ts

Adds logging and diagnostics for debugging voice sessions, canvas calls, and agent behavior.

Tech Stack

Frontend:

• Next.js 15
• React 19
• TypeScript
• tldraw
• Tailwind CSS
• shadcn-style UI components
• LiveKit client SDK

Agent:

• LiveKit Agents for Node.js
• Gemini for reasoning and visual understanding
• Deepgram STT through the LiveKit speech pipeline
• Cartesia TTS through the LiveKit speech pipeline
• Zod for structured protocol validation
• Vitest for unit tests

Deployment:

• Vercel for the web app
• LiveKit Agents deployment for the voice tutor

Key Capabilities

Homework Intake

Students can upload a homework photo, paste from the clipboard, or choose a sample equation. The image is placed onto the whiteboard as a homework card so the student and tutor can work around it.

Voice Tutoring

The student talks to the tutor through a LiveKit voice session. The transcript panel keeps the spoken exchange visible so the student can review what was said.

Canvas-Aware Agent

The browser exposes canvas RPC methods to the agent:

• read the current board context
• prepare a screenshot for visual understanding
• apply structured tutor actions
• clear AI annotations

This allows the tutor to respond to what is actually on the board, not just the latest chat message.

AI Whiteboard Actions

The agent can send structured commands such as:

• write text
• write formulas
• add hint cards
• mark correct or incorrect steps
• draw arrows
• clear previous AI annotations

The web app lays out these actions on the tldraw canvas while trying to avoid covering the student's work.

Requirements

• Node.js 24+ for the web app
• Node.js 22+ for the agent
• LiveKit project credentials
• Google API key for Gemini
• STT and TTS provider access through LiveKit-compatible model plugins
• tldraw license key for production deployments

Environment Variables

Copy the example files:

cp apps/web/.env.example apps/web/.env.local
cp apps/agent/.env.example apps/agent/.env.local

Frontend values:

LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
LIVEKIT_URL=
ALLOW_PUBLIC_TOKEN_ENDPOINT=false
AGENT_NAME=math-tutor
NEXT_PUBLIC_AGENT_NAME=math-tutor
NEXT_PUBLIC_TLDRAW_LICENSE_KEY=

Agent values:

LIVEKIT_URL=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
AGENT_NAME=math-tutor
GOOGLE_API_KEY=
GEMINI_LLM_MODEL=google/gemini-2.5-flash
GEMINI_CANVAS_VISION_MODEL=gemini-2.5-flash
MATH_TUTOR_STT_MODEL=deepgram/nova-3
MATH_TUTOR_TTS_MODEL=cartesia/sonic-3

Run Locally

Install and start the web app:

cd apps/web
npm install
npm run dev

The web app runs at:

http://localhost:3000

Install and start the agent:

cd apps/agent
npm install
npm run dev

Open the web app, choose a sample equation or upload homework, then start the tutoring session.

Test and Build

Web app:

cd apps/web
npm test
npm run build

Agent:

cd apps/agent
npm test
npm run build

Roadmap

• Broader math coverage beyond equation solving.
• Better OCR and math recognition for real homework photos.
• Student progress memory and misconception tracking.
• More robust Socratic tutoring policies.
• Teacher and parent review flows.
• Support for geometry, graphs, and multi-step word problems.

License

See LICENSE.