Medyt

Talk to your gallery. It listens, plans, and acts.

🎤 “Organize my gallery into albums”  ·  🎤 “Find every photo of my mother wearing red”  ·  🎤 “Get rid of the old screenshots and blurry stuff”

🎤 “…everything from the beach into one album.” Spoken → planned → moved, hands-free.

Talk to it in chat

Albums it built for you

Tags + description per photo

_{More real device captures below · assets/screens/live-views/}

---

💥 The problem

Everyone's gallery is a graveyard: thousands of photos, hundreds of screenshots you'll never open again, blurry duplicates, and no way to find "that photo of mom in the red shirt at the beach" without scrolling for ten minutes. Organizing it manually is exactly the kind of tedious, judgment-heavy work nobody ever does.

💡 The idea

Medyt is not a chatbot describing your photos. It's an agent that plans and executes multi-step actions: it scans your gallery, analyzes what it sees, proposes a folder taxonomy, moves files, recognizes the people you've named, remembers your standing preferences ("never touch photos of my mother"), stages deletion candidates for your review — and asks for confirmation, by voice, before anything destructive. Nothing is ever hard-deleted; everything is undoable.

🔒 Privacy by architecture, not by promise

The agent loop, the planner, all tool execution, and the entire database live on the phone — in every mode:

	🔒 Privacy mode (default)	⚡ Speed mode (opt-in)
Planner	Gemma 4 on-device	Gemma 4 on-device
Image/video analysis	Gemma 4 on-device	Nemotron via stateless Worker — processed in memory, never stored
Faces	on-device only	on-device only — always
Works in airplane mode	✈️ yes	falls back to Privacy
Bytes leaving the phone	zero	pixels only, by your choice

“Your photos may visit the cloud if you choose speed.
Your family's faces never do.”

🧠 How the agent works

The planner is Gemma 4 on-device, using Gemma 4's native function-calling format. Each turn it emits exactly one action — a tool call, a question to the user, or a finish summary — and Kotlin executes it. Safety is enforced in code, not in prompts: staging deletions requires confirmation, protections are subtracted automatically, and every mutation is logged for undo.

Full current-behavior guide: docs/09-agent-behavior.md — voice flow, find-vs-analyze policy, tools, and code map.

Voice → chat → agent

Step	What happens
Bottom-bar mic	New chat thread → Chat screen → push-to-send STT → agent runs with spoken reply
Chat mic	Same flow on the current thread
STT	On-device SpeechRecognizer (push-to-send; pauses don't end capture)
TTS	Gradium when Voice=Cloud / Auto+Speed; else platform TTS

Find vs analyze

You ask	Agent should
"Find cooking photos and move to Cooking"	organize_media on already-analyzed tags — no vision run
"Analyze my gallery"	analyze_batch — background vision queue
"Find beach photos" (never analyzed)	Report 0 matches — not auto-analyze

Kotlin blocks analyze_batch unless your words explicitly request analysis (IntentPolicies). Tag/topic searches default to status=ANALYZED.

Where does the image/video analysis run? On-device by default. When you analyze a photo or video — a single item or the whole gallery — Gemma 4 does the vision right on the phone (Privacy mode): nothing leaves the device and it works in airplane mode. Cloud vision (Nemotron on Nebius, via the stateless Worker) is used only if you opt into Speed mode, and even then only pixels are sent and nothing is stored. Faces are analyzed on-device, always.

▶️ Example run — “find all my cooking photos and put them in a Cooking album”

organize_media(folder="Cooking", filter={tags_any:["cooking"], status:ANALYZED})  → 12 moved
finish → 🔊 "I moved 12 cooking photos into Cooking."

Uses tags from a prior analyze pass. No analyze_batch unless you also said “analyze”.

▶️ Example run — “find the photos of my mother wearing red and put them in an album”

find_person("mother")                                          → person_id 3 · 212 photos
scan_gallery(person_id=3, clothing=[{item:shirt, color:red}])  → 18 matches
create_folder("Mom in Red")                                    → category 14
move_media([...18 ids], 14)                                    → done · undoable (action_log #91)
finish → 🔊 "I found 18 photos of your mother wearing red and put them in Mom in Red."

No destructive step → no confirmation needed. A deletion flow inserts a voice-confirmed ask_user — and the stage_for_deletion tool rejects the call in Kotlin if it didn't happen.

📱 Inside the app

Real captures from a physical device — not mockups.

Gallery	Albums	Agent chat	People (on-device faces)
Analyze · Speed (cloud)	Analyze · Privacy (on-device)	Review (never hard-deletes)	Media detail · tags

_{Full set (settings, filters, logs, chat history): assets/screens/live-views/ · design mockups: assets/screens/}

🛠️ Technology

Layer	Technology	Role
📱 App	Kotlin + Jetpack Compose	Gallery UI, agent loop, tool executor, Room DB
🧠 On-device AI	Gemma 4 edge (QAT) via LiteRT-LM · GPU + MTP speculative decoding	Planner (every mode) + local vision (Privacy mode)
☁️ Cloud perception	NVIDIA Nemotron 3 Nano Omni on Nebius (primary) · NVIDIA NIM (fallback) · Nemotron 3 Ultra 550B for complex-plan escalation	Speed-mode image & video analysis
🔀 Backend	Cloudflare Workers	Stateless proxy: hides keys, fans out keyframes, rate-limits — stores nothing
🎙️ Voice	On-device STT (push-to-send) · Gradium TTS (Cloud/Auto+Speed) · platform TTS (Local)	Mic → chat → agent; see 09-agent-behavior.md
👤 Faces	ML Kit + MobileFaceNet embeddings (LiteRT) · on-device clustering	"Find photos of my mother" — local-only
🗄️ Database	Room (SQLite)	Media index · categories · people · preference memory · action log
⌨️ Dev tooling	Cursor + project skills in .cursor/skills/	Entire codebase built in Cursor during the event

🤖 What makes it an agent (and not an image analyzer)

1. 🧩 Multi-step planning — one voice intent becomes a chain of tool calls with reasoning over intermediate results
2. 🧠 Preference memory — "never suggest deleting photos of my mother" is saved once, enforced in Kotlin on every future deletion pass
3. ⚖️ Two-signal deletion evidence — items are staged only when independent signals agree (e.g. blur and low keep-score), minus protections, plus explicit confirmation
4. 👥 People as first-class context — named face clusters are injected into the planner's context, so natural references resolve to database queries
5. ↩️ Everything is reversible — action log + undo tool + a Review-for-Deletion folder instead of deletion

📁 Repository guide

Path	Contents
agent.md	🤖 Executor entry point — build order, rules, map
docs/00…09-*.md	📚 Overview → plan → design → agent flows → schemas/spec → tasks → setup → agent behavior (current)
docs/09-agent-behavior.md	🤖 How the agent works today — voice, find-vs-analyze, tools, code map
docs/07-setup-guide.md	🛠️ Keys, phone, models, troubleshooting — start here to run the app
docs/knowledge/	🧾 OKF knowledge base for every external technology (live-validated API details)
.cursor/skills/	⚡ Cursor skills: project conventions + per-technology guides
app/	📱 Android application (built during the event)
local.properties.example	🔑 Template for local.properties (Worker URL + app key; gitignored)
worker/	☁️ Cloudflare Worker (built during the event)

🚀 Running it

Full step-by-step guide: docs/07-setup-guide.md (keys, Android Studio, phone, models, troubleshooting).

Short version:

1. Open the repo root in Android Studio (not app/ alone) → Run on a physical device (8 GB+ RAM).
2. Copy local.properties.example → local.properties and set sdk.dir (Android Studio may create this for you).
3. Privacy / offline demo: no API keys. Side-load the on-device model (see below), tap Initialize.
4. Speed mode / online voice (optional): deploy the Worker, set WORKER_BASE_URL + MEDYT_APP_KEY in local.properties (same token in Worker secrets — you generate it with openssl rand -hex 32).

# Worker (optional)
cd worker && npm install && npm test
cp .dev.vars.example .dev.vars    # or: npx wrangler secret put …
npx wrangler deploy

# App tests
./gradlew :app:testDebugUnitTest

# Side-load models (required for on-device Gemma — not bundled in the APK)
adb push gemma-e4b.litertlm /data/local/tmp/
adb push mobilefacenet.tflite /data/local/tmp/   # optional, for face tools

🔒 Privacy mode needs no accounts, no keys, no internet. That's the point.

📦 On-device models — not in the APK (and not downloadable in-app yet)

Gemma 4 edge QAT is several GB and license-gated, so it is not bundled in the APK. Download on a dev machine (Hugging Face + HF_TOKEN or huggingface-cli login) and adb push to the phone — see the setup guide.

No in-app downloader yet (HF auth + progress/resume cut for time). After push, open Settings → Initialize and wait for the green Gemma ready badge (smoke test). Privacy analysis runs one image at a time for LiteRT GPU stability.

🏆 Hackathon compliance

• ✅ Built entirely during RAISE 2026 (July 4–5) — see commit history
• 🌍 Team: remote track
• 🙏 Acknowledgments: .cursor/skills/gemma-dev adapted from google-gemma/gemma-skills (Apache-2.0)
• 🤝 Sponsor platforms used: Cursor · Google Gemma/LiteRT · Nebius · NVIDIA · Cloudflare · Gradium

🔜 What's next

The hackathon build is demoable — watch the ▶ 1-minute demo on Vimeo, or browse live captures in assets/screens/live-views/ (gallery, albums, chat, device settings, review, media detail, logs) and the raw device recordings in assets/videos/ (device analyze, batch analyze, agent organize). What follows is the product gap list, ordered by what unlocks a trustworthy daily driver.

1. Smarter agent — tools, reasoning, follow-through

The on-device Gemma planner is the bottleneck. Today it often stops after one tool (scan without move), mis-picks tools, or fails to chain steps. Kotlin guards (IntentPolicies, auto-organize_media) paper over that — the planner itself needs to get better.

Need	Why
Richer tool set	More actions the user can ask for in one intent: scheduled rules, preference CRUD, date/context filters, duplicate hints, album maintenance
Reliable multi-step plans	Find → move, scan → stage → ask → delete must complete without hand-tuned Kotlin shortcuts
Chat context in planning	Each run still gets a single intent; thread history is UI-only. Feed last N turns (or a summary) into the planner
Stronger reasoning over metadata	Use tags, dates, keep-score, person_id, and album state — not re-run vision for every “find” request
Graceful degradation	When Gemma can't emit valid JSON, fall back to deterministic intents for common phrases instead of opaque failures

2. Memory & preferences — remember, enforce, show

save_preference / forget_preference exist and protections feed KNOWN CONTEXT, but the loop is thin.

Need	Why
Preference UI	View, edit, and delete rules in Settings — not only via voice
Proactive enforcement	"Never delete photos of Mom", "Always put cooking in Cooking" should apply on every scan, staging pass, and scheduled job without re-stating
Preference kinds in practice	protection, deletion_rule, organization, taste are specced — need clearer examples, validation, and conflict handling
Agent-initiated memory	When the user states a lasting rule mid-task, save it before continuing (planner rule exists; reliability doesn't)

3. Scheduled & contextual cleanup

Daily gallery scan exists (Settings → Device). Scheduled deletion / review does not.

Need	Why
Deletion schedules	e.g. "Every Sunday, review screenshots older than 90 days" or "Monthly pass on blurry + low keep-score"
Context-aware candidates	Combine signals: media_kind, tags, taken_after/before, age, blur, keep-score, duplicates — minus active protections
Background staging → Review	Auto-stage matches into Review for Deletion; user approves in the Review screen (never hard-delete silently)
Schedule + voice	"Remind me to clean old screenshots" should create a persisted rule the agent can edit or disable

4. App polish (still open)

Item	Notes
Chat persistence	Room stores thread title/preview only — not full message history
Physical MediaStore albums	Categories are Room-only today (AlbumWriter.NoOp)
Filter modal	People picker, time/quality chips, live result counts
In-app Gemma download	Replace adb push with HF auth + resumable download
MediaStore delete consent	Permanent deletes on API 30+ need the system permission dialog

Shipped in this repo (Jul 5)

Voice push-to-send STT · new chat per bottom-bar mic · find/move on ANALYZED metadata · organize_media + Kotlin move guards · Privacy analyze one-at-a-time · Review select/delete/keep · scan pause/resume · daily scan · people face crops · docs/09-agent-behavior.md

Trackers: docs/todo.md · docs/knownissues.md · setup: docs/07-setup-guide.md

---

Apache-2.0 · Made with 🎙️ + ☕ at RAISE Summit Hackathon 2026