Dudley

A portable, bilingual voice assistant built around ESP32-S3, local speech models, and a Python control server.

About the project

Dudley began as my Electrical and Electronics Engineering graduation project: a voice assistant that should feel like a physical product rather than a microphone attached to a chatbot. The system combines a battery-powered ESP32-S3 device, a local Python speech pipeline, a browser-based control interface, and optional service integrations.

The device captures speech over I2S and streams it to the local server over Wi-Fi. The server detects active speech and the wake word, transcribes the request, routes direct commands to the appropriate integration, and uses an OpenRouter model for conversational requests. The reply is synthesized locally and played through the device.

Highlights

• English and Turkish speech recognition with automatic language handling
• Local voice activity detection, wake-word detection, speech recognition, and speech synthesis
• ESP32-S3 audio streaming and device control over WebSocket
• Direct commands for Spotify, YouTube, browser navigation, Gmail, Google Calendar, news, and web search
• Browser interface for conversation, status, voices, and device controls
• Portable hardware with a 24-pixel status ring, onboard speaker, and rechargeable battery
• Reproducible one-command setup with a project-local Python 3.11 runtime

How it works

1. The INMP441 microphone sends raw I2S audio to the ESP32-S3.
2. Firmware streams audio to the local server through a WebSocket connection.
3. Silero VAD removes silence and the Dudley wake-word model decides when to listen.
4. Faster-Whisper transcribes active speech and detects the spoken language.
5. The command router handles supported actions locally or sends a conversational query through OpenRouter.
6. Kokoro produces English speech; Coqui XTTS v2 can optionally provide Turkish speech.
7. The generated audio is returned to the ESP32-S3 and played through the MAX98357A amplifier.

Layer	Main components	Responsibility
Device	ESP32-S3, INMP441, MAX98357A, WS2812	Audio capture/playback, LED feedback, Wi-Fi transport
Local server	Flask, Socket.IO, Silero VAD, OpenWakeWord, Faster-Whisper	Real-time audio processing and command routing
Speech output	Kokoro 82M, optional Coqui XTTS v2	Local English and Turkish speech synthesis
Reasoning	OpenRouter	Conversational responses and intent assistance
Interfaces	Web UI, Android companion shell	Conversation and device management

Hardware

Prototype electronics	Parametric enclosure layout

Part	Role
ESP32-S3 development board	Main controller, Wi-Fi link, and I2S processing
INMP441	Digital MEMS microphone
MAX98357A	I2S class-D audio amplifier
8 Ω / 5 W speaker	Voice output
24-pixel WS2812 ring	Visual state and activity feedback
TP4056	Li-ion charging and protection
MT3608	Adjustable DC boost converter
18650 Li-ion cell	Portable power source used in the prototype

The editable enclosure model is available as dudley-enclosure.scad. Change render_part near the end of the file to render an individual section, the assembled enclosure, an exploded view, or the print layout.

The complete prototype wiring reference is available in the circuit schematic. Take normal Li-ion safety precautions and verify polarity, charging current, insulation, and converter output before connecting the controller.

Interfaces

Current browser interface	Thesis mobile prototype	Thesis device controls

The screenshots on the right document the Android controller developed for the thesis prototype. The public android/ directory currently contains the maintained companion-app shell; the server-side device registration and control APIs are included in this repository.

Integrations

Integration	What Dudley can do	Credentials
OpenRouter	Conversational answers and intent handling	Required
Spotify	Search, playback, queue, library, and volume control	Optional
YouTube	Search, playback, navigation, and video controls	Optional
Browser automation	Open sites, search, navigate tabs, scroll, and zoom	Optional
Gmail	List and read recent messages	Optional Google OAuth
Google Calendar	Read, create, search, and delete events	Optional Google OAuth
News and web search	RSS, GNews, Tavily, and DuckDuckGo-backed queries	Optional keys for enhanced providers

A broader command reference is available in docs/COMMANDS.md.

Quick start

The launcher keeps the runtime inside the project directory. It downloads uv, installs a managed Python 3.11 runtime, synchronizes locked dependencies, prepares the speech models, and opens the local interface. The first run requires an internet connection and takes longer because models and packages are downloaded.

git clone https://github.com/muhammedsaban/dudley.git
cd dudley

Windows

1. Download or clone this repository.

2. Double-click start.bat, or run it from Command Prompt:

   start.bat

3. Enter your OpenRouter API key when the setup wizard asks for it.

4. Dudley starts locally and opens the browser interface.

macOS and Linux

chmod +x start.sh
./start.sh

The generated runtime, models, and secrets stay under ignored .runtime/ and backend/.local/ directories. They are installation data, not repository files, and can be removed safely when Dudley is not running; the launcher will recreate them on the next start.

Configuration

Only an OpenRouter API key is required for conversational use. The setup wizard writes configuration to backend/.local/secrets/config.env.

Variable	Purpose	Required
OPENROUTER_API_KEY	Access to the configured language model	Yes
OPENROUTER_MODEL	OpenRouter model ID; defaults to google/gemini-2.5-flash-lite	No
SPOTIFY_CLIENT_ID, SPOTIFY_CLIENT_SECRET	Spotify control	No
GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET	Gmail and Calendar OAuth	No
TAVILY_API_KEY	Tavily web search	No
GNEWS_API_KEY	GNews results	No
YOUTUBE_API_KEY	YouTube Data API search	No
DUDLEY_ENABLE_COQUI=1	Enable optional Coqui XTTS v2 support	No

See backend/.env.example for every supported setting. Never commit backend/.local/ or a populated firmware secrets file.

ESP32-S3 firmware

The firmware is a PlatformIO project in firmware/esp32/.

1. Install PlatformIO in VS Code or as a CLI tool.

2. Copy firmware/esp32/include/secrets.example.h to firmware/esp32/include/secrets.h if you want to compile Wi-Fi settings into the device.

3. Set the Wi-Fi and local-server values in secrets.h.

4. Connect the ESP32-S3 and upload:

   cd firmware/esp32
   pio run --target upload
   pio device monitor

secrets.h is ignored by Git. The firmware also includes fallback values, so the project remains buildable without committing private network details.

Repository layout

.
├── backend/              Python server, web UI, audio pipeline, and integrations
├── firmware/esp32/       PlatformIO firmware for the ESP32-S3 device
├── hardware/enclosure/   Parametric OpenSCAD enclosure source
├── android/              Android companion-app shell
├── docs/                 Command and integration documentation
├── images/               README and project documentation assets
├── start.bat             Windows launcher
├── start.ps1             Windows setup and runtime script
└── start.sh              macOS/Linux launcher

Prototype evaluation

These results were recorded for the completed thesis prototype. They are useful as a reference, not as universal benchmarks: latency depends on the selected models, network connection, operating system, and host hardware. The thesis test system used an NVIDIA RTX 3050 6 GB GPU and a Whisper Large V3 Turbo configuration; the public launcher selects practical Faster-Whisper defaults for the detected machine.

Measurement	Result
Silero VAD, 1 second of audio	11.2 ms mean
Wake-word inference, 80 ms chunks	9.40 ms mean / 13.10 ms p95
Quiet-room wake-word detection	94%
Medium-noise wake-word detection	87%
High-noise wake-word detection	74%
Turkish / English language detection	95% / 95%
Kokoro synthesis	0.31 s short / 0.72 s long response
Coqui XTTS v2 synthesis	2.41 s short / 5.73 s long response
End-to-end request time	1.52 s best / 4.24 s mean / 6.96 s worst
Estimated battery life	3.8–9.6 hours, depending on workload

Current limitations

• Conversational requests require an internet connection and an OpenRouter account.
• Initial setup downloads large speech and language-model assets.
• CPU-only speech recognition is slower than the GPU-backed thesis test system.
• A single microphone performs best in relatively quiet or medium-noise rooms.
• The complete mobile controller shown above is preserved as thesis prototype work; the public Android source is currently a companion shell.
• Third-party integrations may require their own developer applications, OAuth consent, or subscriptions.

Roadmap

• Complete and document the Android controller flow
• Add guided device discovery and Wi-Fi provisioning
• Improve far-field capture with a microphone array and acoustic echo cancellation
• Expand offline intent handling and reduce cloud dependence
• Add reproducible hardware fabrication files and assembly instructions

Academic context

Dudley was developed by Muhammed Saban as a 2026 undergraduate graduation project in Electrical and Electronics Engineering at Konya Technical University. The project name is a reference to Homer Dudley, whose work helped shape early electronic speech systems.

License

This project is released under the MIT License.