🤖 DAX AI Assistant – Handsfree

DAX AI Assistant is a voice-activated, hands-free system designed to enhance road safety and convenience for e-hailing and delivery drivers in Malaysia.

📂 Project Structures

📁 Folder	✨ Content
handsfree-be	Containing backend application & installation guides
handsfree-fe	Containing frontend application & installation guides
handsfree-eval	Containing jupyter notebook for machine learning model evaluation

📚 Project Resources

🔗 Resource	🌐 Link
🧠 Pitch Deck	View on Google Drive
📃 Documentation	View on Google Document
🎨 Figma Prototype	View on Figma

🧩 Key Functionalities

All functionalities are hands-free!

📦 Module	🛠️ Prime Function
📢 Trip Initiation	Accept / reject incoming orders Screen navigation Read order details (current / incoming) Get latest message updates from the passenger chat Send messages to passenger
🌐 In-Ride Communication	Translates conversations between driver and passenger on the fly via voice
📞 On Demand Communication	Initiates calls to customer Initiates calls to emergency contacts
🌧️ Navigation Support	Route clarification with a summary of current route Real-time updates on traffic conditions Immediate guidance in flood situation
🚨 Incident Response	Shaking-triggered SOS alert Continue / cancel SOS alert

🧑‍💻 User Interaction Scenario

• Drivers trigger actions via wake word and voice commands.

• Assistant reads aloud order details and messages.

• Multilingual translation happens in real time during conversation.

• Drivers call customers or trigger SOS alerts hands-free.

• Weather navigation support and general driving advice is voice-initiated.

🏗️ Solution Architecture

• Frontend: React Native + TypeScript

• Backend: Python + Flask

• Communication Protocol: Web Socket for real-time audio streaming and bidirectional communication

• AI Agent: SmolAgents + Gemini

• Noise Reduction: DeepFilterNet 3

• Speech Recognition:

  1. Mesolitica - Speech-to-Text (STT) model for Malaysian dialects
  2. Google Speech Recognition - For wake word detection

📊 Data Utilization

• No custom training — using pre-trained machine learning models

• Mesolitica for STT in Manglish and Malay

• Gemini for accurate, low-hallucination multilingual translation

• DeepFilterNet 3 for noise reduction

🎯 Personalization Strategies

🗣️ Language Customization: Supports Malaysian English, Bahasa, Mandarin, Tamil

👨‍👩‍👧‍👦 Emergency Contact Setup: Voice or manual addition for SOS feature

🔇 HEX AI Toggle: Enable or disable voice assistant with a button