Skip to content

Commit 11291c3

Browse files
tensorboyclaude
tensorboy
and
claude
committed
feat: add WebGazer eye tracking with global click calibration
- Integrate WebGazer.js for implicit eye gaze tracking calibration - Add global mouse click capture using uiohook-napi for app-external clicks - Implement calibration sample visualization with face snapshots - Add activity summary, audio processor, and gesture control services - Improve Life Tree visualization and UI components - Add camera preview and timeline tree components - Various UI/UX improvements and bug fixes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 23ea1d2 commit 11291c3

62 files changed

Lines changed: 9954 additions & 244 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,107 @@ Select "Ollama" in Hawkeye settings.
227227
└───────────────┴───────────────┴───────────────┴─────────────────┘
228228
```
229229

230+
### 🔮 Future: Multi-Modal HCI Pipeline
231+
232+
Hawkeye is evolving into a full multi-modal human-computer interaction system that combines **audio understanding**, **visual perception**, and **gesture control**.
233+
234+
```
235+
┌─────────────────────────────────────────────────────────────────────────────┐
236+
│ HAWKEYE MULTI-MODAL HCI PIPELINE │
237+
├─────────────────────────────────────────────────────────────────────────────┤
238+
│ │
239+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
240+
│ │ INPUT LAYER │ │
241+
│ ├─────────────────────────────────────────────────────────────────────┤ │
242+
│ │ 📷 Camera ────▶ MediaPipe Holistic │ │
243+
│ │ • Face: 468 landmarks │ │
244+
│ │ • Pose: 33 keypoints │ │
245+
│ │ • Hands: 21 × 2 keypoints │ │
246+
│ │ │ │
247+
│ │ 🎙️ Microphone ─▶ Silero VAD ─▶ Audio Buffer │ │
248+
│ └─────────────────────────────────────────────────────────────────────┘ │
249+
│ │ │ │
250+
│ ▼ ▼ │
251+
│ ┌──────────────────────────────┐ ┌──────────────────────────────────┐ │
252+
│ │ VISUAL PROCESSING │ │ AUDIO PROCESSING │ │
253+
│ ├──────────────────────────────┤ ├──────────────────────────────────┤ │
254+
│ │ Face Tracker │ │ DiariZen / Pyannote │ │
255+
│ │ ├─ Multi-face detection │ │ ├─ Speaker diarization │ │
256+
│ │ ├─ Face ID assignment │ │ ├─ "Who is speaking?" │ │
257+
│ │ └─ Lip movement analysis │ │ └─ Speaker embeddings │ │
258+
│ │ │ │ │ │
259+
│ │ Gesture Recognizer │ │ Whisper (smart-whisper) │ │
260+
│ │ ├─ Hand pose classification │ │ ├─ Speech-to-text │ │
261+
│ │ ├─ Dynamic gesture detect │ │ ├─ Language detection │ │
262+
│ │ └─ Custom gesture mapping │ │ └─ Timestamp alignment │ │
263+
│ └──────────────────────────────┘ └──────────────────────────────────┘ │
264+
│ │ │ │
265+
│ ▼ ▼ │
266+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
267+
│ │ FUSION & MATCHING LAYER │ │
268+
│ ├─────────────────────────────────────────────────────────────────────┤ │
269+
│ │ │ │
270+
│ │ Audio-Visual Matching │ │
271+
│ │ ├─ Lip-sync correlation (who's lips match the audio?) │ │
272+
│ │ ├─ Face-voice association (learn speaker identity) │ │
273+
│ │ └─ Active speaker detection (LoCoNet / AS-Net) │ │
274+
│ │ │ │
275+
│ │ Context Aggregation │ │
276+
│ │ ├─ Combine: transcription + speaker ID + face ID + gesture │ │
277+
│ │ └─ Generate unified interaction events │ │
278+
│ │ │ │
279+
│ └─────────────────────────────────────────────────────────────────────┘ │
280+
│ │ │
281+
│ ▼ │
282+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
283+
│ │ ACTION EXECUTION │ │
284+
│ ├─────────────────────────────────────────────────────────────────────┤ │
285+
│ │ │ │
286+
│ │ Gesture → Command Mapping │ │
287+
│ │ ├─ 👍 Thumbs Up → Confirm action │ │
288+
│ │ ├─ ✋ Open Palm → Pause / Stop │ │
289+
│ │ ├─ 👆 Point Up → Scroll up │ │
290+
│ │ ├─ 👇 Point Down → Scroll down │ │
291+
│ │ ├─ ✌️ Victory → Screenshot │ │
292+
│ │ ├─ 🤏 Pinch → Zoom in/out │ │
293+
│ │ └─ 🖐️ Swipe → Switch window / tab │ │
294+
│ │ │ │
295+
│ │ Voice Command + Gesture = Enhanced Control │ │
296+
│ │ └─ "Open browser" + Point → Open browser at pointed location │ │
297+
│ │ │ │
298+
│ └─────────────────────────────────────────────────────────────────────┘ │
299+
│ │ │
300+
│ ▼ │
301+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
302+
│ │ OUTPUT │ │
303+
│ ├─────────────────────────────────────────────────────────────────────┤ │
304+
│ │ │ │
305+
│ │ 📝 Attributed Transcription │ │
306+
│ │ "Alice: Let's review the code changes" │ │
307+
│ │ "Bob: I'll share my screen [👆 pointing at screen]" │ │
308+
│ │ │ │
309+
│ │ 🎮 System Control │ │
310+
│ │ Mouse movement, clicks, keyboard shortcuts, app switching │ │
311+
│ │ │ │
312+
│ │ 🌳 Life Tree Update │ │
313+
│ │ Activity tracking, goal inference, habit analysis │ │
314+
│ │ │ │
315+
│ └─────────────────────────────────────────────────────────────────────┘ │
316+
│ │
317+
└─────────────────────────────────────────────────────────────────────────────┘
318+
```
319+
320+
**Key Technologies:**
321+
| Component | Technology | Status |
322+
|-----------|------------|--------|
323+
| Voice Activity Detection | Silero VAD | ✅ Planned |
324+
| Speech-to-Text | Whisper (smart-whisper) | ✅ Implemented |
325+
| Speaker Diarization | DiariZen / Pyannote | 🔄 Research |
326+
| Active Speaker Detection | LoCoNet (CVPR 2024) | 🔄 Research |
327+
| Body Tracking | MediaPipe Holistic | ✅ Planned |
328+
| Gesture Recognition | MediaPipe Gesture | ✅ Planned |
329+
| Face-Voice Matching | Custom Fusion | 🔄 Research |
330+
230331
<br/>
231332

232333
## 📦 Project Structure

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
},
2222
"devDependencies": {
2323
"@types/node": "^20.10.0",
24+
"electron": "^40.1.0",
2425
"typescript": "^5.3.3"
2526
},
2627
"packageManager": "pnpm@10.16.1",

‎packages/core/src/ai/types.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ export type IntentType =
108108
| 'communication' // 沟通/写作
109109
| 'data_process' // 数据处理
110110
| 'system_config' // 系统配置
111+
| 'voice_command' // 语音指令
111112
| 'unknown'; // 未知
112113

113114
export interface IntentEntity {

0 commit comments

Comments
 (0)