Cadence Labs

a bridge between human ingenuity, safety, and convenience.

A VR-teleoperated bimanual robotic arm system that lets an operator work in environments a human body shouldn't enter — sustained lunar presence, fire response, post-disaster reconnaissance, chemical labs, nuclear decommissioning, high-radiation zones. The operator wears a Meta Quest 3S at a safe desk; their hand motion becomes the robot's hand motion in real time. Demonstrations recorded this way train an ACT imitation-learning policy that handles repetitive sub-tasks autonomously, so the operator only makes judgment calls. An independent ESP32-S3 safety sentinel is the local second check that vetoes unsafe motion before the operator's command reaches the motor.

The "one arm holds the pan, the other stirs" cooking demonstration is one example of the pattern — but the product is the bridge, not the recipe.

Built for StarkHacks 2026 @ Purdue University.

Team Information

Selin Ozsaracoglu — preliminary Meta Quest hand-tracking algorithm
Aaryan Lath — operator dashboard / UI, hand-tracking enhancements (3D visualization, dual-hand support, sentinel integration), multi-camera YOLO fusion, ESP32 firmware design
Afia Ava — bimanual arm calibration, teleoperated episode recording, LeRobot dataset publishing, ACT training pipeline
Godfred Obrigya-Acquah — Mechanical and Electrical Engineer (CAD, L298N motor driver integration, sensor harness, power routing)

Items

LeRobot SO-100 arm (2×, bimanual follower pair)
Meta Quest 3S
AMD camera (2×)
Logitech camera (1×)
Ring light
Lamp light

Approach

The approach is environment-variable: the same arm architecture and training recipe applies to any real-world scenario. Rather than relying on pre-labelled object positions or prescripted trajectories, the system learns entirely from Meta Quest hand-tracking demonstrations via an ACT (Action Chunking Transformer) policy. Training from the operator's natural hand motion (vs. a leader-arm rig) captures all finger articulation the arm needs to reproduce — one-take teleoperation becomes one-take dataset recording.

Technical Implementation


Episode Count	65
Number of Steps	58,435 frames @ 30 fps (~32 min of teleoperation)
Input	Meta Quest hand tracking (21 joint landmarks + 6DOF wrist pose) + top + side cameras (640×480 @ 30 fps, AV1)
Action space	12-DOF: 6 joints × 2 arms (`left_/right_ × shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper`)
Policy	ACT (Action Chunking Transformer) on `bi_so100_follower`
Dataset	`afiaava/utensil_pickup_stir` on Hugging Face — top + side camera videos previewable in the Data Studio

Ease of Use

The operator wears the Meta Quest and performs the target task naturally. The Quest streams hand-pose data into the LeRobot recording pipeline, which captures synchronized joint states and camera frames. Once a policy is trained on MI300X, it runs fully autonomously on the Ryzen AI MiniPC with on-device inference.

How the pieces fit together

Teleop (Aaryan's path): Meta Quest 3S hand-tracking-streamer → UDP to Mac → calibration engine (Logical Home Pairing with per-hand anchor) → ikpy IK solver → joint angles. Today these print to console; next iteration closes the loop onto the physical SO-100 over USB serial.

Bimanual demonstration recording (Afia's path): the bi_so100_follower pair is calibrated and driven through the LeRobot v0.4.1 recording pipeline with two fixed cameras (top + side). 65 episodes of the utensil_pickup_stir task — one arm stabilizes the pan, the other handles the utensil — were captured at 30 fps, producing 58,435 synchronized frames of joint states + RGB video, published as afiaava/utensil_pickup_stir on Hugging Face. Each episode's top + side videos are previewable directly in the HF Data Studio viewer.

Where it's going — symphony: the two paths merge in the next iteration. The Quest hand stream becomes an alternative recording input (vs. leader-arm teleop) so new episodes can be captured faster, with richer finger articulation than a leader rig can capture. Afia's existing 65-episode dataset is the training seed — an ACT policy fine-tuned on MI300X using it becomes the autonomous "Rover mode" behavior. The ESP32 sentinel is the independent hazard veto that makes any of this safe to deploy remotely.

What the dashboard does

A Flask-based operator dashboard renders the whole thing — a live 3D hand skeleton, multi-camera object recognition for Rover mode, and the safety-sentinel panel — fullscreen inside the Quest's built-in browser.

Architecture

Meta Quest 3S
    │
    │ UDP :9000  (hand-tracking-streamer app)
    ▼
Desktop (macOS, Python)
    ├─ mini_ng_hand_listener.py     UDP receiver + packet parser
    ├─ calibration_engine.py        Logical Home Pairing (anchor + scale)
    ├─ run_pipeline.py              CLI: IK solver (ikpy) + joint-angle printout
    └─ mini_ng_dashboard.py         Flask operator UI (Teleop / Rover tabs)
                                      + multi-camera MJPEG + /api/hand
                                      + YOLOv8n fused detection across USB cams
                                      + record-to-MP4+JSONL for the demo video
    │
    ▼
SO-101 follower arms  (future: drive joints over Robot_Bus)

Hardware

Meta Quest 3S headset
2× SO-101 robot arms (5 revolute joints + gripper)
2× USB-to-serial cables
Multiple USB webcams (1× overhead + 3× fused detection cams for Rover mode)
macOS development machine on the same Wi-Fi network as the headset

Setup

1. Set up the Python environment

cd quest
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install ikpy ultralytics

2a. Run the CLI pipeline (IK dry run, no UI)

python run_pipeline.py

On the headset:

Launch the hand-tracking-streamer app and tap start stream.
Hold your right hand at a comfortable rest pose.
Type a + Enter in the terminal to anchor, then move your hand.

2b. Run the operator dashboard (demo UI, Teleop + Rover tabs)

python mini_ng_dashboard.py
# → dashboard: http://0.0.0.0:5050/

Open http://<mac-ip>:5050/ in the Quest's built-in browser, go fullscreen, and start the hand-tracking-streamer. The Teleop tab shows the overhead cam, a live 3D wireframe of the tracked hand, and a status ticker. The Rover tab shows the multi-camera YOLO detection montage with cross-camera confirmation, the on-board safety-sentinel panel, and the pitch copy about sustained remote presence.

Important: run_pipeline.py and mini_ng_dashboard.py both bind UDP :9000 — run only one at a time.

2c. Double-click to launch (macOS)

Start Demo.app at the project root is an AppleScript bundle — double-click from Finder and it kills any stale dashboard, launches a fresh one in the background, opens Safari, and launches the Quest streamer over USB via adb. Stop Demo.app tears it down.

Try the hand tracking on your own Quest

If you have a Meta Quest 3 / Quest 3S and a Mac (or Linux/Windows box with Python 3.11+), you can run the Pilot tab yourself end-to-end. You will not need a physical robot arm — this shows the live 3D hand skeleton, multi-camera object recognition, and the safety-sentinel panel all in the browser.

What you need

Meta Quest 3 or 3S (older Quests may work if hand tracking is enabled)
USB-C data cable Quest ↔ Mac (data cable, not charge-only)
Mac with macOS 13+, Python 3.11+, ffmpeg/brew optional
Wi-Fi network both devices can join with peer-to-peer allowed (venue Wi-Fi like eduroam or campus Wi-Fi usually blocks peer UDP — use a personal hotspot if your only Wi-Fi is enterprise)

1. Enable developer mode on the Quest (one-time)

Sign up as a Meta developer at developer.oculus.com and create an "organization" (can be a name made up on the spot).
On your phone, in the Meta Horizon app → Menu → Devices → pick your headset → Headset settings → Developer mode → On.
Reboot the Quest.

2. Sideload `hand-tracking-streamer` onto the Quest

This is the Unity app that captures the 21-landmark hand skeleton and broadcasts it over UDP. Source: wengmister/hand-tracking-streamer.

Easiest path (SideQuest GUI): install SideQuest on your Mac, plug the Quest in via USB-C, tap Allow USB debugging inside the headset, download the latest hand-tracking-streamer.apk from the project's GitHub Releases page, drag-and-drop it into SideQuest's "Install APK file" button.
Scripted path (adb): if you already have android-platform-tools via Homebrew (brew install --cask android-platform-tools), run:
```
adb devices              # should list your Quest as "device"
adb install hand-tracking-streamer.apk
```

3. Set the destination IP inside the streamer app

Find your Mac's current IP on the Wi-Fi:

ipconfig getifaddr en0        # e.g. 10.10.10.147

Put the headset on, launch the streamer from the Quest app library (Unknown Sources). On its main screen:

Set Protocol = UDP
Set Host / IP = the Mac's IP
Set Port = 9000
Set Hand side = Both
Tap Save / Apply then tap Start Stream.

Leave this app running — the Quest can keep it in the background while you open the browser.

4. Run the dashboard on the Mac

From a fresh terminal:

git clone https://github.com/aaryan-lath/Aether-Logic.git
cd Aether-Logic/quest
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install ikpy ultralytics
python mini_ng_dashboard.py
# → dashboard: http://0.0.0.0:5050/

The first run downloads the YOLOv8n weights (~6 MB) on-demand.

5. Open the dashboard inside the Quest's browser

On the Quest, open Meta Quest Browser (globe icon). Type the Mac's IP into the address bar:

http://10.10.10.147:5050/      # use YOUR Mac's IP here

Tap the fullscreen icon on the page. Pick the Pilot tab. Move your hand — the 3D skeleton follows. Pinch your thumb and index; the gripper bar fills.

6. If nothing shows up

/api/hand returning "present": false? The Quest isn't reaching the Mac. Confirm both are on the same Wi-Fi with the same /24 subnet (ipconfig getifaddr en0 on Mac vs the IP shown in the streamer app).
Still nothing? Run python quest/raw_probe.py — it logs every incoming UDP packet on :9000 with source IP. Silent probe + configured streamer = Wi-Fi is blocking peer UDP; switch to a hotspot.
Pilot tab video is black? macOS needs Camera permission for the terminal/Python — open System Settings → Privacy & Security → Camera → enable your terminal app, then restart mini_ng_dashboard.py.
Pilot tab video shows your room from the wrong angle? You have a USB webcam that's being picked instead of the built-in — override it with PILOT_CAM_INDEX=N python mini_ng_dashboard.py where N is the index you want (cameras get logged at startup).

Demo recording

The submission video uses two parallel captures:

Quest 3S native screen recorder — POV of the operator looking at the dashboard. Hold the Oculus button → Record Video. Files land in Internal storage/Oculus/VideoShots/; pull them to the Mac with quest/pull_quest_pov.sh.
Mac-side dashboard recorder — overhead cam frames + hand-pose JSONL land in quest/recordings/<ts>.{mp4,jsonl}.

Diagnostics

python raw_probe.py — shows every incoming UDP packet with source IP, size, and a heartbeat every second. Useful when the pipeline reports no packet in 1s.
python udp_probe.py — simpler stand-alone probe.
python test_calibration_engine.py — 6 unit tests for the anchor/delta math (all pass).

Troubleshooting

No packets arriving. Run raw_probe.py. If the heartbeat ticks but packets stay at 0, the issue is between the Quest and the Mac. Confirm both are on the same Wi-Fi, that the streamer's destination IP matches the Mac's current IP, and that macOS Firewall isn't blocking the Python binary (/usr/libexec/ApplicationFirewall/socketfilterfw --getappblocked $(which python)).

Port already in use. Another process holds :9000. Find and kill it: lsof -iUDP:9000 then kill -9 <PID>.

Pipeline binds but sees nothing. Force-restart the streamer app on the headset (Stop then Start). Quest often sleeps the streamer when the headset is removed.

Project structure

Aether-Logic/
├── quest/                          # Mac-side teleop pipeline
│   ├── run_pipeline.py             # CLI IK dry run (joint-angle printout)
│   ├── mini_ng_hand_listener.py    # UDP ingress + parser (HandFrame iterator)
│   ├── calibration_engine.py       # Logical Home Pairing
│   ├── mini_ng_dashboard.py        # Flask operator UI, Teleop + Rover tabs
│   ├── templates/dashboard.html    # two-tab UI + 3D hand wireframe (Three.js)
│   ├── recordings/                 # saved <ts>.mp4 + <ts>.jsonl
│   ├── raw_probe.py                # diagnostic: raw UDP + heartbeat
│   ├── udp_probe.py                # diagnostic: packet dump
│   ├── test_calibration_engine.py  # 6 unit tests (pass)
│   └── requirements.txt
├── lerobot/
│   ├── urdf/SO101/                 # SO-101 URDF + STL assets
│   └── scripts/ik_sanity_check.py
├── firmware/esp32/                 # ESP32 safety sentinel — descoped for submission
│   ├── CLAUDE.md                   # retrospective + design doc for revival
│   └── mock_telemetry.py           # host-side telemetry publisher for UI rehearsal
├── Start Demo.app                  # one-click Finder launcher
└── Stop Demo.app

Key parameters

Parameter	Value	Meaning
`HAND_TO_ARM_SCALE`	0.5	10 cm hand motion = 5 cm arm motion
`WORKSPACE_HALFWIDTH_M`	0.30	Arm target clamped to ±30 cm cube around home
`PINCH_CLOSED_CM`	1.5	Pinch distance mapped to gripper = 0.0 (closed)
`PINCH_OPEN_CM`	8.0	Pinch distance mapped to gripper = 1.0 (open)
`LIMIT_WARN_FRACTION`	0.85	Joint flagged `!` past 85% of URDF limit

Future Implementation

Close the loop onto the physical SO-101 over the Robot_Bus (USB serial).
Revive the ESP32 Safety Sentinel firmware (see firmware/esp32/CLAUDE.md) and wire hazard detection into the dashboard's live telemetry panel.
Port the YOLO detector from the MiniPC CPU to the Qualcomm NPU via ONNX export / AI Hub.
Record 25 teleop episodes, train ACT on MI300X, deploy the checkpoint for autonomous Rover operation.
Extend the use case beyond the kitchen: fire response, post-disaster reconnaissance, contaminated labs, nuclear decomm, and sustained lunar/space presence — anywhere human judgment is needed but human bodies shouldn't be.

Acknowledgements

wengmister/hand-tracking-streamer — Quest UDP hand streaming
ikpy — inverse kinematics
LeRobot — robot driver + dataset format
Ultralytics YOLOv8 — multi-camera object detection

Name		Name	Last commit message	Last commit date
Latest commit History 22 Commits
.claude		.claude
Start Demo.app/Contents		Start Demo.app/Contents
Stop Demo.app/Contents		Stop Demo.app/Contents
firmware/esp32		firmware/esp32
lerobot		lerobot
quest		quest
utensil_pickup_stir @ a8dbce8		utensil_pickup_stir @ a8dbce8
.gitignore		.gitignore
.gitmodules		.gitmodules
.mcp.json		.mcp.json
CLAUDE.md		CLAUDE.md
README.md		README.md
claude-code-safety.md		claude-code-safety.md
start_demo.command		start_demo.command
stop_demo.command		stop_demo.command

Navigation Menu

Repository files navigation

Cadence Labs

Team Information

Items

Approach

Technical Implementation

Ease of Use

How the pieces fit together

What the dashboard does

Architecture

Hardware

Setup

1. Set up the Python environment

2a. Run the CLI pipeline (IK dry run, no UI)

2b. Run the operator dashboard (demo UI, Teleop + Rover tabs)

2c. Double-click to launch (macOS)

Try the hand tracking on your own Quest

What you need

1. Enable developer mode on the Quest (one-time)

2. Sideload `hand-tracking-streamer` onto the Quest

3. Set the destination IP inside the streamer app

4. Run the dashboard on the Mac

5. Open the dashboard inside the Quest's browser

6. If nothing shows up

Demo recording

Diagnostics

Troubleshooting

Project structure

Key parameters

Future Implementation

Acknowledgements

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Cadence Labs

Team Information

Items

Approach

Technical Implementation

Ease of Use

How the pieces fit together

What the dashboard does

Architecture

Hardware

Setup

1. Set up the Python environment

2a. Run the CLI pipeline (IK dry run, no UI)

2b. Run the operator dashboard (demo UI, Teleop + Rover tabs)

2c. Double-click to launch (macOS)

Try the hand tracking on your own Quest

What you need

1. Enable developer mode on the Quest (one-time)

2. Sideload hand-tracking-streamer onto the Quest

3. Set the destination IP inside the streamer app

4. Run the dashboard on the Mac

5. Open the dashboard inside the Quest's browser

6. If nothing shows up

Demo recording

Diagnostics

Troubleshooting

Project structure

Key parameters

Future Implementation

Acknowledgements

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

2. Sideload `hand-tracking-streamer` onto the Quest