PyKaboo 🐭📷

Point a camera at your mice and... peekaboo!
PyKaboo sees them, draws their pose, names what they're doing, and fires your TTLs in real time — all from one tidy Windows app.

---

PyKaboo is a Windows desktop app for synchronized camera acquisition, planner-driven recording, live animal detection, and closed-loop Arduino TTL control. Basler, FLIR, and USB cameras all live under one interface, and the whole recording workflow stays tied to a multi-trial session plan so your filenames, metadata, and triggers are always aligned.

It started as a careful acquisition tool. It grew a brain. 🧠

Every chip is computed per mouse (the two directed views of the scene), so you can tell who is doing what. The label, the colour, and the connector line all match the animal's identity.

✨ Highlights

• 🗂️ Planner-first workflow — trial rows drive filename, metadata, and recording duration; the planner auto-restores on launch and auto-advances to the next pending trial.
• 🎥 Many cameras, one window — Basler, FLIR (machine-vision + thermal), and USB; add up to three auxiliary cameras that all record in sync.
• 🧩 Live segmentation + pose — choose MAMIR Unified for one-forward visible/amodal masks, causal IDs, and geometry pose, or pair RF-DETR/YOLO segmentation with an independent pose source.
• 🧠 Live social behavior detection — detect social behaviors frame-by-frame (see below) and overlay them on the preview and the recorded video.
• ⚡ Closed-loop TTL — turn a detected behavior (or an ROI / proximity / mask-contact event) into an Arduino pulse for optogenetics and stimulation, in real time.
• 🔊 Synchronised ultrasound (USV) — record ultrasonic vocalizations from one or more mics, each WAV time-locked to the video, with a live spectrogram.
• 🧾 Frame-aligned exports — MP4 + metadata/TTL/behavior CSVs, all zero-referenced to the first recorded frame so every clock lines up.

Full design notes live in pykaboo_live_behavior/INTEGRATION.md.

🖥️ The interface

PyKaboo keeps you in one place from setup to acquisition — connect hardware, plan trials, watch the live stream, and record with metadata, audio, and TTLs already aligned. Tools live in slide-in side panels so the preview keeps the space.

The acquisition workspace: a big live preview with health/FPS telemetry up top and one-click Acquisition / Recording cards along the bottom.

🗂️ Plan the whole session up front

The Recording Planner drives everything: each row sets the filename, metadata, and duration for a trial. Import a CSV or build rows inline; finished trials are marked Acquired and the next pending one is auto-selected.

🎥 Many cameras, one click

Basler, FLIR, and USB sources are auto-detected and listed together — here a FLIR Blackfly and a USB camera. Add up to three auxiliary cameras with the camera button up top; every connected stream records in sync with one Start Recording.

🧠 Live detection + behavior

Pick a segmentation model and mouse count, toggle overlays (masks / boxes / keypoints / behavior), choose the behavior method (rule-based or ML), then use the ROIs / TTL / Rules tabs to draw zones, map DO pins, and arm trigger rules.

The Model menu has two intentionally different workflows:

• MAMIR Unified (mask + pose + track) runs one visible-plus-amodal detector forward. Visible masks feed the overlay and eight-point geometry pose; aligned amodal masks feed the causal mask-IoU tracker during short occlusions. No second pose network is started. When frames arrive back-to-back, one bounded ordered CPU stage finalizes the previous identity/geometry result while the next GPU forward runs; paced input stays on the minimum-latency inline path.
• RF-DETR Seg / YOLO Seg preserve the existing modular workflow and can use YOLO pose, mask geometry, model-class identity, or no keypoints.

Automatic (recommended) chooses CUDA FP16 on a supported NVIDIA GPU and FP32 on MPS/CPU. Advanced controls remain available for confidence, preview inference width, acceleration, mask display, and recording outputs. On CUDA, raw-frame resize and normalization stay on the GPU; a compatible TensorRT engine also uses exact CUDA-graph replay with eager fallback. Recordings store the effective runtime, numerical version, model tier, and model hashes. Masks remain losslessly crop-packed through live tracking and geometry pose, then expand only when rendering or export requires dense pixels. See MAMIR Unified live inference for the exact runtime and installation contract. See live inference performance for the standard and unified execution paths, acceleration behavior, output-parity contract, and measured latency.

⚡ Arduino & TTL control

Connect the Firmata board, map signal roles to pins, drive DO1-8 live outputs, generate barcode/sync, and add extra Arduino devices — every line is logged frame-aligned in the metadata CSV.

🔊 Synchronised ultrasound (USV)

Record ultrasonic vocalizations from one or more mics (Pettersson / USB), each WAV synchronised to the video, with a live RMS/peak meter and a real-time kHz spectrogram.

🚀 Quick start

🎬 See it in action

Two mice, live, on plain bedding — masks, pose skeletons, and a per-mouse behavior subtitle drawn straight onto the video (and burned into the recorded overlay MP4):

nose2nose — both heads meet, both chips agree at 100%.	nose2body — the actor (pink) sniffs the partner's flank; the recipient reads none.

Requirements

• Windows 10 or 11, Python 3.10 recommended
• ffmpeg on PATH
• Camera SDKs for vendor hardware: Basler (Pylon + pypylon), FLIR machine-vision (Spinnaker + PySpin), FLIR thermal (flirpy)
• An NVIDIA GPU is strongly recommended for live detection (CUDA / TensorRT)

Overlay recording tests every available FFmpeg build instead of trusting the first executable on PATH. Automatic mode tries a functioning NVENC, Quick Sync, VideoToolbox, AMF, or Media Foundation H.264 encoder and falls back to portable libx264 if the device, driver, or concurrent session is unavailable. It tests a different hardware block before reusing the raw recorder's encoder. The fallback is automatic; a failed hardware probe does not lose the recording.

The dependency files intentionally share one ML stack (torch>=2.7,<2.9, torchvision>=0.22,<0.24, rfdetr>=1.7,<1.8, and numpy>=1.26,<2) so the modular and MAMIR Unified modes run in one interpreter without duplicate RF-DETR environments. The mask-geometry pose path automatically uses cached native kernels when supported; it retains the exact NumPy implementation as a transparent fallback. Live geometry landmarks are stabilized more strongly while the animal is stationary and respond progressively faster as it moves; this affects the live presentation/control stream, not the compatible scientific geometry extractor.

Install (conda)

conda env create -f environment.yaml
conda activate CamApp

Install (venv)

python -m venv .venv
.\.venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Optional MAMIR Unified model

Install the reviewed MAMIR runtime into the same environment (authorized repository access is required), then place a compatible Small, Medium, or Large Unified checkpoint in models/ (mask_cpv3_large_amodal.pth remains the bundled default name):

python scripts/install_mamir_unified.py

Run the same command after pulling a PyKaboo update that changes the pinned MAMIR revision. The installer is safe to repeat: it force-replaces an older package even when both packages report the same version, then verifies the six-output engine contract and all three native model tiers before it reports success.

With an existing source checkout, avoid a network clone:

python scripts/install_mamir_unified.py --mamir-root C:\path\to\MAMIR

The checkpoint can live elsewhere when MAMIR_UNIFIED_CHECKPOINT points to it. PyKaboo never downloads or replaces model data at runtime. When several Unified checkpoints are installed, choose one from Live Detection → Unified weights. PyKaboo discovers models in models/, the currently selected folder, and any directories listed in MAMIR_UNIFIED_MODEL_DIRS.

TensorRT .engine files are local deployment artifacts. Keep the .pth as the portable source of truth and use the in-app build prompt on each target computer; do not assume an engine built for a different GPU, TensorRT, CUDA/runtime, driver, or operating-system stack is portable.

Run

python main.py

On Windows prefer run_pykaboo.bat, which finds a Python runtime that can import PySide6, cv2, and PySpin (so FLIR cameras don't vanish when launched from the wrong interpreter, and torch loads in the right DLL order).

🗂️ Planner workflow

• Import a CSV plan or build rows directly in the Recording Planner.
• The current planner is saved automatically and restored on the next launch.
• Select a row to load its metadata into the session form.
• Ctrl+C / Ctrl+V on a row copies trial content onto other rows.
• Right-click rows for duplicate, copy, paste, move up/down, apply, and remove.
• When a trial finishes recording it is marked Acquired and the next pending row is selected.

📦 Outputs

Each recording can produce:

• <name>.mp4 and <name>_overlay.mp4 (masks + pose + behavior subtitles)
• <name>_metadata.csv / .json / .txt
• <name>_ttl_states.csv, <name>_ttl_counts.csv
• <name>_live_detections.csv and <name>_tracking_dlc.csv (DLC-style keypoints)
• <name>_masks_coco.json (segmentation masks)
• <name>_behavior_summary.csv
• <name>_<backend>_<n>.mp4 (+ metadata) per auxiliary camera

Timestamp columns (timestamp_software, timestamp_camera, timestamp_ticks) are elapsed seconds starting at 0 on the first recorded frame; camera_frame_id is rebased to 0 so it can be compared with frame_id to spot dropped frames.

🔌 Arduino & TTL

PyKaboo supports:

• StandardFirmata for generic TTL monitoring and output control
• StandardFirmataBarcode for the custom barcode/sync workflow (StandardFirmataBarcode)

Additional Arduino devices: the primary board (gate/sync/barcode + DO1-8 live outputs) is unchanged. To drive extra outputs or log extra inputs, use Additional Arduino Devices at the bottom of the Arduino Setup panel — add a board, pick its COM port, and set each pin to Input (sampled per camera frame) or Output. Each input/output is logged as a frame-aligned dev<id>_<label>_ttl column in *_metadata.csv. Auxiliary boards must each use their own COM port, and the roster persists between launches.

🛠️ Build a Windows EXE

python -m pip install -r requirements-dev.txt
.\scripts\build_release.ps1 -Version v2026.04.12 -PythonExe python -Clean

See camApp-live-detection.spec and scripts/build_release.ps1.

🧰 Troubleshooting

• ffmpeg not found — add FFmpeg to PATH and restart the shell.
• PySpin import errors — use a Spinnaker-compatible wheel and keep numpy<2.
• No live inference output — check the checkpoint path and that the ML/behavior packages are installed; a GPU is strongly recommended.
• MAMIR Unified is not installed — run python scripts/install_mamir_unified.py in the same environment used to launch PyKaboo, then restart the app.
• Empty / 1-frame overlay MP4 — fixed: a degenerate detection used to crash the overlay writer; it now skips bad frames and keeps recording.
• Vendor camera won't connect — confirm it opens in the vendor SDK viewer first.

---

Made for behavioural neuroscience rigs that need to see, decide, and act — frame by frame.