Skip to content

Commit 5ea6171

Browse files
adsahu-nvadsahu-nv
committed
Add new skills for Multi-modal AI Studio
- Introduced multiple new skills in skills.yaml, including voice-vision-session, visual-question-answering, riva-speech, llm-backend-setup, preset-configuration, and device-management. - Each skill includes detailed documentation in their respective SKILL.md files, covering setup, usage, and configuration options for ASR, LLM, TTS, and device management. - Enhanced the overall functionality of the Multi-modal AI Studio by providing comprehensive guides for deploying and managing AI sessions with voice and vision capabilities.
1 parent 61b0116 commit 5ea6171

7 files changed

Lines changed: 1075 additions & 0 deletions

File tree

‎.agents/skills/device-management/SKILL.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
---
2+
name: device-management
3+
description: >
4+
Enumerate and configure server-side USB cameras, microphones, and speakers
5+
for Multi-modal AI Studio. Stream camera preview via WebRTC or MJPEG.
6+
Use when deploying on Jetson or headless servers with USB peripherals.
7+
license: Apache-2.0
8+
metadata:
9+
author: NVIDIA Corporation
10+
version: "1.0"
11+
---
12+
13+
# Device Management
14+
15+
Manage server-attached USB cameras, microphones, and speakers.
16+
17+
## Overview
18+
19+
When running on a Jetson or headless server, MMAS can use USB devices attached directly to the server (instead of browser media). The server enumerates V4L2 cameras and ALSA audio devices and exposes them via REST APIs.
20+
21+
## Instructions
22+
23+
### List USB cameras
24+
25+
```bash
26+
curl -s http://localhost:8092/api/devices/cameras | python3 -m json.tool
27+
```
28+
29+
Response:
30+
```json
31+
[
32+
{"id": "/dev/video0", "label": "USB 3.0 Camera (Server USB)"},
33+
{"id": "/dev/video2", "label": "HD Webcam (Server USB)"}
34+
]
35+
```
36+
37+
### List microphones
38+
39+
```bash
40+
curl -s http://localhost:8092/api/devices/audio-inputs | python3 -m json.tool
41+
```
42+
43+
Response:
44+
```json
45+
[
46+
{"id": "alsa:hw:1,0", "label": "Blue Yeti Nano (Server USB)"},
47+
{"id": "alsa:hw:2,0", "label": "NVIDIA Jetson Thor AGX APE (Server USB)"}
48+
]
49+
```
50+
51+
### List speakers
52+
53+
```bash
54+
curl -s http://localhost:8092/api/devices/audio-outputs | python3 -m json.tool
55+
```
56+
57+
### Camera preview (MJPEG stream)
58+
59+
```bash
60+
# Stream in browser or with curl
61+
curl -s "http://localhost:8092/api/camera/stream?device=/dev/video0" --output -
62+
```
63+
64+
The MJPEG stream is `multipart/x-mixed-replace` with JPEG frames. It also pushes frames into the internal `FrameBroker` for VLM consumption.
65+
66+
### Camera preview (WebRTC)
67+
68+
Connect via WebSocket at `/ws/camera-webrtc?device=/dev/video0` for low-latency video preview with WebRTC signaling (offer/answer/ICE).
69+
70+
### Microphone preview
71+
72+
Connect via WebSocket at `/ws/mic-preview` to stream server mic audio levels. Send a config message first, then receive `user_amplitude` JSON events.
73+
74+
### Configure devices in a session
75+
76+
```json
77+
{
78+
"devices": {
79+
"video_source": "usb",
80+
"video_device": "/dev/video0",
81+
"audio_input_source": "alsa",
82+
"audio_input_device": "hw:1,0",
83+
"audio_output_source": "alsa",
84+
"audio_output_device": "hw:0,0"
85+
}
86+
}
87+
```
88+
89+
| Field | Values | Description |
90+
|-------|--------|-------------|
91+
| `video_source` | `browser`, `usb`, `local`, `none` | Where camera frames come from |
92+
| `audio_input_source` | `browser`, `usb`, `alsa` | Where microphone audio comes from |
93+
| `audio_output_source` | `browser`, `alsa` | Where TTS audio plays |
94+
95+
## Input Schema
96+
97+
Camera list endpoint:
98+
- No parameters required
99+
100+
Audio endpoints:
101+
- No parameters required
102+
103+
Camera stream:
104+
- `device` (string, optional): V4L2 device path (e.g., `/dev/video0`)
105+
106+
## Output Schema
107+
108+
Device list:
109+
```json
110+
[
111+
{"id": "device_path_or_id", "label": "Human-readable name (Server USB)"}
112+
]
113+
```
114+
115+
## Guidelines
116+
117+
- Cameras create multiple `/dev/video*` nodes — MMAS filters out metadata nodes automatically using V4L2 capability checks
118+
- USB mics may only support 48kHz natively — MMAS uses `plughw:` ALSA devices for automatic rate conversion to 16kHz
119+
- Only one process can open a V4L2 camera at a time — if WebRTC is active, MJPEG will fail on the same device
120+
- Connect USB devices before starting the server for reliable enumeration
121+
- The `FrameBroker` stores frames in a ring buffer (max 100 frames, 10s max age) for the VLM pipeline

‎.agents/skills/llm-backend-setup/SKILL.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
---
2+
name: llm-backend-setup
3+
description: >
4+
Configure and connect to an OpenAI-compatible LLM or VLM backend for Multi-modal AI Studio.
5+
Covers vLLM, Ollama, and OpenAI API setup including Docker commands,
6+
model listing, health checks, and warmup. Use when setting up the language model backend
7+
before running voice or vision sessions.
8+
license: Apache-2.0
9+
metadata:
10+
author: NVIDIA Corporation
11+
version: "1.0"
12+
---
13+
14+
# LLM Backend Setup
15+
16+
Configure an OpenAI-compatible LLM/VLM backend for Multi-modal AI Studio.
17+
18+
## Overview
19+
20+
MMAS connects to any OpenAI-compatible API (`/v1/chat/completions`). Supported backends:
21+
22+
| Backend | Default Port | Vision Support | Best For |
23+
|---------|-------------|---------------|----------|
24+
| vLLM | 8010 | Yes (image + video) | Production VLM serving |
25+
| Ollama | 11434 | Yes (images only) | Easy local setup |
26+
| OpenAI API | N/A | Yes (GPT-4V) | Cloud-based |
27+
28+
## Prerequisites
29+
30+
- Docker with NVIDIA runtime configured (for containerized backends)
31+
- GPU with sufficient VRAM for the chosen model
32+
- For vLLM with Cosmos-Reason2: download the model first — see [INSTALL.md](../../../INSTALL.md) or [vlm_guide.md](../../../docs/vlm_guide.md) for model setup instructions
33+
34+
## Instructions
35+
36+
### Option A: Ollama (easiest)
37+
38+
```bash
39+
# Install Ollama
40+
curl -fsSL https://ollama.com/install.sh | sh
41+
42+
# Pull a model
43+
ollama pull nemotron-3-nano:4b # Text-only LLM
44+
ollama pull gemma3:4b # Vision LLM (images only, no video)
45+
46+
# Ollama auto-starts on port 11434
47+
curl -s http://localhost:11434/v1/models | python3 -m json.tool
48+
```
49+
50+
### Option B: vLLM (recommended for VLMs)
51+
52+
> **Model setup**: Download the Cosmos-Reason2 model before running vLLM.
53+
> See [INSTALL.md](../../../INSTALL.md) or [vlm_guide.md](../../../docs/vlm_guide.md) for download instructions and Jetson AI Lab links.
54+
55+
```bash
56+
export MODEL_PATH=/path/to/cosmos-reason2-8b-fp8
57+
58+
sudo docker run -d --network host --runtime=nvidia \
59+
--name vllm-cosmos \
60+
-v $MODEL_PATH:/models/cosmos-reason2-8b:ro \
61+
ghcr.io/nvidia-ai-iot/vllm:latest \
62+
vllm serve /models/cosmos-reason2-8b \
63+
--served-model-name nvidia/cosmos-reason2-8b-fp8 \
64+
--max-model-len 8192 \
65+
--gpu-memory-utilization 0.7 \
66+
--reasoning-parser qwen3 \
67+
--media-io-kwargs '{"video": {"num_frames": -1}}' \
68+
--enable-prefix-caching \
69+
--port 8010
70+
```
71+
72+
> **GPU memory cleanup**: If vLLM fails with OOM after stopping another GPU container:
73+
> ```bash
74+
> sudo sysctl -w vm.drop_caches=3
75+
> ```
76+
77+
### Option C: OpenAI API
78+
79+
Set your API key as an environment variable:
80+
81+
```bash
82+
export OPENAI_API_KEY=sk-...
83+
python -m multi_modal_ai_studio \
84+
--llm-api-base https://api.openai.com/v1 \
85+
--llm-api-key "$OPENAI_API_KEY" \
86+
--llm-model gpt-4o
87+
```
88+
89+
### Verify the backend
90+
91+
```bash
92+
# List available models
93+
curl -s http://localhost:8092/api/llm/models?api_base=http://localhost:8010/v1 | python3 -m json.tool
94+
95+
# Health check
96+
curl -s http://localhost:8092/api/health/llm?api_base=http://localhost:8010/v1
97+
# {"status": "ok"}
98+
99+
# Warm up the model (first request is slow due to CUDA graph capture)
100+
curl -s -X POST http://localhost:8092/api/llm/warmup \
101+
-H "Content-Type: application/json" \
102+
-d '{"api_base": "http://localhost:8010/v1", "model": "nvidia/cosmos-reason2-8b-fp8"}'
103+
```
104+
105+
### Direct API test
106+
107+
```bash
108+
curl -s http://localhost:8010/v1/chat/completions \
109+
-H "Content-Type: application/json" \
110+
-d '{
111+
"model": "nvidia/cosmos-reason2-8b-fp8",
112+
"messages": [{"role": "user", "content": "Hello"}],
113+
"max_tokens": 50,
114+
"stream": false
115+
}' | python3 -m json.tool
116+
```
117+
118+
## Input Schema
119+
120+
MMAS LLM configuration fields:
121+
122+
- `api_base` (string, required): Backend URL (e.g., `"http://localhost:8010/v1"`)
123+
- `model` (string, required): Model name as served by the backend
124+
- `api_key` (string, optional): API key for authenticated endpoints
125+
- `temperature` (float, optional, default: 0.7): Sampling temperature
126+
- `max_tokens` (integer, optional, default: 512): Maximum response tokens
127+
- `extra_request_body` (string, optional): JSON merged into every request (e.g., `'{"chat_template_kwargs": {"enable_thinking": false}}'`)
128+
129+
## Output Schema
130+
131+
Health check:
132+
```json
133+
{"status": "ok"}
134+
```
135+
136+
Model list:
137+
```json
138+
{
139+
"models": ["nvidia/cosmos-reason2-8b-fp8"],
140+
"default_model": "nvidia/cosmos-reason2-8b-fp8"
141+
}
142+
```
143+
144+
## Guidelines
145+
146+
- Only one GPU-heavy container should run at a time on Jetson (shared memory)
147+
- Stop and remove old containers before starting new ones: `docker stop <name> && docker rm <name>`
148+
- Ollama does not support `video_url` — use image-only vision or vLLM for video
149+
- vLLM first request is slow (CUDA graph compilation); use the warmup endpoint

0 commit comments

Comments
 (0)