Skip to content

BENCHMARK_DASHBOARD.md

Latest commit

 

History

History
453 lines (348 loc) · 12.5 KB

BENCHMARK_DASHBOARD.md

File metadata and controls

453 lines (348 loc) · 12.5 KB

Vision Benchmark Dashboard

Real-time monitoring and benchmarking dashboard for the ROS2 Vision Pipeline services.

Features

The dashboard monitors and displays benchmark data for all vision services:

📍 Pixel to Real Conversion

  • Input: Pixel coordinates (u, v)
  • Output: World coordinates (x, y, z) in meters
  • Tracks all conversion calls with timestamps

🎯 SAM Object Detection

  • Object ID: Unique identifier for each detection
  • Bounding Box: (x1, y1, x2, y2) coordinates
  • Center Point: (u, v) pixel coordinates
  • Confidence: Detection confidence score [0-1]
  • IoU Score: Intersection over Union with previous frame
  • AP (IoU≥0.5): COCO-style Average Precision metric (Pass/Fail)
  • Distance: Object distance in centimeters
  • Metrics: Average IoU, Stability Rate, Average Confidence

🏷️ CLIP Classification

  • Label: Predicted object class
  • Confidence: Classification confidence [0-1]
  • Top-1 Accuracy: Correctness indicator (True/False/N/A)
  • Bounding Box: Region of interest
  • Metrics: Top-1 Accuracy Rate, Average Confidence

🤖 Grasp Detection (GraspNet)

  • Object ID: Associated detected object
  • Pixel Position: (u, v) grasp point in image
  • World Position: (x, y, z) grasp point in 3D space
  • Quality Score: Grasp quality metric [0-1]
  • Grasp Width: Gripper opening width in meters
  • Approach Angle: Grasp orientation/direction
  • Metrics: Average Quality Score, Average Grasp Width

🌐 Scene Understanding

  • Scene ID: Unique scene identifier
  • Total Objects: Number of detected objects
  • Relations: Spatial relationships between objects
    • Subject → Relation → Object (with confidence)
  • Scene Description: Natural language summary
  • Spatial Accuracy: Accuracy of spatial relations (%)
  • Adjacency Accuracy: Accuracy of near/touching relations (%)
  • Metrics: Total Objects, Total Relations, Spatial/Adjacency Accuracy

Installation

  1. Build the package:

    cd ~/final_project_ws
colcon build --packages-select vision --symlink-install
source install/setup.bash

  2. No additional dependencies - Uses built-in Python HTTP server

Usage

Quick Start

# Option 1: Use the convenience script (auto-opens browser)
./testsh/start_benchmark_dashboard.sh

# Option 2: Run directly
ros2 run vision benchmark_dashboard

The dashboard will be available at: http://localhost:8080

Running with Pipeline Services

For full monitoring, run all vision services:

# Terminal 1: Start SAM detector
ros2 run vision simple_sam_detector

# Terminal 2: Start CLIP classifier
ros2 run vision clip_classifier

# Terminal 3: Start GraspNet detector
ros2 run vision graspnet_detector

# Terminal 4: Start Scene Understanding
ros2 run vision scene_understanding

# Terminal 5: Start Pixel-to-Real service
ros2 run vision pixel_to_real_service

# Terminal 6: Start Benchmark Dashboard
ros2 run vision benchmark_dashboard

Then trigger the pipeline:

ros2 service call /vision/run_pipeline std_srvs/srv/Trigger

Using the Dashboard

  1. Auto-refresh: Dashboard updates every 2 seconds automatically
  2. Clear Data: Use the "Clear All Data" button or run: 
    ros2 service call /benchmark/clear_data std_srvs/srv/Trigger
  3. Export Data: Subscribe to the data topic: 
    ros2 topic echo /benchmark/data

Dashboard Features

Real-time Statistics Cards

  • Total service calls
  • Count of each service type
  • Color-coded confidence levels:
    • 🟢 Green: High confidence (≥0.7)
    • 🟠 Orange: Medium confidence (0.4-0.7)
    • 🔴 Red: Low confidence (<0.4)

Interactive Tables

  • Scrollable data tables (up to 50 most recent records)
  • Hover effects for better readability
  • Timestamp formatting
  • Color-coded metrics

Performance Metrics

  • SAM: Average IoU, Stability Rate, Average Confidence
  • CLIP: Top-1 Accuracy, Average Confidence
  • GraspNet: Average Quality Score, Average Grasp Width
  • Scene: Spatial Accuracy, Adjacency Accuracy

Data Collection

The dashboard is non-invasive - it doesn't modify any existing service nodes. It works by:

  1. Subscribing to topics: Monitors /vision/sam_detections and /vision/scene_understanding
  2. Publishing data: Broadcasts collected data on /benchmark/data topic
  3. HTTP API: Serves data via /api/data endpoint for the web interface

API Endpoints

GET /api/data

Returns complete benchmark data in JSON format:

{
  "pixel_to_real": [...],
  "sam_detections": [...],
  "clip_classifications": [...],
  "grasp_detections": [...],
  "scene_understanding": [...],
  "metadata": {
    "start_time": "2026-02-10T...",
    "total_calls": 142
  }
}

Service: /benchmark/clear_data

Clears all stored benchmark data:

ros2 service call /benchmark/clear_data std_srvs/srv/Trigger

Extending the Dashboard

To add custom metrics or data collection:

  1. Modify benchmark_dashboard.py: Add new data collection methods
  2. Update dashboard/index.html: Add new visualization tables
  3. Rebuild: colcon build --packages-select vision --symlink-install

Troubleshooting

Dashboard not loading?

  • Check if port 8080 is available: netstat -tuln | grep 8080
  • Verify node is running: ros2 node list | grep benchmark

No data showing?

  • Ensure vision services are running
  • Trigger the pipeline: ros2 service call /vision/run_pipeline std_srvs/srv/Trigger
  • Check topic connections: ros2 topic info /vision/sam_detections

Data not updating?

  • Check browser console for JavaScript errors (F12)
  • Verify /api/data endpoint: curl http://localhost:8080/api/data

Performance Notes

  • Data is limited to 1000 most recent records per service type to prevent memory issues
  • Tables show 50 most recent records for optimal rendering
  • Auto-refresh interval: 2 seconds (can be modified in HTML)
  • HTTP server runs on a separate thread to avoid blocking ROS2 callbacks

License

Apache-2.0

Author

Final Project - Group 11

Vision Benchmark Dashboard - Implementation Summary

✅ What Was Created

A complete, non-invasive benchmark monitoring system for your ROS2 Vision Pipeline with real-time HTML dashboard.

📦 New Files

1. Backend Node

vision/benchmark_dashboard.py (520 lines)

  • ROS2 node that monitors all vision services
  • Subscribes to topics: /vision/sam_detections, /vision/scene_understanding
  • Publishes: /benchmark/data (JSON stream)
  • Provides: /benchmark/clear_data service
  • Built-in HTTP server on port 8080
  • No modifications to existing service nodes

2. Frontend Dashboard

dashboard/index.html (800+ lines)

  • Beautiful, responsive HTML dashboard
  • Auto-refreshing every 2 seconds
  • Color-coded metrics (green/orange/red)
  • Scrollable data tables
  • Real-time statistics cards

3. Launch Scripts

testsh/start_benchmark_dashboard.sh

  • One-command launcher
  • Auto-opens browser
  • Checks ROS2 environment

testsh/test_benchmark_dashboard.sh

  • Generates sample benchmark data
  • Tests all services
  • Verifies system health

4. Documentation

docs/BENCHMARK_DASHBOARD.md - Full documentation docs/DASHBOARD_QUICKSTART.md - Quick reference guide

5. Configuration

setup.py - Updated with new entry point

📊 Dashboard Features

Data Tables

1. Pixel to Real Conversion

Column Description
Test ID Sequential test number
Timestamp When conversion was called
Input U, V Pixel coordinates
Output X, Y, Z World coordinates (meters)

2. SAM Object Detection

Column Description
Object ID Unique detection ID
Bounding Box (x1, y1, x2, y2) coordinates
Center Point (u, v) pixel center
Confidence Detection confidence [0-1]
IoU Score Intersection over Union
AP (IoU≥0.5) Pass/Fail indicator
Distance Object distance (cm)

Metrics: Avg IoU, Stability Rate, Avg Confidence

3. CLIP Classification

Column Description
Test ID Sequential test number
Label Predicted object class
Confidence Classification confidence [0-1]
Top-1 Accuracy True/False/N/A
Bounding Box Region of interest

Metrics: Top-1 Accuracy Rate, Avg Confidence

4. GraspNet Detection

Column Description
Test ID Sequential test number
Object ID Associated object
Position (u,v) Pixel coordinates
Position (x,y,z) World coordinates (m)
Quality Score Grasp quality [0-1]
Grasp Width Gripper width (m)
Approach top/side/angle

Metrics: Avg Quality Score, Avg Grasp Width

5. Scene Understanding

Column Description
Scene ID Unique scene identifier
Objects List of detected objects
Relations Spatial relationships
Description Natural language summary
Spatial Acc Spatial relation quality (%)
Adjacency Acc Proximity relation quality (%)

Metrics: Total Objects, Total Relations, Spatial/Adjacency Accuracy

🚀 Usage

Quick Start

cd ~/final_project_ws
colcon build --packages-select vision --symlink-install
source install/setup.bash

# Start dashboard (auto-opens browser)
./src/vision/testsh/start_benchmark_dashboard.sh

Generate Test Data

# In another terminal
./src/vision/testsh/test_benchmark_dashboard.sh

Full Pipeline

# Start all services (separate terminals)
ros2 run vision pixel_to_real_service
ros2 run vision simple_sam_detector
ros2 run vision clip_classifier
ros2 run vision graspnet_detector
ros2 run vision scene_understanding
ros2 run vision benchmark_dashboard

# Trigger pipeline
ros2 service call /vision/run_pipeline std_srvs/srv/Trigger

🎨 Dashboard Design

Visual Features

  • ✨ Gradient background (purple/blue)
  • 📊 Real-time statistics cards
  • 📈 Color-coded metrics (green/orange/red)
  • 🔄 Auto-refresh indicator
  • 📱 Responsive design
  • 🎯 Clean, modern UI

Color Coding

  • Green (≥0.7): High confidence/quality
  • Orange (0.4-0.7): Medium confidence/quality
  • Red (<0.4): Low confidence/quality

Performance

  • Data limit: 1000 records per service (memory efficient)
  • Display limit: 50 most recent records per table
  • Refresh rate: 2 seconds
  • Smooth scrolling tables

🔧 Technical Details

Data Flow

Vision Services → ROS2 Topics → Benchmark Node → HTTP Server → HTML Dashboard
                                      ↓
                               /benchmark/data

Architecture

  1. Non-invasive: No changes to existing service nodes
  2. Passive monitoring: Subscribes to topics
  3. Decoupled: Dashboard works independently
  4. Scalable: Easy to add new metrics

API Endpoints

  • GET /api/data - JSON data feed
  • GET / - Dashboard HTML
  • POST /benchmark/clear_data - Clear data (ROS2 service)

📈 Metrics Calculated

SAM Metrics

  • Average IoU across all detections
  • Stability rate (% with IoU ≥ 0.5)
  • Average confidence score

CLIP Metrics

  • Top-1 accuracy rate
  • Average confidence score

GraspNet Metrics

  • Average quality score
  • Average grasp width

Scene Understanding Metrics

  • Spatial accuracy (based on relation confidence)
  • Adjacency accuracy (near/touching relations)

🎯 Key Benefits

  1. No Code Changes: Original services remain untouched
  2. Real-time Monitoring: Live updates every 2 seconds
  3. Comprehensive: All services in one dashboard
  4. Professional UI: Beautiful, intuitive interface
  5. Easy to Use: One command to start
  6. Exportable: Data available via ROS2 topic
  7. Browser-based: No special client needed
  8. Scalable: Handles thousands of records

📝 Notes

  • Dashboard stores last 1000 records per service type
  • Tables display 50 most recent records
  • HTTP server runs on port 8080
  • Auto-opens browser on launch
  • Data persists until cleared or node restart

🚀 Next Steps

  1. Build the package:

    colcon build --packages-select vision --symlink-install
source install/setup.bash

  2. Start the dashboard:

    ./src/vision/testsh/start_benchmark_dashboard.sh

  3. Generate test data:

    ./src/vision/testsh/test_benchmark_dashboard.sh

  4. View at: http://localhost:8080

📚 Documentation

  • Full docs: docs/BENCHMARK_DASHBOARD.md
  • Quick start: docs/DASHBOARD_QUICKSTART.md
  • This summary: docs/DASHBOARD_IMPLEMENTATION_SUMMARY.md

Status: ✅ Complete and Ready to Use!