Enhancement: Add Prometheus metrics endpoint for monitoring

[filthyrake]

Description

System tracks extensive metrics (FPS, inference time, queue depth) but doesn't expose them in Prometheus format for monitoring tools like Grafana.

Impact

• Medium severity
• Limited observability
• Manual monitoring required
• No historical tracking

Implementation

1. Add Prometheus Client

pip install prometheus-client

2. Define Metrics

# src/metrics.py
from prometheus_client import Counter, Gauge, Histogram, Summary

# System metrics
system_uptime = Gauge('system_uptime_seconds', 'System uptime')
cameras_connected = Gauge('cameras_connected', 'Number of connected cameras')

# Per-camera metrics
camera_fps = Gauge('camera_fps', 'Camera FPS', ['camera_id'])
camera_dropped_frames = Counter('camera_dropped_frames_total', 
                                'Dropped frames', ['camera_id'])

# Inference metrics
inference_time = Histogram('inference_time_seconds', 
                          'Inference time', 
                          ['camera_id', 'detector'],
                          buckets=[0.01, 0.02, 0.05, 0.1, 0.2, 0.5])

detection_count = Counter('detections_total',
                         'Total detections',
                         ['camera_id', 'class'])

# Queue metrics
queue_depth = Gauge('queue_depth', 'Queue depth', ['queue_type', 'camera_id'])
queue_overflow = Counter('queue_overflow_total', 
                        'Queue overflows',
                        ['queue_type', 'camera_id'])

# GPU metrics
gpu_memory_used = Gauge('gpu_memory_used_bytes', 'GPU memory used')
gpu_utilization = Gauge('gpu_utilization_percent', 'GPU utilization')

3. Add Metrics Endpoint

# src/web_server.py
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST

@app.get("/metrics")
async def metrics():
    """Prometheus metrics endpoint"""
    return Response(
        content=generate_latest(),
        media_type=CONTENT_TYPE_LATEST
    )

4. Update Components

# Inference engine
def detect(self, frame):
    start = time.time()
    result = self.model(frame)
    
    inference_time.labels(
        camera_id=self.camera_id,
        detector='yolox'
    ).observe(time.time() - start)
    
    for det in result:
        detection_count.labels(
            camera_id=self.camera_id,
            class=det.class_name
        ).inc()

5. Grafana Dashboard

Create dashboards/grafana.json:

{
  "dashboard": {
    "title": "Wildlife Detection System",
    "panels": [
      {
        "title": "Inference Time",
        "targets": [{
          "expr": "rate(inference_time_seconds_sum[5m]) / rate(inference_time_seconds_count[5m])"
        }]
      },
      {
        "title": "Detections by Class",
        "targets": [{
          "expr": "rate(detections_total[5m])"
        }]
      }
    ]
  }
}

6. Alert Rules

# prometheus/alerts.yml
groups:
  - name: detection_system
    rules:
      - alert: HighInferenceLatency
        expr: inference_time_seconds > 0.05
        for: 5m
        annotations:
          summary: "Inference latency above 50ms"
          
      - alert: CameraDisconnected
        expr: cameras_connected < 2
        for: 1m
        annotations:
          summary: "Camera disconnected"

Benefits

• ✅ Historical metrics tracking
• ✅ Real-time dashboards
• ✅ Automated alerting
• ✅ Performance trending

Related

• Issue Add resource usage tracking and alerts #123 (Resource usage tracking and alerts)