feat: Add Prometheus metrics for GTFS processing monitoring #49

BrandonTrigueros · 2025-12-25T04:51:50Z

Pull Request: GTFS Monitoring Metrics Implementation

Issue: #16 - Definición de métricas de monitoreo de GTFS

Resumen

Implementación de métricas de Prometheus para monitorear el procesamiento de feeds GTFS en tiempo real. Esta solución permite observabilidad completa del sistema de ingesta de datos de transporte público.

Cambios Realizados

1. Integración de django-prometheus

Agregado django-prometheus>=2.3.1 a pyproject.toml
Configurado middleware de Prometheus en datahub/settings.py
Endpoint /metrics expuesto en datahub/urls.py

2. Definición de Métricas GTFS

Creado módulo feed/metrics.py con 7 métricas específicas para GTFS:

Counters (acumuladores)

gtfs_schedule_updates_total: Total de actualizaciones de horarios procesadas
gtfs_vehicle_positions_updates_total: Total de actualizaciones de posiciones de vehículos
gtfs_trip_updates_total: Total de actualizaciones de viajes procesadas
gtfs_entities_processed_total: Total de entidades procesadas (por tipo y proveedor)
gtfs_processing_errors_total: Total de errores encontrados durante el procesamiento

Gauge (valores actuales)

gtfs_last_update_timestamp: Timestamp de la última actualización exitosa

Histogram (distribución temporal)

gtfs_processing_duration_seconds: Tiempo de procesamiento de cada feed

3. Instrumentación de Tareas Celery

Modificado feed/tasks.py para registrar métricas en:

get_schedule(): Procesamiento de horarios GTFS Schedule
get_vehicle_positions(): Procesamiento de posiciones de vehículos GTFS-RT
get_trip_updates(): Procesamiento de actualizaciones de viajes GTFS-RT

4. Inicialización Automática

Modificado feed/apps.py para cargar métricas al inicio de Django usando AppConfig.ready()

5. Suite de Tests

Creado feed/test_metrics.py con tests unitarios para validar:

Definición correcta de todas las métricas
Incremento de contadores
Actualización de gauges
Registro de histogramas

Testing

# Verificar endpoint de métricas
curl http://localhost:8000/metrics | grep gtfs_

# Ejecutar tests
docker compose exec web /app/.venv/bin/pytest feed/test_metrics.py -v

# Trigger manual de tareas para verificar métricas
docker compose exec web /app/.venv/bin/python manage.py shell -c \
  "from feed.tasks import get_schedule; get_schedule.delay()"

Ejemplo de Output de Métricas

# HELP gtfs_schedule_updates_total Total number of schedule updates processed
# TYPE gtfs_schedule_updates_total counter
gtfs_schedule_updates_total{feed_id="MBTA-1735081421"} 1.0

# HELP gtfs_vehicle_positions_updates_total Total number of vehicle position updates processed
# TYPE gtfs_vehicle_positions_updates_total counter
gtfs_vehicle_positions_updates_total{provider="MBTA"} 45.0

# HELP gtfs_processing_duration_seconds Time taken to process feed
# TYPE gtfs_processing_duration_seconds histogram
gtfs_processing_duration_seconds_bucket{le="0.005",provider="MBTA",type="schedule"} 0.0
gtfs_processing_duration_seconds_bucket{le="0.01",provider="MBTA",type="schedule"} 0.0
gtfs_processing_duration_seconds_sum{provider="MBTA",type="schedule"} 2.45
gtfs_processing_duration_seconds_count{provider="MBTA",type="schedule"} 1.0

Uso en Producción

Integración con Grafana

Estas métricas pueden ser scrapeadas por Prometheus y visualizadas en Grafana:

# prometheus.yml
scrape_configs:
  - job_name: 'infobus'
    static_configs:
      - targets: ['infobus-web:8000']
    metrics_path: '/metrics'

Queries Útiles

# Tasa de actualizaciones por minuto
rate(gtfs_vehicle_positions_updates_total[5m])

# Errores en las últimas 24h
increase(gtfs_processing_errors_total[24h])

# Percentil 95 de tiempo de procesamiento
histogram_quantile(0.95, gtfs_processing_duration_seconds_bucket)

# Tiempo desde última actualización
time() - gtfs_last_update_timestamp

Archivos Modificados

pyproject.toml - Dependencia django-prometheus
datahub/settings.py - Middleware de Prometheus
datahub/urls.py - Endpoint /metrics
feed/apps.py - Carga automática de métricas
feed/metrics.py - NUEVO - Definiciones de métricas
feed/tasks.py - Instrumentación de tareas
feed/test_metrics.py - NUEVO - Suite de tests

Checklist

Métricas definidas y documentadas
Endpoint /metrics funcional
Tareas instrumentadas con manejo de errores
Tests unitarios implementados
Documentación completa
Zero-downtime deployment compatible

Próximos Pasos (Opcional)

Crear dashboards de Grafana preconfigurables
Agregar alertas de Prometheus para errores críticos
Instrumentar endpoints del API con métricas de latencia
Agregar métricas de business logic (calidad de datos GTFS)

Implements comprehensive monitoring metrics for GTFS feed processing to enable observability of the real-time transit data pipeline. Changes: - Added django-prometheus integration with /metrics endpoint - Defined 7 custom GTFS metrics (counters, gauges, histograms) - Instrumented Celery tasks (schedule, vehicle positions, trip updates) - Added automatic metrics initialization via AppConfig.ready() - Implemented test suite for metrics validation Resolves: #16

Copilot

Pull request overview

This PR adds Prometheus metrics instrumentation to monitor GTFS (General Transit Feed Specification) data processing in real-time. The implementation enables observability for schedule updates, vehicle position tracking, trip updates, and error monitoring through a comprehensive set of counters, gauges, and histograms.

Integration of django-prometheus library with middleware and metrics endpoint
Definition of 7 GTFS-specific metrics (counters, gauges, histograms) for monitoring feed processing
Instrumentation of three Celery tasks (schedule, vehicle positions, trip updates) with metrics recording and improved error handling

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
pyproject.toml	Added django-prometheus>=2.3.1 dependency
uv.lock	Updated lock file with django-prometheus 2.4.1 and gtfs-realtime-bindings wheel
datahub/settings.py	Configured PrometheusBeforeMiddleware and PrometheusAfterMiddleware in MIDDLEWARE list and added django_prometheus to INSTALLED_APPS
datahub/urls.py	Exposed Prometheus metrics endpoint at root path via django_prometheus.urls
feed/apps.py	Added ready() method to auto-import metrics module on Django startup
feed/metrics.py	New file defining 7 Prometheus metrics for GTFS processing monitoring
feed/tasks.py	Instrumented get_schedule(), get_vehicle_positions(), and get_trip_updates() tasks with metrics recording, histogram timers, and enhanced error handling
feed/test_metrics.py	New unit test suite verifying metric definitions and basic increment/update functionality

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

feed/tasks.py

feed/test_metrics.py

feed/tasks.py

feed/test_metrics.py

feed/tasks.py

- Remove duplicate timestamp update in get_schedule - Improved error handling consistency - Code cleanup per review recommendations

BrandonTrigueros · 2025-12-25T05:10:57Z

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot · 2025-12-25T05:11:05Z

@BrandonTrigueros I've opened a new pull request, #50, to work on those changes. Once the pull request is ready, I'll request review from you.

[WIP] Add Prometheus metrics for GTFS processing monitoring

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 10 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

feed/tasks.py

feed/metrics.py

feed/tasks.py

BrandonTrigueros · 2025-12-25T05:17:27Z

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot · 2025-12-25T05:17:34Z

@BrandonTrigueros I've opened a new pull request, #51, to work on those changes. Once the pull request is ready, I'll request review from you.

…nbounded cardinality, and security issues Co-authored-by: BrandonTrigueros <[email protected]>

Co-authored-by: BrandonTrigueros <[email protected]>

fix: Resolve syntax errors, unbounded cardinality, and security issues in GTFS metrics

Co-authored-by: Copilot <[email protected]>

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

feed/tasks.py

feed/test_metrics.py

feed/tasks.py

BrandonTrigueros · 2025-12-27T18:56:21Z

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot · 2025-12-27T18:56:29Z

@BrandonTrigueros I've opened a new pull request, #52, to work on those changes. Once the pull request is ready, I'll request review from you.

… labels Co-authored-by: BrandonTrigueros <[email protected]>

fix: Improve GTFS metrics reliability and test accuracy

BrandonTrigueros requested review from Copilot and fabianabarca December 25, 2025 04:51

BrandonTrigueros added enhancement New feature or request monitoring labels Dec 25, 2025

Copilot started reviewing on behalf of BrandonTrigueros December 25, 2025 04:52 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

BrandonTrigueros self-assigned this Dec 25, 2025

fix: Address Copilot review feedback

e7f6d2b

- Remove duplicate timestamp update in get_schedule - Improved error handling consistency - Code cleanup per review recommendations

BrandonTrigueros requested a review from Copilot December 25, 2025 05:05

Copilot started reviewing on behalf of BrandonTrigueros December 25, 2025 05:06 View session

Initial plan

fcfb80f

Copilot AI mentioned this pull request Dec 25, 2025

[WIP] Add Prometheus metrics for GTFS processing monitoring #50

Merged

Merge pull request #50 from simovilab/copilot/sub-pr-49

5407fb5

[WIP] Add Prometheus metrics for GTFS processing monitoring

Copilot AI reviewed Dec 25, 2025

View reviewed changes

Initial plan

2eb9995

Copilot AI mentioned this pull request Dec 25, 2025

fix: Resolve syntax errors, unbounded cardinality, and security issues in GTFS metrics #51

Merged

Copilot AI and others added 3 commits December 25, 2025 05:21

fix: Address code review feedback - fix syntax errors, indentation, u…

bd1ef48

…nbounded cardinality, and security issues Co-authored-by: BrandonTrigueros <[email protected]>

refactor: Use specific exception types for better error handling

6f3dea1

Co-authored-by: BrandonTrigueros <[email protected]>

Merge pull request #51 from simovilab/copilot/sub-pr-49-again

05dbe82

fix: Resolve syntax errors, unbounded cardinality, and security issues in GTFS metrics

BrandonTrigueros requested a review from Copilot December 25, 2025 05:27

Copilot started reviewing on behalf of BrandonTrigueros December 25, 2025 05:27 View session

Apply suggestion from @Copilot

a8555ba

Co-authored-by: Copilot <[email protected]>

Copilot AI reviewed Dec 25, 2025

View reviewed changes

feed/tasks.py Outdated Show resolved Hide resolved

feed/tasks.py Outdated Show resolved Hide resolved

feed/test_metrics.py Outdated Show resolved Hide resolved

feed/tasks.py Show resolved Hide resolved

Initial plan

bd6d956

Copilot AI mentioned this pull request Dec 27, 2025

fix: Improve GTFS metrics reliability and test accuracy #52

Merged

Copilot AI and others added 2 commits December 27, 2025 18:59

fix: Address code review feedback - add error handling and fix metric…

124fad5

… labels Co-authored-by: BrandonTrigueros <[email protected]>

Merge pull request #52 from simovilab/copilot/sub-pr-49-another-one

6dc0b36

fix: Improve GTFS metrics reliability and test accuracy

feat: Add Prometheus metrics for GTFS processing monitoring #49

Are you sure you want to change the base?

feat: Add Prometheus metrics for GTFS processing monitoring #49

Uh oh!

Conversation

BrandonTrigueros commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request: GTFS Monitoring Metrics Implementation

Resumen

Cambios Realizados

1. Integración de django-prometheus

2. Definición de Métricas GTFS

Counters (acumuladores)

Gauge (valores actuales)

Histogram (distribución temporal)

3. Instrumentación de Tareas Celery

4. Inicialización Automática

5. Suite de Tests

Testing

Ejemplo de Output de Métricas

Uso en Producción

Integración con Grafana

Queries Útiles

Archivos Modificados

Checklist

Próximos Pasos (Opcional)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BrandonTrigueros commented Dec 25, 2025

Uh oh!

Copilot AI commented Dec 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BrandonTrigueros commented Dec 25, 2025

Uh oh!

Copilot AI commented Dec 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BrandonTrigueros commented Dec 27, 2025

Uh oh!

Copilot AI commented Dec 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

BrandonTrigueros commented Dec 25, 2025 •

edited

Loading