Contract Health Checks and Monitoring #803

Title: feat: implement contract health checks and alert on indexing failures

Labels: enhancement observability reliability
Complexity: medium
Branch: feat/contract-health-monitoring

Problem Context

A contract's indexing can silently fail: the Soroban RPC goes down, the contract becomes inactive on testnet, or a new event type breaks the ABI decoder. Operators don't know until a user complains. Proactive health checks with alerting enable rapid response to issues before they impact users.

Scope

Included:

• ContractHealthCheck model tracking status: healthy, degraded, failed
• Periodic task: ping Soroban RPC for each contract, verify latest events are fresh
• Alert rules: contract not indexed for >30 minutes, ABI decode errors spike
• Health endpoint: GET /api/health/?contract_id=X returning status and last event time
• Admin dashboard showing contract health status

Not included:

• Automatic contract pause/suspension (Phase 2)
• Automated alert routing to Slack/Email (see Issue Build Webhook Subscription Manager UI for managing event notifications #29)

Implementation Guidelines

Files to update:

• django-backend/soroscan/ingest/models.py — add ContractHealthCheck
• django-backend/soroscan/ingest/tasks.py — add check_contract_health periodic task
• django-backend/soroscan/ingest/views.py — add health endpoint
• django-backend/soroscan/ingest/admin.py — register health model

Model sketch:

class ContractHealthCheck(models.Model):
    class Status(models.TextChoices):
        HEALTHY = 'healthy'
        DEGRADED = 'degraded'
        FAILED = 'failed'

    contract = models.OneToOneField(TrackedContract, on_delete=models.CASCADE)
    status = models.CharField(max_length=16, choices=Status.choices)
    last_event_time = models.DateTimeField(null=True)
    minutes_since_last_event = models.IntegerField()
    error_message = models.TextField(blank=True)
    checked_at = models.DateTimeField(auto_now=True)

Health check task:

@app.task
def check_contract_health():
    for contract in TrackedContract.objects.filter(is_active=True):
        latest_event = ContractEvent.objects.filter(
            contract=contract
        ).latest('created_at')
        
        minutes_since = (now() - latest_event.created_at).total_seconds() / 60
        
        if minutes_since > 30:
            health = ContractHealthCheck.objects.get_or_create(contract=contract)[0]
            health.status = 'degraded' if minutes_since < 120 else 'failed'
            health.minutes_since_last_event = int(minutes_since)
            health.error_message = f"No events for {minutes_since:.0f} minutes"
            health.save()

Health endpoint:

GET /api/contracts/{id}/health/
{
  "status": "healthy",
  "lastEventTime": "2026-03-22T10:15:00Z",
  "minutesSinceLastEvent": 2,
  "errorMessage": null
}

Constraints:

• Health checks must complete in <5 seconds per contract
• Gracefully handle RPC timeouts (don't block other checks)
• Health status change must trigger alert (see Issue Build Webhook Subscription Manager UI for managing event notifications #29)
• Thresholds must be configurable via environment

Acceptance Criteria

• ContractHealthCheck model with migration added
• check_contract_health task runs every 5 minutes
• Contract status set to degraded if no events for >30 minutes, failed if >2 hours
• GET /api/contracts/{id}/health/ returns current health status
• Admin dashboard displays health status for all contracts
• Health status changes trigger alerts (integration with Issue Build Webhook Subscription Manager UI for managing event notifications #29)
• Integration test verifies health check logic