feat(ci): post-deploy verification for managed and enterprise orgs

[joelorzet]

What

A separate pipeline that runs after the whole prod CI Pipeline finishes green and verifies that managed and enterprise organizations are executing workflows correctly on the just-shipped build. If an org regresses, it pages that client's own PagerDuty service. Client identities are sourced from AWS Parameter Store, not hardcoded in this (public) repo.

How it works

• Triggered via workflow_run on the "CI Pipeline" workflow completing with conclusion success on prod. It fires only once the whole run (build, e2e, deploy, release, docs-sync) is green, not after any single step.
• Runs inside the cluster (the internal routes are HMAC authed and WAF blocked on the public hostname): it signs the request like deploy/scripts/digest-cron.sh and curls the in-pod service URL, polling once a minute for at most 5 minutes and stopping early on the first flagged org.
• The endpoint GET /api/internal/post-deploy-verification resolves the cohort (managed slugs plus any enterprise-plan org) gated to orgs with a schedule due in the window, then flags an org that has no executions, any system_error, or an error rate over the threshold.

Monitoring window

Polls every minute for 5 minutes, anchored at the check start so only executions on the new build count. This was confirmed sufficient for the current managed clients given their execution cadence. The window and thresholds are query-param overridable from the workflow, so tuning does not need a code change.

Managed-client identities (Parameter Store, single source of truth)

• The hardcoded slug list is gone. MANAGED_ORGS_CONFIG (a JSON dict {"<slug>":{"pagerdutyRoutingKey":"..."}}) is injected at deploy from Parameter Store and read by both the metrics scraper (managed-client error gauge) and the verification route. Empty/unset yields no managed cohort (bounded gauge, safe default).
• Remaining client names were scrubbed from the backfill script, unit-test fixtures, and code comments.

Alerting (PagerDuty, per client)

• Replaces the Discord notification. On a flagged org the endpoint pages that org's own service via a per-client routing key (falling back to DEFAULT_PAGERDUTY_ROUTING_KEY for enterprise orgs without a dedicated service), deduped to one incident per org per deploy. Slugs and routing keys never enter the CI job.
• Severity: no executions and system_error are P2; a user-error-rate spike is P3. "No executions" is concluded on the loop's final poll.
• The CI job now only fails red on problems (aggregate counts); PagerDuty does the paging, and per-org detail stays in internal logs.

Data exposure

This repository is public, so Actions logs are world readable. The pipeline surfaces only aggregate counts. Per-org slugs, names, and sample error text are written to internal observability and PagerDuty only, never to CI logs.

Deploy notes

• Runtime env only (not NEXT_PUBLIC), so no Docker build arg is needed. Added to the keeperhub-stack values (prod + staging): MANAGED_ORGS_CONFIG + DEFAULT_PAGERDUTY_ROUTING_KEY on the app, and MANAGED_ORGS_CONFIG on the metrics-collector. The legacy standalone values under deploy/keeperhub/ and deploy/metrics-collector/ are not used by the stack deploy and were left untouched.
• Before merge/deploy: create the managed-orgs and default-pagerduty-routing-key Parameter Store entries per env, and confirm the app/metrics IRSA SSM policy covers the path prefix. The pipeline needs no new AWS permissions.
• Follow-up (infra repo): point the Grafana managed_org_slugs_regex at the same Parameter Store value so the cohort has one source across app and alerting.