Enhancement Proposal: pagerdutyv2 service to support custom_details, links, and Change Events API

[chauncey-garrett]

Problem

Current pagerdutyv2 service only supports 7 basic fields and cannot utilize PagerDuty's Change Events API for deployment tracking. This limits incident context and prevents PagerDuty ML from correlating deployments with incidents.

Proposed Enhancement

Add support for:

1. custom_details - JSON object for rich incident context
2. links - Array of links for incident navigation
3. images - Array of images for visual context
4. dedup_key - Custom deduplication key
5. timestamp - Event timestamp
6. Change Events API endpoint (/v2/change/enqueue)

Use Case

We have 100 microservices with independent on-call rotations. PagerDuty ML requires custom_details (cluster, namespace, stage, etc.) to correlate deployment changes with incidents. Current workaround uses custom webhooks.

Webhook examples

Alert Events Template (with urgency selection):

template.app-sync-failed: |
  webhook.pagerduty-alerts: |
    method: POST
    body: |
      {
        "routing_key": "{{if eq .context.stage \"production\"}}{{if index .app.metadata.annotations \"pagerduty.io/high-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/high-urgency-key\")}}{{else}}{{.context.pagerdutyAlertsRoutingKey}}{{end}}{{else}}{{if index .app.metadata.annotations \"pagerduty.io/low-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/low-urgency-key\")}}{{else}}{{.context.pagerdutyAlertsRoutingKey}}{{end}}{{end}}",
        "event_action": "trigger",
        "dedup_key": "{{.app.metadata.name}}-sync-failed-{{.context.stage}}",
        "payload": {
          "summary": "🚨 {{.app.metadata.name}} sync failed in {{.context.stage}}",
          "severity": "error",
          "source": "{{.context.instance}}",
          "component": "{{.app.metadata.name}}",
          "group": "{{.app.spec.project}}",
          "custom_details": {
            "app_name": "{{.app.metadata.name}}",
            "project": "{{.app.spec.project}}",
            "stage": "{{.context.stage}}",
            "cluster": "{{.app.spec.destination.server}}",
            "namespace": "{{.app.spec.destination.namespace}}",
            "error_message": "{{.app.status.operationState.message}}"
          }
        },
        "links": [
          {
            "href": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
            "text": "View Sync Operation"
          }
        ]
      }

Change Events Template (to both services):

template.app-deployed: |
  webhook.pagerduty-changes-high: |
    method: POST
    body: |
      {
        "routing_key": "{{if index .app.metadata.annotations \"pagerduty.io/high-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/high-urgency-key\")}}{{else}}{{.context.pagerdutyChangesRoutingKey}}{{end}}",
        "payload": {
          "summary": "🚀 {{.app.metadata.name}} deployed to {{.context.stage}}",
          "timestamp": "{{.app.status.operationState.finishedAt}}",
          "source": "{{.context.instance}}",
          "custom_details": {
            "app_name": "{{.app.metadata.name}}",
            "project": "{{.app.spec.project}}",
            "stage": "{{.context.stage}}",
            "cluster": "{{.app.spec.destination.server}}",
            "namespace": "{{.app.spec.destination.namespace}}",
            "error_message": "{{.app.status.operationState.message}}"
          }
        }
      }

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: example-api
  namespace: argo-cd
  annotations:
    # PagerDuty routing (required)
    pagerduty.io/high-urgency-key: "PAGERDUTY_ACHIEVEMENTS_HIGH_KEY"
    pagerduty.io/low-urgency-key: "PAGERDUTY_ACHIEVEMENTS_LOW_KEY"
    
    # Optional: Override which events go to high urgency
    # pagerduty.io/high-urgency-events: "on-sync-failed,on-deleted"

References

• PagerDuty Events API v2: https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgw
• PagerDuty Change Events: https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgy

[chauncey-garrett]

Here's a (more verbose) proposal from Claude--take it for what it's worth ;)

ArgoCD Notifications: Enhance PagerDuty Service

Summary

Enhance the built-in pagerdutyv2 notification service to support critical PagerDuty Events API v2 fields that are currently missing, preventing users from creating rich, contextual incidents and leveraging PagerDuty's change tracking capabilities.

Problem Statement

The current pagerdutyv2 service implementation is limited compared to what the PagerDuty Events API v2 supports. Users who need advanced incident management capabilities must resort to custom webhooks, losing the benefits of ArgoCD's built-in service patterns and documentation.

Current Limitations

1. No custom_details support - Cannot add structured metadata (cluster, namespace, stage, commit info)
2. No links array support - Cannot provide multiple contextual links (ArgoCD UI, commits, runbooks, dashboards)
3. No images array support - Cannot attach visual context to incidents
4. No dedup_key customization - Limited control over incident deduplication
5. No Change Events API support - Cannot send deployment context without creating incidents

Impact

Without these fields:

• Incident responders lack context - No cluster/namespace/stage information in incidents
• Slow MTTR - Responders must manually find ArgoCD links, commit history, and runbooks
• No change correlation - PagerDuty ML cannot learn relationships between deployments and incidents
• Poor deduplication - Cannot customize incident grouping per environment/issue type
• Users implement workarounds - Organizations with 100+ microservices build custom webhook solutions

Proposed Enhancement

Add support for all PagerDuty Events API v2 fields while maintaining backward compatibility.

API Design

1. Custom Details

Add customDetails field to include structured metadata in incidents.

Template example:

template.app-sync-failed: |
  message: Sync failed for {{.app.metadata.name}}
  pagerdutyv2:
    summary: "🚨 {{.app.metadata.name}} sync failed in {{.context.stage}}"
    severity: error
    customDetails:
      app_name: "{{.app.metadata.name}}"
      project: "{{.app.spec.project}}"
      stage: "{{.context.stage}}"
      cluster: "{{.app.spec.destination.server}}"
      namespace: "{{.app.spec.destination.namespace}}"
      error_message: "{{.app.status.operationState.message}}"
      repository: "{{.app.spec.source.repoURL}}"
      revision: "{{.app.status.sync.revision}}"

PagerDuty payload:

{
  "routing_key": "...",
  "event_action": "trigger",
  "payload": {
    "summary": "🚨 my-service sync failed in production",
    "severity": "error",
    "custom_details": {
      "app_name": "my-service",
      "project": "default",
      "stage": "production",
      "cluster": "https://kubernetes.default.svc",
      "namespace": "production",
      "error_message": "sync operation failed: ...",
      "repository": "https://github.com/org/repo",
      "revision": "abc123def456"
    }
  }
}

2. Links Array

Add links field to provide multiple contextual URLs.

Template example:

template.app-sync-failed: |
  message: Sync failed for {{.app.metadata.name}}
  pagerdutyv2:
    summary: "🚨 {{.app.metadata.name}} sync failed"
    severity: error
    links:
      - href: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true"
        text: "View Sync Operation"
      - href: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}"
        text: "View Application"
      - href: "{{.context.runbookUrl}}/argocd-sync-failed"
        text: "Incident Runbook"
      - href: "{{.context.grafanaUrl}}/d/argocd?var-app={{.app.metadata.name}}"
        text: "View Metrics"

PagerDuty payload:

{
  "payload": {
    "summary": "🚨 my-service sync failed",
    "severity": "error"
  },
  "links": [
    {
      "href": "https://argocd.example.com/applications/my-service?operation=true",
      "text": "View Sync Operation"
    },
    {
      "href": "https://argocd.example.com/applications/my-service",
      "text": "View Application"
    },
    {
      "href": "https://runbooks.example.com/argocd-sync-failed",
      "text": "Incident Runbook"
    },
    {
      "href": "https://grafana.example.com/d/argocd?var-app=my-service",
      "text": "View Metrics"
    }
  ]
}

3. Images Array

Add images field to attach visual context (screenshots, architecture diagrams, etc.).

Template example:

template.app-health-degraded: |
  message: Health degraded for {{.app.metadata.name}}
  pagerdutyv2:
    summary: "⚠️ {{.app.metadata.name}} health degraded"
    severity: warning
    images:
      - src: "{{.context.grafanaUrl}}/render/d/argocd?var-app={{.app.metadata.name}}"
        href: "{{.context.grafanaUrl}}/d/argocd?var-app={{.app.metadata.name}}"
        alt: "Resource Health Dashboard"

PagerDuty payload:

{
  "payload": {
    "summary": "⚠️ my-service health degraded",
    "severity": "warning"
  },
  "images": [
    {
      "src": "https://grafana.example.com/render/d/argocd?var-app=my-service",
      "href": "https://grafana.example.com/d/argocd?var-app=my-service",
      "alt": "Resource Health Dashboard"
    }
  ]
}

4. Custom Deduplication Key

Add dedupKey field to control incident grouping.

Template example:

template.app-sync-failed: |
  message: Sync failed for {{.app.metadata.name}}
  pagerdutyv2:
    summary: "🚨 {{.app.metadata.name}} sync failed"
    severity: error
    dedupKey: "{{.app.metadata.name}}-sync-failed-{{.context.stage}}"

PagerDuty payload:

{
  "routing_key": "...",
  "event_action": "trigger",
  "dedup_key": "my-service-sync-failed-production",
  "payload": {
    "summary": "🚨 my-service sync failed",
    "severity": "error"
  }
}

Benefits:

• Multiple failures update one incident (not spam)
• Different issue types create separate incidents
• Per-environment isolation (production vs pre-production)

5. Change Events API Support

Add new pagerdutyv2-change service type for deployment context without creating incidents.

Service configuration:

notifications:
  notifiers:
    service.pagerdutyv2-change.deployments: |
      serviceKeys:
        my-team: $pagerduty-change-events-key

Template example:

template.app-deployed: |
  message: Deployed {{.app.metadata.name}}
  pagerdutyv2-change:
    summary: "🚀 {{.app.metadata.name}} deployed to {{.context.stage}}"
    timestamp: "{{.app.status.operationState.finishedAt}}"
    customDetails:
      app_name: "{{.app.metadata.name}}"
      stage: "{{.context.stage}}"
      revision: "{{.app.status.sync.revision}}"
      commit_author: "{{if call .repo.GetCommitMetadata .app.status.sync.revision}}{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}{{end}}"
      commit_message: "{{if call .repo.GetCommitMetadata .app.status.sync.revision}}{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}}{{end}}"
    links:
      - href: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}"
        text: "View in ArgoCD"

PagerDuty Change Events payload:

{
  "routing_key": "...",
  "payload": {
    "summary": "🚀 my-service deployed to production",
    "timestamp": "2025-11-10T14:30:00Z",
    "source": "Argo CD Production",
    "custom_details": {
      "app_name": "my-service",
      "stage": "production",
      "revision": "abc123def456",
      "commit_author": "John Doe",
      "commit_message": "Fix authentication bug"
    }
  },
  "links": [
    {
      "href": "https://argocd.example.com/applications/my-service",
      "text": "View in ArgoCD"
    }
  ]
}

Benefits:

• Appears as context on related incidents (within 24 hours)
• No alert fatigue from successful deployments
• PagerDuty ML learns which deployments correlate with incidents
• Deployment timeline visible in PagerDuty incident UI

Use Cases

Use Case 1: Enterprise with 100+ Microservices

Context: Organization has 100+ microservices, each with separate on-call rotations and PagerDuty services.

Current workaround: Custom webhook templates for every application.

With enhancement:

notifications:
  notifiers:
    service.pagerdutyv2.alerts: |
      serviceKeys:
        api-service: $api-service-pagerduty-key
        web-service: $web-service-pagerduty-key
        data-pipeline: $data-pipeline-pagerduty-key
        # ... 97 more services

  templates:
    template.app-sync-failed: |
      message: Sync failed
      pagerdutyv2:
        summary: "🚨 {{.app.metadata.name}} sync failed"
        severity: error
        dedupKey: "{{.app.metadata.name}}-sync-failed-{{.context.stage}}"
        customDetails:
          cluster: "{{.app.spec.destination.server}}"
          namespace: "{{.app.spec.destination.namespace}}"
          stage: "{{.context.stage}}"
          error_message: "{{.app.status.operationState.message}}"
        links:
          - href: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true"
            text: "View Sync Operation"
          - href: "{{.context.runbookUrl}}/argocd-sync-failed"
            text: "Incident Runbook"

Benefits:

• Single template for all services
• Rich incident context
• Proper deduplication per app/environment
• Quick access to ArgoCD and runbooks

Use Case 2: Multi-Cluster Environments

Context: Applications deployed across dev, staging, production clusters with different on-call teams.

Current limitation: Cannot include cluster/namespace/stage in incident details.

With enhancement:

template.app-health-degraded: |
  pagerdutyv2:
    summary: "⚠️ {{.app.metadata.name}} degraded in {{.context.stage}}"
    severity: warning
    dedupKey: "{{.app.metadata.name}}-health-{{.context.stage}}"
    customDetails:
      cluster: "{{.app.spec.destination.server}}"
      namespace: "{{.app.spec.destination.namespace}}"
      stage: "{{.context.stage}}"
      health_status: "{{.app.status.health.status}}"
      health_message: "{{.app.status.health.message}}"
    links:
      - href: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}"
        text: "View Application"
      - href: "{{.context.k8sDashboard}}/{{.app.spec.destination.namespace}}/{{.app.metadata.name}}"
        text: "View Kubernetes Resources"

Benefits:

• Responders immediately know which cluster/namespace
• Separate incidents per environment (no cross-environment noise)
• Direct links to cluster dashboards

Use Case 3: Correlating Deployments with Incidents

Context: Team wants to understand if recent deployments caused incidents.

Current limitation: No way to send deployment context without creating incidents.

With enhancement:

# Send Change Events for successful deployments
template.app-deployed: |
  pagerdutyv2-change:
    summary: "🚀 {{.app.metadata.name}} deployed"
    timestamp: "{{.app.status.operationState.finishedAt}}"
    customDetails:
      revision: "{{.app.status.sync.revision}}"
      commit_author: "{{...}}"
      commit_message: "{{...}}"

# Send Alert Events for failures
template.app-sync-failed: |
  pagerdutyv2:
    summary: "🚨 {{.app.metadata.name}} sync failed"
    severity: error

Benefits:

• Deployment context appears on incidents automatically
• PagerDuty ML learns correlations
• Responders see recent changes when investigating
• No alert fatigue from successful deployments

Implementation Considerations

1. Backward Compatibility

All new fields are optional - existing configurations continue working unchanged.

Current template (still works):

template.app-sync-failed: |
  message: Sync failed
  pagerdutyv2:
    summary: "Sync failed for {{.app.metadata.name}}"
    severity: error

Enhanced template (opt-in):

template.app-sync-failed: |
  message: Sync failed
  pagerdutyv2:
    summary: "Sync failed for {{.app.metadata.name}}"
    severity: error
    customDetails:
      cluster: "{{.app.spec.destination.server}}"
    links:
      - href: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}"
        text: "View Application"
    dedupKey: "{{.app.metadata.name}}-sync-failed"

2. API Mapping

Map template fields directly to PagerDuty Events API v2:

Template Field	PagerDuty API Field	Type	Required
summary	payload.summary	string	Yes
severity	payload.severity	enum	Yes
customDetails	payload.custom_details	object	No
links	links	array	No
images	images	array	No
dedupKey	dedup_key	string	No
component	payload.component	string	No
group	payload.group	string	No
class	payload.class	string	No

3. Change Events Service

Implement pagerdutyv2-change as new service type:

// Existing
type PagerDutyV2Service struct {
    // Alert Events (creates incidents)
}

// New
type PagerDutyV2ChangeService struct {
    // Change Events (deployment context, no incidents)
}

Different endpoints:

• Alert Events: https://events.pagerduty.com/v2/enqueue
• Change Events: https://events.pagerduty.com/v2/change/enqueue

Different payloads:

• Alert Events: Requires event_action (trigger/acknowledge/resolve)
• Change Events: No event_action field, requires timestamp

4. Template Syntax

Support both inline and block-style YAML for complex objects:

Inline:

pagerdutyv2:
  customDetails:
    cluster: "{{.app.spec.destination.server}}"
    namespace: "{{.app.spec.destination.namespace}}"

Block-style (for complex scenarios):

pagerdutyv2:
  customDetails: |
    {
      "cluster": "{{.app.spec.destination.server}}",
      "namespace": "{{.app.spec.destination.namespace}}",
      "conditions": {{.app.status.conditions | toJson}}
    }

5. Error Handling

• Invalid JSON in customDetails: Log warning, send incident without custom_details
• Invalid URL in links: Log warning, omit invalid link
• Missing required fields: Fail validation at template load time
• API errors: Retry with exponential backoff (existing behavior)

Documentation Updates

1. ArgoCD Notifications Documentation

Add new section: "PagerDuty Events API v2 Features"

• Custom details for incident context
• Links array for quick access
• Images for visual context
• Custom deduplication keys
• Change Events for deployment tracking

2. Template Examples

Provide complete examples for common scenarios:

• Basic alert (backward compatible)
• Alert with full context (custom_details, links)
• Change event for deployments
• Multi-environment setup with deduplication

3. Migration Guide

For users currently using custom webhooks:

• Side-by-side comparison
• Migration steps
• Testing procedures

Alternatives Considered

Alternative 1: Do Nothing (Status Quo)

Pros:

• No development effort
• No breaking changes

Cons:

• Users forced to use custom webhooks
• Inconsistent notification patterns across integrations
• Lost opportunity for PagerDuty best practices

Alternative 2: Custom Webhook Only

Pros:

• Already possible today
• Maximum flexibility

Cons:

• More complex configuration per organization
• Not documented/supported
• Misses benefit of built-in service patterns

Alternative 3: Partial Implementation (custom_details only)

Pros:

• Smaller scope
• Faster to implement

Cons:

• Still requires custom webhooks for links/images
• Doesn't fully leverage PagerDuty API
• Users still need workarounds

Success Metrics

• Adoption: 25%+ of pagerdutyv2 users adopt new fields within 6 months
• Custom webhook reduction: 50%+ reduction in custom webhook usage for PagerDuty
• MTTR improvement: User-reported faster incident resolution with contextual links
• Community feedback: Positive reception on GitHub issue/PR

References

• PagerDuty Events API v2
• PagerDuty Change Events API
• ArgoCD Notifications Documentation
• ArgoCD PagerDuty Service

Implementation Checklist

• Add customDetails field support to pagerdutyv2 service
• Add links array support to pagerdutyv2 service
• Add images array support to pagerdutyv2 service
• Add dedupKey field support to pagerdutyv2 service
• Implement pagerdutyv2-change service for Change Events API
• Add unit tests for new fields
• Add integration tests with PagerDuty API
• Update ArgoCD documentation with examples
• Create migration guide from custom webhooks
• Add example templates to repository