Alertmanager not setting `severity` when routing alerts to PagerDuty


I’m using Prometheus Alertmanager v0.21.0 to send alerts to PagerDuty. Everything is working fine, except the Severity I’m getting in PagerDuty does not match what I’m setting in alertmanager.

It’s like PD is ignoring the receivers[].pagerduty_configs[].severity key. Here’s my alertmanager ‘receivers’ block. Any suggestions?

- name: pagerduty-default-global-ruleset
  - service_key: <secret>
    description: '{{ .CommonAnnotations.description }}'
    severity: warning
      cluster: '{{ .GroupLabels.cluster }}'
      environment: '{{ .GroupLabels.environment }}'
      namespace: '{{ .GroupLabels.namespace }}'

Maybe it needs to be ‘warning’ with single ticks?

Kubernetes doesn’t distinguish between warning and 'warning' (they are both just strings).

I guess there’s a bug in alertmanager, or some problem with the alertmanager > PagerDuty integration. Will dig into the code.

@dmcclure Is there a way I can see the entire payload being sent to PagerDuty (from my alertmanager instance) ?

This is resolved. The issue was use of service_key in my receiver config (which hits the V1 API I assume). Switching to routing_key (to use the PagerDuty V2 API) resolved the issue.

1 Like