I have been attempting to setup alerting with dyanamic notifications with prometheus alertmanager. I spent the last two days thinking I was doing something wrong, I could not see the severity field in the incident payload. The urgency would be set correctly if I set the severity under the pagerduty_config to ‘critical’, but I never saw the severity field in the incident message payload.

I then set the severity in the pagerduty_config to ‘warning’ and the incident showed up with a ‘Low’ urgency, so clearly the pagerduty_config severity field is being considered in the code logic when generating the pagerduty incident. However, I still do not see the severity field in the incident’s message payload.

Is this expected?

For references see below

receivers:

• name: default-receiver
• name: pagerduty
  pagerduty_configs:

  • description: ‘[{{ .Status | toUpper }}{{ if eq .Status “firing” }}:{{ .Alerts.Firing
    | len }}{{ end }}] {{ .CommonAnnotations.summary }}’
    routing_key: KEY
    severity: ‘info’
    route:
    group_interval: 5m
    group_wait: 10s
    receiver: default-receiver
    repeat_interval: 3h
    routes:

  • matchers:

    • team = sre
      receiver: pagerduty

    # the generated incident payload
    {
      "client": "Alertmanager",
      "client_url": "http://prometheus-alertmanager-0:9093/#/alerts?receiver=pagerduty",
      "contexts": null,
      "description": "[FIRING:1] Spark Operator available replicas is not the same as desired replicas.",
      "event_type": "trigger",
      "incident_key": "KEY",
      "service_key": "KEY",
      "details": {
        "firing": "Labels:
     - alertname = SparkOperatorDown
     - app_kubernetes_io_component = metrics
     - app_kubernetes_io_instance = prometheus
     - app_kubernetes_io_managed_by = Helm
     - app_kubernetes_io_name = kube-state-metrics
     - app_kubernetes_io_part_of = kube-state-metrics
     - app_kubernetes_io_version = 2.8.0
     - cluster = CLUSTERNAME
     - deployment = spark-spark-operator
     - helm_sh_chart = kube-state-metrics-4.30.0
     - instance =IP:PORT
     - job = kubernetes-service-endpoints
     - namespace = spark-operator
     - node = NODE
     - service = prometheus-kube-state-metrics
     - severity = page
     - team = sre
    Annotations:
     - summary = Spark Operator available replicas is not the same as desired replicas.
    Source: http://prometheus-server-8577d64b5d-4hbhz:9090/graph?g0.expr=kube_deployment_status_replicas_available%7Bdeployment%3D%22spark-spark-operator%22%2Cnamespace%3D%22spark-operator%22%7D+%2F+kube_deployment_status_replicas%7Bdeployment%3D%22spark-spark-operator%22%2Cnamespace%3D%22spark-operator%22%7D+%21%3D+1&g0.tab=1
    ",
        "num_firing": "1",
        "num_resolved": "0",
        "resolved": ""
      }
    }