Skip to main content

I have been attempting to setup alerting with dyanamic notifications with prometheus alertmanager. I spent the last two days thinking I was doing something wrong, I could not see the severity field in the incident payload. The urgency would be set correctly if I set the severity under the pagerduty_config to ‘critical’, but I never saw the severity field in the incident message payload.

I then set the severity in the pagerduty_config to ‘warning’ and the incident showed up with a ‘Low’ urgency, so clearly the pagerduty_config severity field is being considered in the code logic when generating the pagerduty incident. However, I still do not see the severity field in the incident’s message payload.

Is this expected?

For references see below

receivers:

  • name: default-receiver
  • name: pagerduty
    pagerduty_configs:
    • description: ‘[{{ .Status | toUpper }}{{ if eq .Status “firing” }}:{{ .Alerts.Firing
      | len }}{{ end }}] {{ .CommonAnnotations.summary }}’
      routing_key: KEY
      severity: ‘info’
      route:
      group_interval: 5m
      group_wait: 10s
      receiver: default-receiver
      repeat_interval: 3h
      routes:

    • matchers:

      • team = sre
        receiver: pagerduty
      # the generated incident payload
      {
        "client": "Alertmanager",
        "client_url": "http://prometheus-alertmanager-0:9093/#/alerts?receiver=pagerduty",
        "contexts": null,
        "description": "[FIRING:1] Spark Operator available replicas is not the same as desired replicas.",
        "event_type": "trigger",
        "incident_key": "KEY",
        "service_key": "KEY",
        "details": {
          "firing": "Labels:
       - alertname = SparkOperatorDown
       - app_kubernetes_io_component = metrics
       - app_kubernetes_io_instance = prometheus
       - app_kubernetes_io_managed_by = Helm
       - app_kubernetes_io_name = kube-state-metrics
       - app_kubernetes_io_part_of = kube-state-metrics
       - app_kubernetes_io_version = 2.8.0
       - cluster = CLUSTERNAME
       - deployment = spark-spark-operator
       - helm_sh_chart = kube-state-metrics-4.30.0
       - instance =IP:PORT
       - job = kubernetes-service-endpoints
       - namespace = spark-operator
       - node = NODE
       - service = prometheus-kube-state-metrics
       - severity = page
       - team = sre
      Annotations:
       - summary = Spark Operator available replicas is not the same as desired replicas.
      Source: http://prometheus-server-8577d64b5d-4hbhz:9090/graph?g0.expr=kube_deployment_status_replicas_available%7Bdeployment%3D%22spark-spark-operator%22%2Cnamespace%3D%22spark-operator%22%7D+%2F+kube_deployment_status_replicas%7Bdeployment%3D%22spark-spark-operator%22%2Cnamespace%3D%22spark-operator%22%7D+%21%3D+1&g0.tab=1
      ",
          "num_firing": "1",
          "num_resolved": "0",
          "resolved": ""
        }
      }
      
Be the first to reply!