prometheus pagerduty_config 'details' section

Hello, I have set up prometheus k8s integration with Pager Duty however I am struggling to see how to customise the alert notification. I can see that the following is set by default here;

[ details: { <string>: <tmpl_string>, ... } | default = {
  firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
  resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
  num_firing:   '{{ .Alerts.Firing | len }}'
  num_resolved: '{{ .Alerts.Resolved | len }}'
} ]

I am a newbie to helm and go templating, however I understand that this ‘template call’ requires that the named template ‘pagerduty.default.instances’ is already defined using ‘define’. I can see that other templates are also called in the examples given in the prometheus documentation, e.g;

pagerduty.default.client
pagerduty.default.clientURL
pagerduty.default.description

I’m struggling to understand where these templates are defined, and how they relate to the .Alerts synthetic time series - those metrics which are created when alerts are fired if I understand correctly.
Ideally what I want is for the alert to display;

  • the ‘alertname’
  • the cluster (I can set this using external_lables)?
  • the podname ($label.pod)
  • the value
  • icon and links to alertmanager to silence alerts

I would be very grateful for some insight here, as I’ve been banging my head on this for weeks!

Thank you :pray:

It looks like some PagerDuty-related templates are defined in template/default.tmp - on these (https://github.com/prometheus/alertmanager/blob/d6e758ad3792ce6a78032f6d2ae706f367422e35/template/default.tmpl#L27-L30) lines specifically. I would suggest reaching out to Prometheus support for information on how they related to .Alerts synthetic time series and the proper syntax to display specific fields, but based on their documentation, I believe that alert name could be defined as {{ .CommonLabels.alertname }}, while other fields would need to be added as external labels (some helpful details here - https://github.com/prometheus-operator/prometheus-operator/issues/455) and accessed via $externalLabels variable.

Thanks for that… that is most helpful after reading through the README and https://github.com/prometheus/alertmanager/issues/259. It looks like;

“The default.tmpl is compiled into the binary. You cannot overwrite anything specified in there.
You have to create a new template file with custom template names and reference those in the configuration.”

Which is what I suspected but it was not documented anywhere. Appreciate your help, thanks!