Skip to main content

Hi,


I have a noisy alarm coming from prometheus that I am not able to silence and is well documented as being unable to turn off -> https://github.com/prometheus-community/helm-charts/issues/1773#issuecomment-1201724576


How am I able to have pagerduty stop sending alerts for this particular alarm?


The alarms looks something like this:

Summary


[FIRING:1] (InfoInhibitor pending config-reloader prometheus-nodes-api alertmanager-prometheus-nodes-api-0 kube-system/prometheus-operator-kube-p-prometheus none)

With some details coming through pagerduty like this:


Labels:

 - alertname = InfoInhibitor

 - alertstate = pending

 - container = config-reloader

 - namespace = prometheus-nodes-api

 - pod = alertmanager-prometheus-nodes-api-0

 - prometheus = kube-system/prometheus-operator-kube-p-prometheus

 - severity = none

Annotations:

 - description = This is an alert that is used to inhibit info alerts.

By themselves, the info-level alerts are sometimes very noisy, but they are relevant when combined with

other alerts.

This alert fires whenever there's a severity="info" alert, and stops firing when another alert with a

severity of 'warning' or 'critical' starts firing on the same namespace.

This alert should be routed to a null receiver and configured to inhibit alerts with severity="info".



 - runbook_url = https://runbooks.prometheus-operator.dev/runbooks/general/infoinhibitor

 - summary = Info-level alert inhibition.

Source: http://prometheus-operator-kube-p-prometheus.kube-system:9090/graph?g0.expr=ALERTS%7Bseverity%3D%22info%22%7D+%3D%3D+1+unless+on%28namespace%29+ALERTS%7Balertname%21%3D%22InfoInhibitor%22%2Calertstate%3D%22firing%22%2Cseverity%3D~%22warning%7Ccritical%22%7D+%3D%3D+1&g0.tab=1

I tried setting an event rule:


body  contains  InfoInhibitor

Suppress + Then stop processing

But still I get this alert.


I then tried setting an ignore rule in event orchestration looking like this:


event.summary matches part 'InfoInhibitor'

Still getting this alert…


All help appreciated.

I can’t see the word ‘InfoInhibitor’ in the summary field of your payload - it has ‘Info-level alert inhibition’

Hi @ChrisFitch - thanks for reply.


Forgot to mention that I already tried Info-level alert inhibition. and shorter versions of that like just inhibition. in event orchestration and still got the alarms.


Also not sure what to put for event rules but have used those keywords in there and again get alarms.


I think the summary in the message is different than the summary that is used by PD event orchestration to filter on.


Any other suggestion?

Hi @robert cannon


Apologies for my delay in responding but I have only just started using Orchestrations, after migrating from Global Event Rules so needed to familiarise myself with the functionality.


I know it might be a silly question but are you sure under the “Incident Data” tab of your event rule you’ve selected “Suppress incident”?


Failing that, when you edit your Routing Rule it will show the last 5 events that have hit the Orchestration and from there you can click on the summary line and it will build the condition for you (I know you might not want to do it at the routing level, but at least you can be sure the format of your condition will be correct and you can then build out your service-level Orchestration).

Reply


Terms & ConditionsCookie settings