I’m trying to configure Pager Duty as a sort of gate keeper to filter out low quality violations from New Relic ping tests. My goal is to trigger a Pager Duty alert through the escalation policy chain only if the server is down for more than 10 minutes. Right now we’re getting alerts in the middle of the night if the server is down for a few seconds.
I have my New Relic Alert Policy incident preferences set to open an incident every time a condition is violated and I’m running tests every 5 minutes. The Synthetic is set to search for an element that does not exist on the page to specifically trigger a violation for testing purposes only. I’ve configured an Event Rule so when the body contains “Ping” the following actions are performed:
* Route to [Backend Critical (New Relic)](https://24datainc.pagerduty.com/services/XXXX) * Suppressing until more than ` 2 alerts` received within `12 minutes` * Then stop processing
I am artificially forcing 2 violations in a row New Relic with the hope that Pager Duty will filter out the first warning and trigger an incident to the escalation policy on the second but it seems like
a. New Relic is only sending Pager Duty one notification and
b. Pager Duty is receiving that first notification from NR and immediately sending it through the escalation policy.
Any advice please?