We are after a similar functionality i.e. create an incident if incidents haven’t been created on a service in a specific timeframe, as this could indicate that there is a problem with the event arriving in PagerDuty.

In the meantime we are thinking of sending a “heartbeat” event every x minutes, which would then get routed to a service and then write a REST API query which would be scheduled to run and check the number of triggered incidents on the service. Then alert accordingly if they fall below an appropriate number via email or alert into one of our other monitoring tools.

We cannot use DeadMansSnitch as we believe it sends data to an external site and also requires licences.

Hi!

PagerDuty is not set up to alert on absence. You’d want to look at your monitoring tools for a solution using those types of triggers. Dead Man’s Snitch is common for single solutions, but DataDog and other tools also provide methods for monitoring the absence of an event.

Chris, I don’t think I’d recommend sending extra events to PagerDuty if they aren’t indicative of an actual problem on a service. That’s a lot of noise and extraneous data for human responders; all of those artificial incidents still have to be resolved and will be compiled to the account metrics. Where are your events getting lost? Are your inputs not receiving successful returns from PagerDuty, or are they getting lost in Event Orchestrations somewhere? I don’t know if integrations are set up to retry, but our Professional Services or Support folks may be able to help you do more troubleshooting there.

You can use the /incidents endpoint in the API to query for the number of incidents on a specific service in a given timeframe. The since and until options accept timestamps, and the total number of incidents will be in the return body as a top level key if you set total to true. There’s more about that endpoint in the developer docs. https://developer.pagerduty.com/api-reference/9d0b4b12e36f9-list-incidents but I had something similar in my github and threw this together as an example: https://github.com/lnxchk/pdgarage-samples/blob/main/incidents_since.sh