Incident creation when nobody is on duty

In our small team we are using PagerDuty as a notification channel for monitoring system. Basically, Grafana parses the logs and when a faulty situation is detected it sends an alert to PagerDuty that, in turn, notifies developer on duty via phone and Slack. It works perfectly fine during the day time when we have a person assigned in the schedule. However, during the night time when nobody is in the service schedule, the alert seems to be just dropped. To overcome that, we are using secondary notification channel directly into Slack, but that creates duplicates during the day time. :disappointed:

Even though the night incidents are not critical due to the nature of our service, it still would be nice to have those alerts documented as PagerDuty incidents (and auto-assign it to the person who’s shift comes next). That way we’d have PagerDuty’s list of incidents as a single reference and track the progress there.

Is it possible to configure a service in PagerDuty that way? A kind of obvious hack that comes to my mind is to create a dummy user account for the nights, but may be there is a better solution?

Any idea of suggestion is very much appreciated. :pray: Thanks!

Hi @VictorPavlushkov,

I’d recommend setting up SupportHours for your Service, having someone On-call for the full 24 hours but define off-hours as Low Urgency, and have everyone use Low Urgency notifications that aren’t intrusive - email, push notification if you really want to know about it when it happens.

Note that your Alerts aren’t being dropped, they’re being Suppressed, and will appear in the Alerts tab (but won’t create an Incident).

Hope that helps,
@simonfiddaman

Hi @VictorPavlushkov,

Configuring services using support hours should work for you.

Please do take a look at the article Defined Support Hours for details on how to get that set up.

Cheers