In our small team we are using PagerDuty as a notification channel for monitoring system. Basically, Grafana parses the logs and when a faulty situation is detected it sends an alert to PagerDuty that, in turn, notifies developer on duty via phone and Slack. It works perfectly fine during the day time when we have a person assigned in the schedule. However, during the night time when nobody is in the service schedule, the alert seems to be just dropped. To overcome that, we are using secondary notification channel directly into Slack, but that creates duplicates during the day time.
Even though the night incidents are not critical due to the nature of our service, it still would be nice to have those alerts documented as PagerDuty incidents (and auto-assign it to the person who’s shift comes next). That way we’d have PagerDuty’s list of incidents as a single reference and track the progress there.
Is it possible to configure a service in PagerDuty that way? A kind of obvious hack that comes to my mind is to create a dummy user account for the nights, but may be there is a better solution?
Any idea of suggestion is very much appreciated. Thanks!