Still Alerted Outside of On Call Schedule

Had an alert on a server from our SolarWinds integration last night. SW sent the alert, PagerDuty accepted the alert, and sent it to the proper Escalation Policy. That policy has a stage 1 of alerting the On Call schedule. And that On Call schedule is only set to run from 3am to 11:30pm - but the alert fired off at 1:18 and 1:19 am, outside of the On Call schedule. It also skipped step 1 of just alerting the On Call (me) and it stepped forward to Step 2 which is to alert 3 people because the first one didn’t respond in 30 minutes… anyone know why the alert would still fire if the On Call schedule isn’t set to start until 3am?

Thanks!

Are you using Support Hours on the PagerDuty Service or just relying on the Escalation Policy and Schedules on each layer?

As described here https://support.pagerduty.com/docs/escalation-policies#escalation-targets-and-limitations, “If nobody is on-call in the first level, we will assign the incident to the next escalation level with an on-call responder.”

May also want to use a rule to set things to Low Urgency during off-hours to prevent escalation up the Escalation Policy.

Doug

Hi Michael,

It does sound like there was a user on-call on the escalation policy which caused the incident to trigger despite level 1 being off-call. The incident would escalate to the next available level if there are gaps in preceding levels.

If you’d like us to take a deeper look into the specific incident, we would be happy to take a look and advise further.

We wouldn’t ask you for the incident here but if you could reach out to Support over email with the incident and perhaps a reference to this Community Post, that would be great.