How to notifiy stakeholders when an incident is created

questions

(Rich Navis) #1

I don’t see how this can be done with pagerduty, which is quite frustrating. We have many systems that are managed by both our systems and applications teams. Whenever a “systems” problem happens, we want to assign (and possibly wake up) a systems resource. We would ALSO like to inform that applications team about the problem when it happens automatically, not relying on a status update. We don’t want to wake the apps team… but just inform them. I can’t figure out how to do this in pd… Feature request?


(Crystal J. Mendoza) #2

Hi Rich,

These are things that are definitely possible!

Here are some articles from our Knowledge Base that I think would be of some assistance to you.

  • Adding Responders -Adding responders enables you to receive assistance from additional users with an incident response.
  • Configuring your escalation policy to notify 2 or more users/teams -To notify 2 or more users/teams at once, you will want to set up an escalation policy with multiple on-call schedules/users on the same escalation level.
  • Response Plays -This enables you to take a complex activity, like assembling a response team of multiple on-calls and an incident commander, and make it available to anyone that needs to use it.
  • Communicate Effectively with Stakeholders - This how-to guide covers using PagerDuty to effectively communicate with stakeholders during an incident that can also lead to more efficient incident resolution.

Please let us know if you have anymore questions!


(Rich Navis) #3

Hi Crystal, I’ve read through the documentation before posting, but couldn’t figure it out, which is why I posted. If I add the users as responders, they will get woken up in the middle of the night if the problem occurs then, which is not what we want. If we add them as Stakeholders, they don’t get automatically notified when an incident is created… Do you have any specific ideas on how to achieve what I would like to do?


(Simon Fiddaman) #4

Hi @RichNavis,

You’re asking for something which I personally feel is outside the best practice / philosophy of receiving alerts - that of alerting only one person (at least initially) about an alert and empowering them to make the call on who else to involve if they need to.

That being said, I’ve been asked to setup similar things for two of our Team Leads who (in this case) want to “feel the pain” of their On-call’s alerts.

First question: how do you want to notify these individuals? By email? By SMS? By push notification? Probably not by calling as you don’t want to wake them up.

Method 1: Add the additional notifier people as assignees in PagerDuty on the first Level of the Escalation Policy. This works well for one person, and would kinda suck for more. Make sure they use only non-intrusive contact methods for their High Urgency notification methods. This is a pretty fragile setup, but it’s all we needed to satisfy the curiosity of those TLs.

Method 2: Create a dummy user who has notification methods for e.g. team email DL and add it to, or make it the only Level 1 contact - make sure you have other Levels which have real Scheduled, obviously.

Method 3: Use Event Rules to automatically trigger a Response Play - either as a Stakeholder notification to a group of Stakeholder users, adding a responder or group of responders (same considerations apply as for Method 1), or as per Method 2 - a dummy user.

Method 4: Use an integration with e.g. StatusPage.io to automatically trigger a notification (per Service) when there’s any active Incidents. This will not contain the specifics of the alert(s).

Method 5: Just push all notifications to Slack (or other comms tool) and use the notification rules in Slack to notify Channel members.

Method 6: Build something which accepts webhooks (e.g. a Lambda function or app) and redistribute the notifications to your recipients.

My recommendation is Method 5 - pushing notifications to Slack.

Hope that helps and good luck,
Simon


(Rich Navis) #5

Thanks for the detailed response Simmon, I appreciate your help, and will likely implement method 2.
It’s unfortunate that we have to find work-arounds for this. Although I do agree with “assigning” the incident to a single person, the idea of “notifying” others about a new incident being a manual process doesn’t make sense to me. Consider that stakeholders have won’t know about an incident until a responder decides to create a status update. In theory, and quite often in practice, a stakeholder may NEVER be notified about an incident .

It seems that pagerduty is so close to getting this right… all they have to do is enable stakeholders to get notified on incident creation through a response play… boom, done. For some reason, they have chosen to only notify stakeholders when an incident responder decides they should be notified.


(Crystal J. Mendoza) #6

Hi Rich,

Just to clarify, you actually can add Stakeholders and send Status Updates automatically upon an incident’s creation.

Here is an article from our Knowledge Base on automatically running a response play at incident creation. Once this is configured in your service settings, Stakeholders will automatically receive status updates without having to be manually added by an incident responder!

Please let us know if you have any further questions or feedback.


(Rich Navis) #7

Thanks Crystal… That is the answer I was looking for. I did not infer from the documentation (my mistake) that since stakeholders can be subscribed to get status messages through response plays, having a status message go out at incident creation is the way to achieve this.


(Crystal J. Mendoza) #8

Rich,

No problem at all, I’m glad I could help clear this up for you!

Please let us know if anything else comes up.


(system) #9

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.