Incidents: Creating transparency across teams

success
official
incidents

(Gabriella Freda) #1

Often times, it’s not just the people on-call who need to know about an incident. Managers and other parts of the business (Support, Sales, etc) may also need to be aware of incidents that arise. Whether these teams are in PagerDuty or not, there are many ways to keep others informed about incidents and events that are triggered in PagerDuty.

Here are some of our recommendations and best practice tips for creating transparency in your organization:

Creating transparency for users that exist in PagerDuty

Add managers to escalation policies

To increase visibility into alerts on certain services and escalation policies, a manager (or any other user/schedule) can be added to the first-level of an escalation policy along with the first user/schedule on-call so that the manager is notified about incidents that are triggered.

Additionally, if the manager added status update notification rules to their profile, they can gain even more visibility into incidents with alerts, but setting up notifications when incidents have been acknowledged, resolved, or escalated.

If this manager is not the primary responder for incidents though, it is recommended that they do not acknowledge, resolve, or escalate incidents before the on-call user has a chance to see it.

Subscribe stakeholders to incidents

Adding subscribers lets you notify stakeholders who aren’t directly involved with resolving the incident. This could be C-level executives concerned about the health of the company, or a Support team interacting with customers during an outage, for instance.

Creating transparency for users outside of PagerDuty

Use webhooks to email multiple teams

Webhooks are tied to services and they can send out HTTP callbacks when different events happen to an incident triggered on a particular service. Many teams set up a webhook on their service so that an email is sent any time an incident is triggered. This is ideal if you want to email a distribution list or list of users who are not users in PagerDuty and do not have access or receive alerts from PagerDuty.

Zapier or a similar service can be used to consume PagerDuty webhooks and send an email based on these webhooks.

Push incidents into your collaboration/chat tool

Integrate PagerDuty with Slack, HipChat, Flowdock, Socialcast or any of our other Collaboration/Chat integration partners. This is a great way to let teams know when incidents are triggered, acknowledged, and resolved in real time, without having to manually alert everybody.

Publish incidents on an internal status page

Use a status page app such as StatusPage.io or Status.io. These tools will group PagerDuty services into components so you can quickly determine the status.

Create internal dashboards with our API

You can use our REST API to create internal dashboards to display open incidents.

Add distribution lists to your escalation policies

An alternative to using webhooks to email distribution lists (as described above), you can create a user with email distribution lists as their email contact method(s) and notification rule(s), and add that user to the appropriate level of your escalation policies.

When an incident is triggered against a service that is using that escalation policy, notifications would go out to all of the email addresses listed as a notification rule under that user profile.

Note: You may want to create a separate distribution list user per team you wish to notify, as not everybody necessarily wants notifications about incidents outside of their team.


Adding a DL in Policies
How do I notify a distribution list or shared group email address?
Integrations: Setting up services and integrations