My organization just recently got a subscription for PagerDuty at the Pro plan level. This has been an uphill battle for me for almost 7 years now to get this as my current leadership believes all alerts and notifications should be via email and should always hit everyone all at once.
I came from an organization that used XMatters, so my experience with targeted alerting and triage is there, but it’s not far different in philosophy than PagerDuty. My argument is that we should only be alerting on critical items, and it should be using a workflow and start at the lowest level for triage then escalate as needed after triage. Instead of everyone getting an email all the time, proper incident management should be done where a ticket is also generated in our ITSM tool, then appropriate communications from the ticket go out at regular intervals to stakeholders. One of the arguments I often get is “We’re a small company”, which I disagree with, we solidly fall into the mid-size enterprise, but even then, size shouldn’t matter in doing things properly and responsibly.
Has others faced this kind of push back, email only and everyone always, and if so, how did you finally convince folks to embrace PagerDuty?Â
My biggest push-back against email is with that, I can’t do any Inbox organization as Outlook mobile will only notify ion things in the Inbox, and I would have to create a notification that would wake me up at 2am so I can see if that email is just noise or real, and usually I get a few dozen emails overnight so that just becomes noise and a marriage killer. I can’t begin to count how many incidents were missed overnight and “dealt with in the morning” instead of properly triaged and addressed.
Communicating Value of PagerDuty
Hi ​
All the experiences you’ve had with other solutions are true for PagerDuty! We see teams greatly improving the reliability of their products and services using PagerDuty. We definitely see teams that rely on blasting alerts out to everyone regardless of how ineffective that is.
- Alert the right people at the right time about the right problems. If your services are 24x7, sending email to everyone does nothing to guarantee that the right person to solve a problem will see it and be available.Â
- Not every issue is the same level of importance. Just like you mentioned, taking time and resources away from other work causes a lot of disruption, and you want to make sure it’s being done for the most important issues in your environment through tools like alert categorization and severity levels.
- Give team members a plan. Knowing when they are expected to be available for incidents helps teams manage workloads and expectations. No one wants to be dragged away from their off hours anyway, but being expected to always be available is terrible for employee retention and work-life balance.
- Use automation to augment human responders, and replace them for well-understood incidents. If a human is just going to log in and run a script, let the computers do it and don’t pull team members out of flow unless there is a problem with the script.
- Alleviate confusion with predictable workflows and tasks. The timeline of incidents can definitely get extended with folks being confused about who is responsible, where to coordinate efforts, and what should be happening.Â
One thing that sometimes folks don’t fully keep in mind is that incidents are work. They’re unplanned work, but it’s still work. Most teams don’t go into their planned work without some kind of process or way to organize and prioritize that work, so it should make sense to also approach unplanned work in the same way.Â
Our Incident Response guide might be helpful for you, too. https://response.pagerduty.com.Â
Good luck! If you’d like your team to talk with one of our Developer Advocates, please reach out! We’re community-team@pagerduty.com.
--mandi
Login to PagerDuty Commons
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.