How-tos & Best Practices

Page it to the Limit: Alert Fatigue with Boaz Barzel https://www.pageittothelimit.com/alert-fatigue-with-boaz-barzel/  Understanding Alert Fatigue and How to Prevent It https://www.pagerduty.com/resources/learn/alert-fatigue/ What’s the Difference Between SLAs, SLOs, and SLIs? https://www.pagerduty.com/resources/learn/what-is-slo-sla-sli/ What is AIOps? https://www.pagerduty.com/resources/learn/what-is-aiops/  Automated Remediation https://autoremediation.pagerduty.com/ Google SRE Workbook https://sre.google/workbook/implementing-slos/ Nobl9 https://www.nobl9.com/ 

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️ Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 5: Prevent.What does “Prevent” mean?Preventing means reviewing metrics, uncovering trends and patterns, and making improvements to your infrastructure, monitoring thresholds, or team performance. When

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 4: Resolve.What does “Resolve” mean?Resolving means coordinating a response, collaborating with others, and surfacing as much information as possible to try to resolve an incident quickly. When you maxi

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 3: AssessWhat does "Assess" mean?Assessing means understanding the full scale and impact of an alert or wide-spread incident. It answers the question “How bad is it?” and determines how the alert or inc

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again. In this article, we will discuss Step 2: Notify.What does “Notify” mean?Notifying means alerting the right people at the right time about an alert. When you leverage tools to notify your team properly, the right people can come together qu

There are 5 steps in the Incident Response Cycle. Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️ Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 1: Optimize. What does "Optimize" mean?Optimizing means configuring your monitoring and alerting such that only actionable events receive immediate attention and non-actionable events are categorized d

Analytics are a great way to provide additional insights regarding metrics and numbers into your incident management cycle. Through our Analytics in our lite plans, users can view reports in regards to their notifications and incidents and through Analytics in our Standard and Enterprise plans, users can view data on Systems, Teams, and Users.The value that analytics can bring to teams includes being able to track: How long someone has been on-call forAre they on-call too frequently? How many incidents have they been notified for? Are they at any risks of experiencing alert fatigue? Managers can use this information to track and see how their users are feeling while they are on-call and make the appropriate adjustments.See the noisiest services to modify their alerting through email filters/event rules or high/low urgenciesThis will help ensure that all incidents that on-call users receive will be actionable and prevent them from alert fatigue.Compare the MTTR/MTTA with other teams, se

Live Call Routing (LCR) is a great way to give your customers or internal team the ability to connect directly with an on call engineer. While this might work great during business hours, it isn’t always optimal at 3am.You could restrict your schedule so that your team is only on call during business hours, but then if someone calls in after hours they’ll receive an error message—because nobody is currently on call—and won’t be prompted to leave a voicemail. PagerDuty won’t trigger an incident either, as there isn’t anyone to assign the incident to at this time. If only there was a way to prompt callers to connect directly during business hours and leave a voicemail after hours…We’ve come up with a workaround where you can now configure an LCR phone number to connect directly during business hours and prompt callers to leave a voicemail after hours.Configure your on call schedule:First, you’ll want to set up a “dummy” user in your account and set them up with a fake phone number (ex. 4