Starting with PagerDuty

Page it to the Limit: Alert Fatigue with Boaz Barzel https://www.pageittothelimit.com/alert-fatigue-with-boaz-barzel/  Understanding Alert Fatigue and How to Prevent It https://www.pagerduty.com/resources/learn/alert-fatigue/ What’s the Difference Between SLAs, SLOs, and SLIs? https://www.pagerduty.com/resources/learn/what-is-slo-sla-sli/ What is AIOps? https://www.pagerduty.com/resources/learn/what-is-aiops/  Automated Remediation https://autoremediation.pagerduty.com/ Google SRE Workbook https://sre.google/workbook/implementing-slos/ Nobl9 https://www.nobl9.com/ 

The Onboarding Checklist will guide you through the PagerDuty implementation experience from planning to launch. Download a copy of the onboarding checklistWe’ve identified key milestones in each phase to make sure you’re set up for success. Be sure to reference the links and set clear dates to keep you on track along the way! The Planning PhaseGoals and Success Metrics Establish User Roles Outline Desired Workflow: Sample WorkflowsThe Configuration PhaseAdd/Import Users Onboard Admins and Managers PagerDuty University Training: https://university.pagerduty.com/admin-account-owner-permissions Admin & Account Owner Permissions Create an On-Call Schedule PagerDuty University Training: Schedule & Overrides Create an Escalation Policy PagerDuty University Training: Escalation Policies Configure Services and Integrations - we integrate with over 700 tools - view our Integration Guides PagerDuty University Training: Creating a Service with Integrations Trigger a Test Incident to Test

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️ Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 5: Prevent.What does “Prevent” mean?Preventing means reviewing metrics, uncovering trends and patterns, and making improvements to your infrastructure, monitoring thresholds, or team performance. When

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 4: Resolve.What does “Resolve” mean?Resolving means coordinating a response, collaborating with others, and surfacing as much information as possible to try to resolve an incident quickly. When you maxi

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 3: AssessWhat does "Assess" mean?Assessing means understanding the full scale and impact of an alert or wide-spread incident. It answers the question “How bad is it?” and determines how the alert or inc

There are 5 steps in the Incident Response Cycle: Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again. In this article, we will discuss Step 2: Notify.What does “Notify” mean?Notifying means alerting the right people at the right time about an alert. When you leverage tools to notify your team properly, the right people can come together qu

There are 5 steps in the Incident Response Cycle. Response Cycle - Step 1: Optimizing your Alerting ↗️Response Cycle - Step 2: Notifying the right people at the right time ↗️Response Cycle - Step 3: Assessing the impact of an incident ↗️Response Cycle - Step 4: Resolving Incidents Through Collaboration ↗️Response Cycle - Step 5: Preventing problems from recurring ↗️ Addressing each step in the incident response cycle will ultimately help your team drive down incident resolution times by (1) focusing on actionable alerts, (2) getting the right people to review those alerts, and (3) understanding the full impact of the problem so that (4) the appropriate remediation steps can be taken and (5) reviewed to make sure the same problem does not happen again.In this article, we will discuss Step 1: Optimize. What does "Optimize" mean?Optimizing means configuring your monitoring and alerting such that only actionable events receive immediate attention and non-actionable events are categorized d

Analytics are a great way to provide additional insights regarding metrics and numbers into your incident management cycle. Through our Analytics in our lite plans, users can view reports in regards to their notifications and incidents and through Analytics in our Standard and Enterprise plans, users can view data on Systems, Teams, and Users.The value that analytics can bring to teams includes being able to track: How long someone has been on-call forAre they on-call too frequently? How many incidents have they been notified for? Are they at any risks of experiencing alert fatigue? Managers can use this information to track and see how their users are feeling while they are on-call and make the appropriate adjustments.See the noisiest services to modify their alerting through email filters/event rules or high/low urgenciesThis will help ensure that all incidents that on-call users receive will be actionable and prevent them from alert fatigue.Compare the MTTR/MTTA with other teams, se

Live Call Routing (LCR) is a great way to give your customers or internal team the ability to connect directly with an on call engineer. While this might work great during business hours, it isn’t always optimal at 3am.You could restrict your schedule so that your team is only on call during business hours, but then if someone calls in after hours they’ll receive an error message—because nobody is currently on call—and won’t be prompted to leave a voicemail. PagerDuty won’t trigger an incident either, as there isn’t anyone to assign the incident to at this time. If only there was a way to prompt callers to connect directly during business hours and leave a voicemail after hours…We’ve come up with a workaround where you can now configure an LCR phone number to connect directly during business hours and prompt callers to leave a voicemail after hours.Configure your on call schedule:First, you’ll want to set up a “dummy” user in your account and set them up with a fake phone number (ex. 4

Some customers want to have their new employees shadow the on-call team as part of their onboarding. There are two simple ways to accomplish this:Creating an On-Call Shadowing Schedule:Create a blank scheduleAdd the shadowing schedule to the escalation policy that these new users will be shadowing As you onboard new users to your team, temporarily add them to the schedule using overrides If you’re onboarding a lot of people simultaneously, it might be helpful to create a shadowing schedule for each team. You can automate the process of adding shifts by utilizing our API.Having Users Temporarily Shadow an On Call Team 24/7:Add the new user to your PagerDuty account Locate the escalation policy that this user will be shadowing Add the user directly to the appropriate level of the escalation policy Remove the user once their shadowing time period ends

PagerDuty offers tailored instructor-led training sessions that cover both foundational and advanced topics, ensuring flexibility in scheduling and location to meet your team’s specific needs! Browse all instructor-led trainings and enroll: PagerDuty University Instructor-Led Training

Whether you’re an IT professional, a developer, or a business leader, PagerDuty certifications validate your expertise in optimizing your organization’s response to incidents, ensuring maximum uptime, and minimizing downtime.PagerDuty University offers a unique blend of product and thought-leadership certifications, focused on skills that are highly sought after in today’s digital world. Join the community of certified PagerDuty professionals who are shaping the future of real-time operations! Learn more and enroll: PagerDuty University Certification

PagerDuty University offers a self-paced and fully on-demand learning resource that brings together dozens of free courses that walk learners through every step of the process of using PagerDuty. The entirely free courses detail the setting up, the scheduling, the mobile app, and more.From PagerDuty 101 to Thought Leadership, learn at your own pace with PagerDuty University’s on-demand courses! Find more and enroll: PagerDuty University On-Demand 

The PagerDuty University team leads lives PD 101 trainings to give new managers and responders an opportunity to learn PagerDuty best practices. These 30 minute sessions will be offered Monday-Thursday once monthly. Choose your preferred timezone and register for the next training here Who should attend?Incident responders Admins/managers Account owners New Users Customers Folks in a trialWhat’s covered?Invite users to your account Create schedules and overrides Set up escalation policies Configure services and integrations Utilize extensions and API access keys Add teams Respond to incidents Manage user profilesCan’t make any of the dates? Submit any questions to pdusupport@pagerduty.com!

Incidents can come to PagerDuty in many different ways. Often times, events are automatically sent to PagerDuty via the Events API when certain thresholds are met in your monitoring tools. However, there may be some instances when a user needs to manually trigger an incident, or perhaps you need a way for other teams (outside PagerDuty) to page the on-call responder for a critical issue.Check out the below suggestions to make it easier for teams to trigger incidents and notify on-call responders.Send an email from your inbox to trigger an incidentThe easiest way to trigger an incident (without logging in or being a user in PagerDuty) is by sending an email.If you have an email integration 1 set up on your account, you can have people within your organization (or even your customers) send an email to one of your integration’s email addresses. This will then trigger an incident in PagerDuty and notify who is on-call.Here are some ideas for managing these email addresses: Post the relevan

With the ability to add more than one integration per service, PagerDuty enables you to model your services as they actually exist within your infrastructure. Benefits include:The ability to drive more clarity and accountability by aligning services with component ownership so that alerts are not just from siloed tools Creating a single, consolidated service dashboard in PagerDuty for each of your internal systems Better control of maintenance windows for a particular service Responders will be able to more easily diagnose which system is having problems, leading to lower resolution times Better leverage PagerDuty’s analytics capabilities for a more complete view of which systems are having problems. Reporting by service on the System Report will show accurate incident counts, MTTA, MTTR, and escalation counts for all monitoring tools tied to each system, not just one. This also allows you to see which components are healthy, and which are having consistent problems that may need to be

Often times, it’s not just the people on-call who need to know about an incident. Managers and other parts of the business (Support, Sales, etc) may also need to be aware of incidents that arise. Whether these teams are in PagerDuty or not, there are many ways to keep others informed about incidents and events that are triggered in PagerDuty.Here are some of our recommendations and best practice tips for creating transparency in your organization:Creating transparency for users that exist in PagerDuty Add managers to escalation policiesTo increase visibility into alerts on certain services and escalation policies, a manager (or any other user/schedule) can be added to the first-level of an escalation policy along with the first user/schedule on-call so that the manager is notified about incidents that are triggered.Additionally, if the manager added status update notification rules to their profile, they can gain even more visibility into incidents with alerts, but setting up notificat

Escalation policies determine who is notified when an incident is triggered and what should happen if nobody responds to that incident.We recommend keeping the following items in mind to make your escalation policies more meaningful:Check for schedule gapsIf the schedules that you are adding to your escalation policies have schedule gaps, check to make sure that there is somebody on-call at the times that you need an incident to trigger.The reason for this is because incidents will not be created if nobody is on-call in an escalation policy; incidents can only be created if somebody is on-call on any level of an escalation policy.Example:If you have an escalation policy with 3 levels, each containing a schedule where nobody is on-call between 6pm-9am, then if an event comes in at 10pm, no incident will trigger and nobody will be notified.If you have an escalation policy with 3 levels, with the first 2 levels containing schedules where nobody is on-call between 6pm-9am and the last leve

Is the concept of an on-call rotation new to your team? Or does your existing on-call rotation need some tweaking since more people are being added to a rotation?If so, we recommend reading some of our best practice tips for setting up your on-call schedules:Exchange Shifts During Business HoursThe most popular time to handoff your on-call shift is Midnight, followed by 8:00 AM and 9:00 AM, then 5:00 PM and 6:00 PM. Despite the popularity of the midnight swap, as PagerDuty, we recommend having your handoff occur during business hours, preferably when both parties are present in the office. Unless you both happen to be in the office at midnight, in which case, go home.Switching your shift while on-site gives you the opportunity to talk to the next person going on-call about any issues that occurred during the previous shirt or to give a heads up on anything they may want to be on the look-out for.At GREE, their team syncs up every Monday morning to review alerts from the previous week,

An extremely common question that I receive when introducing a new team to PagerDuty (or any notification tool, really), is “how should I set up my notifications?”Here’s an answer that has worked well for me.A Sample Notification Escalation policyNotify on AssignmentThis is a sample notification policy to alert me when an incident is assigned to me. It’s intention is to be “helpfully persistent”. This is catered to my order-of-notification preference, as well as my “dexterity issues” during the wee hours. Feel free to tweak as desired.| Timeframe | Method || ----------- | ------ || Immediately | Email || Immediately | Push || 1 Minute | Push || 2 Minutes | SMS || 3 Minutes | Call |One common mistake is to have all notification types trigger at the same time. Unfortunately, what ends up happening is that in the time that it takes you to start responding to the one notification, another will trigger on top of the first. You can’t acknowledge the Push Notification before the SMS comes in,

If you want to group your services, escalation policies, or on-call schedules, you can do so using Teams, or by using a standardized naming scheme (i.e. a standardized prefix). The example below shows a set of services with the prefix Beam -.You can quickly sift through your services, escalation policies, and on-call schedules by searching for the desired prefix (shown below). While the example shows the Services list, the Escalation Policies, Schedules, and other lists also have search functions.

So you want a follow-the-sun response to your alerting?Create s Schedule, selecting UTC as the time zone because it doesn’t change. Start with any one of your regions Select the user(s) for this region Hit the Restrict on-call shifts to specific times checkmark If this is a 24x7 schedule, just the first times-of-day option and add your window in UTC (e.g. 06:00 - 20:00) If this schedule has gaps (like weekends), you’ll need the second option, creating additional blocks until you’ve reached your desired coverage.NOTE: If you end up with holes in your Schedule, alerts received during these holed times will be Suppressed (blackholed).Select your Rotation type and choose a Handoff time which matches the start of your natural blocks (e.g. Monday at 06:00 as per the example above) Repeat with any other region(s), adding a new Layer for each region Add the Schedule to an Escalation Policy (or multiple Policies if you have further unique escalation points, e.g. for SME On-calls).NOTE: During D

Advanced Permissions allow you to specify different levels of permissions for users across your PagerDuty account and within your team. These levels of permissions include object-level permissions, team-level permissions, and account-level permissions. You can read more about advanced permissions here.If you’re interested in adopting advanced permissions within your organization, consider the following best practices tips:1. Follow the principle of least privilege: Grant users as much access as they need to be effective in their role. In practice, this means you’ll want to limit the number of users with a Global Admin or Manager base role, especially at larger organizations. Everybody else should have either an observer, responder , or restricted access base role with additional permissions given at either the team or object level.2. Add all users and escalation policies to a team to set team roles: Team roles give users specific access to a team, especially when they have limited acce