Skip to main content

Recently active topics

18
xamici
xamiciCommunity Manager
 Announcements
August 28 Kafka Outages – What Happened and How We’re Improving
August 28 Kafka Outages – What Happened and How We’re Improving

Hey PagerDuty community! 👋We experienced service disruptions on August 28 that affected some customers in our US Service Regions, and we owe you complete transparency about what happened.The overview: A Kafka system failure at 3:53 UTC triggered cascading issues that disrupted event processing for several hours. While no data was lost, some customers experienced 502 errors, delayed notifications, and degraded platform functions.Here's a sneak peek of what you'll find in our full incident report:Detailed technical breakdown of the Kafka failure and cascading effects Complete timeline from initial incident through full resolution How a second, smaller incident actually helped us identify the root cause Why our external communication was delayed (and how we're fixing that) Specific improvements we're implementing to prevent recurrence Lessons learned from our response and recovery processWe know reliable alerting is mission-critical for your operations, and we're committed to earning bac

110
xamici
xamici
xamiciCommunity Manager
 Announcements
🎧 New Episode: Monitoring to Observability With Satbir Singh
🎧 New Episode: Monitoring to Observability With Satbir Singh

The transition from traditional monitoring technologies to modern observability tools can leave teams confused. Satbir Singh joins us to talk about the new goals of observability tooling and how it will help teams conquer challenges in complex systems.LISTEN HERE ⬇️ 🎧 Listen on Page it to the Limit or in your favorite podcast app: Spotify | Apple | Pocket Casts 🎧 Page It to the Limit is a podcast that focuses on what it means to operate software in production. Hosted by the PagerDuty Developer Relations Team, we cover the leading practices used in the software industry to improve both system reliability and the lives of the people responsible for supporting it.

20
xamici
A
alexcsmNew Member
 Ask a product question
Incident Workflow - Custom Webhook POST Signing

I was looking into adding custom workflow or custom incident actions to our service. - https://support.pagerduty.com/main/docs/incident-workflows- https://support.pagerduty.com/main/docs/custom-incident-actions​​What’s the best way to authenticate the URL request? Looks like Pagerduty V3 webhooks are automatically sent with a signature (e.g. X-PagerDuty-Signature). Then in our lambda, I’m able to use this handy webhookv3 client library to verify the signature. ​- https://developer.pagerduty.com/docs/webhooks-overview- https://github.com/PagerDuty/go-pagerduty/blob/main/webhookv3/webhookv3.goHowever, it doesn’t appear this header is attached to either the custom workflow nor incident action POST

111
lupimiguel
M
MylesNew Member
 Ask a product question
PagerDuty on iOS cannot use other built-in ringtones from the app.

 Hi,My iOS (18.6.2) iPhone 14 Pro is unable to use other built-in ringtones from the PagerDuty app (e.g., "PagerDuty Alert") when receiving High-Urgency push notifications. It always plays the default "ding ding ding"-type sound, even though I have already enabled Critical Alerts for High-Urgency notifications.

598
lupimiguel
xamici
xamiciCommunity Manager
 Announcements
📢 What's New: Platform Release Notes, August 2025 📕

🚀 Check what’s new in PagerDuty and come back to let us know!

50
xamici
I
illia.kaliuzhnyiNew Member
 Ask a product question
PagerDuty Scheduling: Fair Rotation of Weekdays Without Manual Lists

Dear PagerDuty Community,We’ve been struggling to configure our PagerDuty schedule for a team of exactly 10 developers, with two on-call shifts per workday (10 shift slots per week). This challenge was raised in the forum several years ago for 7 slots and 7 engineers, but it seems it hasn’t received enough attention from the PagerDuty team.Here’s a simplified example to illustrate the problem:Suppose, we have a team of 5 engineers - A, B, C, D, and E - and one on-call slot per workday. If we simply configure a schedule with these five engineers filling the five weekday slots, each engineer ends up with a fixed day every week (e.g., A always on Mondays, B always on Tuesdays, etc.). This is not desirable — we want the days to rotate so that A is on Monday in week one, Tuesday in week two, Wednesday in week three, and so on.The only workable solution we’ve found so far is to manually create a long, repetitive list of people, such as:ABCDE_BCDEA_CDEAB_DEABC_EABCDThis does achieve fair dist

683
I
A
alopezmeNew Member
 Ask a product question
Timestamp on workflow payloads

Hello,I am trying to create an incident workflow which makes an  HTTP POST requests to en external server. I need to send the incident creation time in epoch timestamp format. It looks like {{incident.created_at}} is ISO 8601 . How can I get this timestamp? Alex

171
lupimiguel
C
cloaked1New Member
 Ask a product question
How to re-alert on acknowedged alert after incident remains unresolved

I’ve read that there’s a way to re-alert on an ack’ed alert if the alert remains unresolved after so long. How does one set this up? I cannot find anything relative to that in escalation policies.TIA,Jim

242
C
T
tjhNew Member
 Ask a product question
Feedback: Why is Pagerduty so hard to use?

I’m a peon.  I am a sad end-user who has to go oncall every now and then.My god this thing is terrible to use!!I want to quickly put in a override because I’ll be doing some work and I’ll create alarms.  So I need to take over that time.Here’s what I’m presented with when I login.  I can go into each of those menus above, none of them are clear or related to the fact I’m oncall and want to do a thing relating to it.Finally after fiddling around I get to schedules. No, that’s useless. Can’t do an override in there (or can I...)Oh! Go to schedules and there’s a big blue “Create Override” button.  Great! I click that, fill in all the details and get a message: Get REALLY annoyed by this stage.  Fluff around, FINALLY realise I have to back to Schedules above, find the person who’ll be doing the shift I need to override, click on their name and then add it in there.I mean I got there in the end but why did I have to fight with this tool for 10 minutes to figure this out? Shouldn’t this be t

701
lupimiguel
A
aaronlrobertswwNew Member
 Ask a product question
Service not showing for Incident Workflow

Hi folks, I recently created an Incident Workflow which triggers automatically for a service and everything is fine and working no problem.However, now, I want to add a new kind of workflow that can be triggered manually for that same service, and for some reason, searching that service always returns “No Matches Found”..Even if I enter into the workflow I that was already created a month or so ago, and “Edit” that service, I can see it populated, but let’s say I remove it and type it again, it will return “No Matches Found”.If I clone a workflow, the service is populated with no issue, but obviously I need to create a manual trigger and I still encounter the issue.It’s incredibly frustrating as this is holding up a major change to the way we handle certain incidents, so any advice on this matter would be greatly appreciated.I've already raised a support request via Chat, but no response, hence I’m here.Thanks!

201
lupimiguel
O
Omri EfratNew Member
 Ask a product question
FedRAMP API Availability

Hi TeamI’d like to confirm whether there is a dedicated FedRAMP- API version available, or if it is the same API as the standard version. If there is a separate endpoint or documentation for the FedRAMP API, could you please share the link?Thank you for your help,

81
lupimiguel
A
amit82New Member
 Ask a product question
Non linear escalation

Hey,I want the escalation policy to be non linear.currently, level 1 can escalate to level 2 and level 2 escalate to level 3.I want level 1 to be able to select if to escalate to level 2 or 3. is it possible and how can I accomplish it?thanks.

151
lupimiguel
xamici
xamiciCommunity Manager
 Announcements
Resources + On-Demand: Demo Roundups! Beyond the Incident: Mastering Post-Incident Reviews for Continuous Learning
Resources + On-Demand: Demo Roundups! Beyond the Incident: Mastering Post-Incident Reviews for Continuous Learning

What happens after an incident matters just as much as how you handle it.Anojan Gunasekaran, Senior Product Manager for Incident Analysis at PagerDuty, showcased how to transform post-incident reviews from a bureaucratic necessity into a powerful tool for organizational improvement.Learn how to structure reviews that help facilitate meaningful discussions, identify systemic issues, and create actionable recommendations that prevent future incidents.👀 Watch a sneak peak:  📚 Resources shared during the session: Read on the Blog: Beyond Outages: The Post-Incident Reviews We Should Have Had Knowledge Base: Post-Incident Reviews Take the Post-Incident Review Product Tour PagerDuty Guidelines for the Safe and Secure Use of Generative AI 📺 Watch the full webinar on-demand 

130
xamici
E
erenNew Member
 Ask a product question
can't use API User Tokens

when you go to porfile → user setting, you have the option to create API User Tokens, but I can’t use them at allI downloaded the Postman collection from this page developer.pagerduty.com/docs/rest-api-overview and entered the owner’s Token, but the response for all requests is 401 unauthorizedam I using the wrong documentation or what 

252
E
danieljcafonso
danieljcafonsoCommunity Manager
 Announcements
📝 New Blog Post: PagerDuty + Arize: Building End-to-End Observability for AI Agents in Production

PagerDuty + Arize: Building End-to-End Observability for AI Agents in ProductionPagerDuty teamed up with Arize to bring end-to-end observability to AI agents in production. From automated testing to real-time alerting, we’re building new ways to manage incidents in the age of AI.Read the full story to see how we’re doing it and what you can learn from our journey ⬇️ https://www.pagerduty.com/eng/pagerduty-arize-building-end-to-end-observability-for-ai-agents-in-production

210
danieljcafonso
mwalls
mwallsCommunity Manager
 How-tos & Best Practices
Resources for the talk "Tackling Alert Fatigue with SLOs, Automation, and Machine Learning"

Page it to the Limit: Alert Fatigue with Boaz Barzel https://www.pageittothelimit.com/alert-fatigue-with-boaz-barzel/  Understanding Alert Fatigue and How to Prevent It https://www.pagerduty.com/resources/learn/alert-fatigue/ What’s the Difference Between SLAs, SLOs, and SLIs? https://www.pagerduty.com/resources/learn/what-is-slo-sla-sli/ What is AIOps? https://www.pagerduty.com/resources/learn/what-is-aiops/  Automated Remediation https://autoremediation.pagerduty.com/ Google SRE Workbook https://sre.google/workbook/implementing-slos/ Nobl9 https://www.nobl9.com/ 

450
mwalls
A
ABillingsleyNew Member
 Ask a product question
PagerDuty+ServiceNow+Solarwinds - No incident sent to PagerDuty

Issue Summary:We're experiencing a problem where ServiceNow incidents triggered by Solarwinds alerts are not creating corresponding incidents in PagerDuty, even though they meet the priority criteria (P1–P3).Context: Solarwinds is our alerting tool, integrated with ServiceNow (Yokohama, PD Spoke 1.4.0, PD 8.2.0). ServiceNow is configured to forward P1–P3 incidents to PagerDuty. We're using assignment group services only in PagerDuty (no business/technical services). This issue began after integrating Solarwinds through ServiceNow Event Management. What we've observed: Some PagerDuty webhooks become disabled intermittently, though health checks often return normal. The issue is inconsistent—not all Solarwinds-triggered SNOW incidents fail to generate PD incidents. We suspect a possible problem with the PD Spoke workflow or webhook handling, but haven't confirmed. If anyone has run into a similar issue or has suggestions on what to check next, I’d greatly appreciate your insigh

281
danieljcafonso
S
soumyajyotiiiNew Member
 Ask a product question
Need Help Displaying Custom Details from Grafana Alerts in Slack

Hi PagerDuty team! 👋I'm trying to get custom details from Grafana alerts to display in our Slack notifications, but only seeing basic information.Current Setup:✅ Grafana → PagerDuty integration working (using Events API V2) ✅ PagerDuty → Slack integration connected (service: my-service-prod → channel: #team-alerts) ✅ Alerts are flowing through successfullyThe Issue: Our Grafana alerts send rich custom details to PagerDuty (verified in PagerDuty incident view):severity: "critical" namespace: "production" pod: "api-service-7d9c5-xyz" cluster: "us-west-2-prod" component: "api-service" description: "Pod is in CrashLoopBackOff state..." runbook_url: "https://wiki.internal/runbooks/..."However, in Slack we only see:🔴 Pod CrashLoopBackOff AlertService: my-service-prodUrgency: Low[Acknowledge] [Resolve]What We Need: Display the custom_details fields in Slack notifications so our on-call engineers can see critical context without clicking through to PagerDuty.What I've Tried:Configured PagerD

341
danieljcafonso

Badge winners

  • First Responder
    illia.kaliuzhnyihas earned the badge First Responder
  • First Responder
    Myleshas earned the badge First Responder
  • First Responder
    cloaked1has earned the badge First Responder
  • First Responder
    erenhas earned the badge First Responder
  • First Responder
    jlarrimorehas earned the badge First Responder
Show all badges
Terms & ConditionsCookie settings