I’m not sure if anyone encountered similar problem.
We just suddenly cannot receive any webhook of pagerduty from Sep26, the endpoint is definitely reachable and had been working for years.
I can see the webhook integration of most services had been disabled, if I manually enable them , they just got back to be disabled almost immediately.
it did happen around my yearly renewal ,so I wonder if it’s something to do with that.
Best answer by lupimiguel
Hi @Bing,
Here’s a rundown of what can be happening considering other cases we’ve had:
PagerDuty auto-disables a webhook after repeated delivery failures (non-2xx responses, timeouts, redirects, etc.). We’ve seen 3xx redirects (e.g., 302) trigger temporary disablement even when an endpoint “seems” fine from a browser or Postman. Example community case: Google Apps Script returning redirects caused disablements until the endpoint was fixed, and Support confirmed the 302s were the culprit. Reference: Webhook to Google Apps Script | Community
Similar reports show endpoints “working” in other contexts but PD disables the webhook because the target isn’t accepting the PD payload/auth as sent, or returns non-2xx under load. Example: ServiceNow case where the webhook kept disabling due to an auth issue on the endpoint: ServiceNow Community thread
High-signal next checks (even if things look “unchanged”)
Eliminate redirects entirely:
Ensure the exact webhook URL returns 2xx directly (no 3xx, no HTTP→HTTPS hop, no trailing-slash redirect, no CDN edge redirect, no auth redirect). PD won’t follow 3xx reliably and repeated 3xx can disable the sub. See similar case: Community thread
Return success fast:
Respond 200/204 within a few seconds, then do any heavy work asynchronously. Slow responses or timeouts count as failures. You can Send Test Event from the UI to verify: Webhooks KB – Test Event
Validate payload acceptance:
Make sure your endpoint accepts PD’s JSON as-is. Common silent failures:
400 due to schema validation differences
415 due to strict Content-Type checks
413 if request body limits are too small
401/403 if your server now requires auth but the PD webhook has none
Check TLS and URL invariants:
No expired/rotating cert chains, no SNI/cipher suite issues, and the hostname in the webhook URL hasn’t started redirecting to a different host. DNS or CDN config shifts can introduce 3xx at the edge without a firewall change.
WAF/CDN rules:
WAFs sometimes block PD’s IP ranges, rate-limit, or challenge requests differently than Postman. Double-check that the current PD webhook IPs are allowlisted for your region as per the KB note pointing to developer docs: Webhooks KB – IPs and safelisting note
Compare raw requests:
Temporarily repoint the PD webhook to a catcher (e.g., webhook.site or RequestBin) and also hit your endpoint with the same headers from curl. If the catcher receives PD’s POSTs but your endpoint disables again, the delta is on the endpoint/CDN/WAF path. This capture-method is suggested in the related community thread: Community thread
If it keeps auto-disabling, please reach out to Support so we can review delivery attempts on your account.
Here’s a rundown of what can be happening considering other cases we’ve had:
PagerDuty auto-disables a webhook after repeated delivery failures (non-2xx responses, timeouts, redirects, etc.). We’ve seen 3xx redirects (e.g., 302) trigger temporary disablement even when an endpoint “seems” fine from a browser or Postman. Example community case: Google Apps Script returning redirects caused disablements until the endpoint was fixed, and Support confirmed the 302s were the culprit. Reference: Webhook to Google Apps Script | Community
Similar reports show endpoints “working” in other contexts but PD disables the webhook because the target isn’t accepting the PD payload/auth as sent, or returns non-2xx under load. Example: ServiceNow case where the webhook kept disabling due to an auth issue on the endpoint: ServiceNow Community thread
High-signal next checks (even if things look “unchanged”)
Eliminate redirects entirely:
Ensure the exact webhook URL returns 2xx directly (no 3xx, no HTTP→HTTPS hop, no trailing-slash redirect, no CDN edge redirect, no auth redirect). PD won’t follow 3xx reliably and repeated 3xx can disable the sub. See similar case: Community thread
Return success fast:
Respond 200/204 within a few seconds, then do any heavy work asynchronously. Slow responses or timeouts count as failures. You can Send Test Event from the UI to verify: Webhooks KB – Test Event
Validate payload acceptance:
Make sure your endpoint accepts PD’s JSON as-is. Common silent failures:
400 due to schema validation differences
415 due to strict Content-Type checks
413 if request body limits are too small
401/403 if your server now requires auth but the PD webhook has none
Check TLS and URL invariants:
No expired/rotating cert chains, no SNI/cipher suite issues, and the hostname in the webhook URL hasn’t started redirecting to a different host. DNS or CDN config shifts can introduce 3xx at the edge without a firewall change.
WAF/CDN rules:
WAFs sometimes block PD’s IP ranges, rate-limit, or challenge requests differently than Postman. Double-check that the current PD webhook IPs are allowlisted for your region as per the KB note pointing to developer docs: Webhooks KB – IPs and safelisting note
Compare raw requests:
Temporarily repoint the PD webhook to a catcher (e.g., webhook.site or RequestBin) and also hit your endpoint with the same headers from curl. If the catcher receives PD’s POSTs but your endpoint disables again, the delta is on the endpoint/CDN/WAF path. This capture-method is suggested in the related community thread: Community thread
If it keeps auto-disabling, please reach out to Support so we can review delivery attempts on your account.
hey! Thank you ! The rundown has been proved to be most helpful!
We found our webhooks work under webhook.site, so we basically could rule out PagerDuty. After we bypass cloudflare between pagerduty and our backend, everything works again. I guess somehow cloudflare ate all the webhook requests and also didn’t show them on the dashboard.
