Re-trigger incidents after they've been resolved

According to https://support.pagerduty.com/docs/incidents#section-incident-states,

Resolved - A resolved incident has been fixed. Once an incident is resolved, no additional notifications are sent and the incident cannot be triggered again.

I’m using PagerDuty + Prometheus and I would like to trigger new incidents whenever an incident is resolved, but the underlying signal hasn’t been resolved (i.e. Prometheus is still firing alerts that are being reported to PagerDuty).

Is this possible? I know incidents in are grouped by group_by (https://www.pagerduty.com/docs/guides/prometheus-integration-guide/), but in this case I’m looking to re-trigger incidents that have the same set of alert labels.

Thanks!

HI Ravi,

Hope I am understanding this correctly but, by “retrigger” do you mean open back up an Incident that was already ready resolved?

If so, I am afraid this is not possible, once and Incident has been marked as Resolved you will need to have a fresh one Open up, we are unable at this point to “unresolve” and Incident.

Let me know if I misunderstood anything.

John

Hi John,

Yes that’s pretty much what I’m looking to do. In the scenario where an incident is resolved, but the underlying metric (i.e. alert from prometheus) is still in alerting range. It doesn’t really matter if it’s an existing or new incident – the end goal is trigger another alert for the on-caller as long as the underlying metric is alerting.

Ravi

Hi Ravi,

Thank you for clarifying for us.

The system should actually work for you in the way you’re hoping, as you can not deduplicate alerts into ones that are resolved.

Whenever trigger events are sent in with the same Dedup Key, which with the Prometheus Integration relates to that group_by value you mentioned, the events will deduplicate into any open incidents there may be with that same key. If there is no open incident with that dedup_key or the initial incident with that dedup_key has been resolved, the next trigger event to come in will trigger a new incident.

We appreciate your feedback about wanting such an event to be able to re-trigger such an incident, that had already been resolved, though here

Thanks for clarifying Malcolm!

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.