Update exisiting incident instead of creating a second one?

New to PagerDuty so apologies if this is obvious.

Setup: New Relic monitoring a system process on two EC2 instances, if it stops on either of them an alert is sent to PagerDuty. If both stop two alerts are sent. It’s the same New Relic condition monitoring both instances.

What I want to happen:

  • If the process on Server-A or Server-B goes down (not both) a P2 incident is raised.
  • If then the process on the second server goes down (so now both down) the same incident is raised to a P1.
  • If the process on both servers goes down at the same a P1 incident is raised.

Problem is right now PagerDuty will open two separate incidents. Is there a clever way for it to check for an open one and if it matches a criteria, say host name, it just updates the existing one instead of creating a second one?

I can see if we were using an API for reporting we could use the dedup_key but I don’t think this applies to alerts from New Relic.

Hi Alan,

You can link incidents by sending them over with the same de-duplication key. When an alert is triggered, a second alert with the same dedup key will be merged into the original, but can update several of its parameters, including the priority.

All incidents from New Relic generate a de-duplication key. I do not know if you can customize the event coming out of New Relic to use a pre-determined dedup key. This is also passed as the incident_key by some integrations. You will want to consult NR documentation or their support to find out if you can manipulate alerts from two processes to send the same de-dup key. Otherwise, PagerDuty generates a random key for all incoming alerts that is a UUID.

Here’s some documentation on how de-duplication works in PagerDuty. You can even set a custom de-dup key if you aren’t able to configure New Relic to send one that will fit your use-case. Our Dynamic Field Extraction & Replacement feature allows you to extract data from any field in order to add it to another–such as the dedup_key or incident_key parameter.

I hope this helps. If you have any further questions, please don’t hesitate to reach out.