Alerts are being surpressed/aggregated with a resolved incident due to a too-general dedup_key
I'm having trouble with alert suppression in PagerDuty and need some help.
I created an incident with a very general dedup_key, and now all similar alerts being sent to that event orchestration are being suppressed and aggregated with this resolved incident. I've updated the event orchestration to use more specific dedup_keys for future alerts, but the resolved incident still has the old dedup_key and is supressing new alerts. Alert grouping is disabled as I only want the event orchestration to use the dedup_key and nothing else.
How can I ensure new alerts create new incidents and aren't suppressed by the old dedup_key? The incident is resolved. It doesn’t look like I can delete the incident. Do I need to create a new event orchestration?
Page 1 / 1
@Christian VanderZiel Hello! I consulted the ProdDev team and found out that it’s not possible to dedup an event for a resolved incident. Maybe you have a rule to suppress events with certain fields or as a fallback?
Thank you! That’s good to know.
I re-examined my alerts and incidents.
Here is what I think happened based on what I’ve found:
Alert 1: Generated an incident, which was then resolved.
Alert 2: Had the same text as Alert 1. It was suppressed but not associated with the resolved incident.
Alerts 3-6: These alerts have similar (but not the same text) as Alert 2. These were suppressed and grouped under Alert 2. Below is an image of the grouping in the Alert Log that I am seeing.
Because the text of Alert 1 and Alert 2 are identical, I thought they were the same alert. I mistakenly reasoned that Alerts 3-6 were suppressed due to the dedup_key.
While Alerts 2-6 are not associated with the resolved incident, they are nevertheless grouped together. Is that grouping significant or related somehow to their suppression, or is this grouping arbitrary?
@Christian VanderZiel
While Alerts 2-6 are not associated with the resolved incident, they are nevertheless grouped together. Is that grouping significant or related somehow to their suppression, or is this grouping arbitrary?
In PagerDuty parlance, combining multiple events into a single Alert is called deduplication and is entirely based on the dedup_key . It’s possible that the events/data sent from your monitoring tool have the same dedup_key or you might be setting/overriding the dedup_key using an Event Orchestration rule.You can see which Event Orchestration rules are being applied to each event by clicking the “Show Details” links visible in the “Alert Logs” section of the screenshot you included.
Thank you! I had not noticed that it linked to the specific rule, which it seems to do so. That is very helpful!
In this case, it is a fallback rule.
It’s not critical to understand this, but the fact that they are being suppressed by the same rule cause them to group together?
