Intelligent Grouping - post Acknowledged alerts

We’re having issues with grouping where it groups new alerts to an already Acknowledged alert. We definitely do not want new alerts to go into an existing Acknowledged alerts.

We’ve confirmed that Intelligent grouping groups alerts upon initiation of the alert, which makes sense. Although it shouldn’t group after alerts are sitting in Acknowledged, correct?

Can someone at least confirm Intelligent grouping groups alerts “after” Acknowledgment or not? Grouping “after” Acknowledgment doesn’t make much sense.

How do we disable this if it actually does this? Our only other action is to turn off intelligent grouping.

Thank you

Hi Larry,

When Intelligent Alert Grouping is turned on for a service, it will automatically group incoming alerts to related open incidents. If you have not resolved an incident, then the alert grouping would still be possible.

You have the option to discontinue Alert grouping on a service. You can then manually merge incidents that you want grouped together. If you wish to stop the Alert Grouping, then you will just need to visit your service’s page and selecting Settings. Once there you can turn off Alert Grouping.

Thanks,

Abbott B
Technical Support Specialist

@abbott Thanks for your reply!
What we’re hitting now as an issue is that since IG is set to group alerts within 5 minutes of the last related alert, so it keeps on grouping. This could possibly go on indefinitely if we keep having related alerts.
Consequently, acknowledged alerts can have alerts grouping for a long time without us knowing unless we explicitly look.
What do you think we could do about this?
Time based grouping does the same thing, correct?
Content based is tough b/c it will group for 24 hours unless we get it perfect.
We’ve now had to turn IG off because of this issue. :frowning: What do you think?
Thank you

Hey Larry,

I understand your concern here about continued grouping into acknowledged alerts. There isn’t a way to turn this off. IG will group regardless of whether the existing incident is acknowledged or not. The intelligent grouping system is a learning system, however, so you might be able to train it away from this behavior by ungrouping previously grouped alerts. It will learn on responder behavior, and this could stop the post-ack grouping, but without a previous similar use-case where a customer has successfully done so, I can’t guarantee it.

Your best bet at this time, if you don’t wish to try training the grouping over the next few weeks is to turn it off. I’m happy to enter a feature request with our product team around your requirement that grouping not occur once an incident has been acknowledged. Can you tell me a bit more about your specific use-case and how making this change would ease a pain point for you? I would like to include it in the feature request. If explaining this would require you reveal information that you would rather not have on this public forum, you can write in directly to us at support@pagerduty.com and reference this ticket number: #347868.

I look forward to hearing from you.

Best,

Hello,

Hope it’s OK for you to let me add my 2 cents to this discussion. :slight_smile:

My understanding is IG aims to reduce alerting noise and have a better relationship between incidents (ie: tickets, here in PagerDuty) and real disruptions (ie: only one incident to ack/resolve for the same cause, even if many probes detect the disruption).
Not grouping an alert because the incident is acknowledged is not the sense. And for us will increase noise.

But: I agree with this: as the incident responder I want to be able to decide (optional) to be informed when a new alert is added/resolved into an incident
eg: be notified via my 1st notifications rules (or the last notification channel used for the incident) when an alert is added/resolved
eg: have a message into the linked Slack channel (if I set it) when an alert is added/resolved

And to be sure the incident responder checked all the alerts - even only to define if he had to split them into a new incident - until it becomes a habit, maybe add a confirmation when closing an incident with more than 1 alert.

Regards,
SĂ©bastien

PS: you should not have alerts added indefinitely as you should not have indefinitely new probes detecting the same issue, and the ones which trigger the incident should send the same event (using dedup key) until they resolve.

Hello SĂ©bastien,

Thank you for joining the thread, thats what Community space is all about :slight_smile:

We appreciate your feedback on this and to each my colleague Tatiana I can see where the concern comes from and I have shared it with our product team letting them know what updates you’d like to see in the future.

John

@sébastien.de.nef I understand what you mean and thanks. Although providing an option for not grouping anything into an acknowledged alert will just provide a bit of safety for customers. If they want less noisy keep grouping, if they want more noisy more safe without turning IG completely off, it seems like a good feature toggle to have.
@Inactive-Member-7431265 , @abbott - Yes, please enter this feature request. Thank you so much!
Yes, this would keep grouping before ack’d, then stop after ack’d so that an on-call tech knows that something they’ve already ack’d will not get new alerts that they haven’t seen. This would greatly increase our confidence in the Pagerduty IG feature.

THanks again!

Hey all,

Will do ! Thanks for contributing on our community space :slight_smile:

John