[Stackdriver integration] disable resolve events

We use Stackdriver integration for PagerDuty.
Some incidents on Stackdriver are generated from error-log based metric. The metric value is increased by PagerDuty when it discovers an error in the log during some time frame.
If no error happens in the next time frame, the metric is decreased back.
The problem is that as a result of metric decreasing , the incident is marked as “resolved” on Stackdriver and it also sends a “resolve” event to PagerDuty causing PagerDuty to resolve the incident.
Is there any way to ignore these “resolve” events on PagerDuty? (this is not possible on Stackdriver)

Hello,

Thanks for reaching out on community. Have you tried suppressing Event Rules to suppress an Alert coming in? You could take a specific phrase, that comes in with Resolve Events and use this to suppress the event.

Check out the Event Management support doc above and let us know if this helps.

John

Thank you for help, John.

I tried “Event Rules” approach, but it didn’t work.
All the Resolve events from Stackdriver have “returned to normal” and “Incident lasted” phrases in it’s description (see example https://miro.medium.com/max/1992/1*ETNEmjmQP9Kb6TWK2Qfgsw.png).

As long as I understand, this description is parsed by PagerDuty into “Summary” string on the JSON representing the event. You can see this JSON when you create an “Event Rule” to the right of “When events match these conditions” dialog (under something like “View a recent API event…”).

I tried to put into “When events match these conditions” event fields “returned to normal” or “Incident lasted” (for example, Summary contains returned to normal) and choose “Suppress Alert” at “Do these things” dialog, but it had no effect. When Resolve event arrived, the Alert just changed it’s state from “Triggered” to “Resolved”.

Hi Ivan,

We should be able to help you get these Resolve events to be suppressed and not resolve out any of your open incidents.

I’m curious to see an event that came in that resolved a recent incident as well as review the current Event Rule you have in place on the service that this incident triggered on.

I wouldn’t ask for these details from your account here, but if you can reach out to us over email with a recent incident that had been automatically resolved, we’d be happy to take a look. If you can make a reference to this Community post as well when you write in, this context will help our team troubleshoot efficiently.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.