Global Event orchestration. Cannot resolve incident, due to no fields sent to forward resolve event to the same service

#1

I am in doubt about how to resolve incidents using global event orchestration.

I have integration with uptimerobot/zabbix, and I use a summary field to forward incidents to the proper service, but then when the incident revolved through the API, there are no fields at all except basic ones such as routing_key and dedup_key… as result resolve event not routed to the proper service and incident have to be closed manually.

Is any way to solve the issue?

#2

Yes, if you know what those incoming events/alerts look like, you can create a specific rule to look for that condition, ensure the dedupe key matches the original trigger event/alert, and set the rule to “always resolve”. See here: https://support.pagerduty.com/docs/event-orchestration#alert-actions

1 Like
#3

@dmcclure Thanks for the response.
I don’t get your answer, could you please elaborate?

I have one Global Event Orchestration that forwards events to the proper service based on the summary field.

  1. Trigger event -> The Event summary field has the name of the service -> a list of rules when the summary contains the “name” it is routed to the proper service and opens an incident.
  2. Resolve event -> the Event has no payload at all except dedup_key and event_action: resolve. as result, I cannot forward it to the proper service and close.

The same dedup_key does not close incident when it belongs to another service and in my case all “resolve” events belong to the so-called “last resort service”.

  1. Trigger event

    {
    “payload”: {
    “summary”: “hosting”,
    “severity”: “critical”,
    “source”: “name: service_name”
    },
    “routing_key”: “xxxx”,
    “dedup_key”: “samplekeyhere”,
    “links”: [
    {
    “href”: “https://example.com/”,
    “text”: “Link text”
    }
    ],
    “event_action”: “trigger”
    }

  2. resolve event
    {
    “routing_key”: “xxx”,
    “dedup_key”: “samplekeyhere”,
    “event_action”: “resove”
    }

#4

You’d need something in the resolve event so you can route it to the correct service. Without this there’s no way to match up with the trigger event using the dedup key. Can you add more metadata to your resolve events to help with this?

#5

I wish I can, but as I mentioned it is a 3rd party integrations. I can do it for zabbix, it has some scripting for it, but unfortunately cannot do it with uptimerobot and others.

Looks like Global Event Orchestration pretty useless thing, as it is not possible to close incident, even in the official documentation the “resolve” incident as simple as I described above.

I wish dedup_key to be global, not per service, otherwise, it makes no sense in most cases.

#6

hello,
We are facing the same issue. Did you find a solution?
We switched to event orchestration yesterday and since then the incidents no longer auto-resolve.
We are also using zabbix.
It worked fine with the event ruleset without having specific rules for it.

thanks