At our organization we generate PagerDuty incidents by using email alerts.
The emails are sent to service integrations. Each service has an individual email. So even without Global Orchestration, we can process the emails into useful incidents using the Service Orchestrations. We aren’t licensed for AIOPs, so we can’t currently use Global Orchestrations.
A typical email might from our SCOM monitoring and have a subject line something like  “ SCOM: Health Service Heartbeat Failure Resolution State: New” or from one of our storage systems like “Storage D1A: Warning filesystem <filesystem> at 80%”.
This email is sent to our PagerDuty integration email and processed using the Service Orchestration rules. For repeated alerts, we can set a custom dedup_key such as the server name or filesystem name. If the dedup_key matches on subsequent alerts, then PagerDuty appends the new alert to the old ticket.
In addition, the dedup_key allows us to close incidents indirectly by using an event rule:  “If events match … (event.custom_details.subject matches regex Closed)” and setting the Alert Behavior to ”Always resolve an alert” checkbox. This effectively allows us to put “Closed” in the subject line. If the email alert otherwise matches the dedup_key it will resolve the alert, which in turn resolves the incident.
All great so far. But what happens when I want to change an existing incident without closing it?
This is precisely the issue I’m having with storage alerts. We send an email first as a “Warning” when the filesystem is filling, but not yet full (say, 80% full). Then, when the filesystem is 100% full, we send out a “Critical” email.
If this second “Critical” email matches the dedup_key, the second alert is simply appended to the existing “Warning” incident without changing the incident priority. That’s not useful.
Of course, my first thought for the second “Critical” email was to use the event rule Step 2: What action(s) should be applied → Incident Data → Basic Event → Set incident priority to … <my custom priority>”.  This works as expected with NEW incidents. But if the dedup_key matches an existing open incident, it appears that the service orchestration ignores the Incident Data section of the rule.Â
Â
Is there some logic or mechanism that I am missing? Can I change an existing incident’s priority or notes with an email? Is there another mechanism perhaps in Global Orchestration or other AIOps-licensed features that would work?
So far, the only solution seems to be to automatically create a separate incident by adding the priority to the dedup_key.Â
