Event Orchestration - Notify after X number of Alerts?

Is that logic based on the Dedup key or is it events to a specific service?     We may have multiple events going to the same service, but the type of event (based on depud key) is where we would want this logic.

Sounds like we would need to make our service setup more granular

Yes, it would be events to a specific service after they’ve potentially cascaded through other rules.

There’s effectively no limit to how many services you can have in your account, so making things more granular based on the actions you team expects to take, or the SDLC step represented, or the environment, or whatever other characteristic can help you manage alerts and information better. 

So if you were trying to capture, say, failed software builds, you’d separate those out from the production alerts for the service they build for sure, and possibly use change events to track failures onto the production services for information.

Couple common examples:

• You can have a service based on environment, so “Production App A” and “Staging App A” get the same types of alerts for testing and verification, but staging is configured to notify at a lower urgency and to not notify overnight.

• For stuff like maybe ETL jobs, where the run is effectively the production workload, folks might divide those up with a different service for each step in the pipeline, especially if some part is particularly troublesome.

What you definitely don’t want is to have a single service that’s “TEAM A” and dump all kinds of random stuff in there and try to do more sophisticated things via rules. That makes the longterm management of the environment much more difficult, especially adding new services, decommissioning old services, or changing ownership of services, and gives you no way of using the Service Graph feature in a way that makes sense. 

Happy to chat in real time or set up a call if you want to talk through what you’re up to. Our team is community-team@pagerduty.com.