Skip to main content

Short Version

If a “resolve” message is sent to the /enqueue endpoint of the v2 API and the dedup-key finds no corresponding triggered alert, is the message just silently dropped?


Long Version

As part of our custom CI/CD system we send deployment messages to both the /enqueue and /change/enqueue endpoints of the v2 API.


The idea is that both successful and failed deployments send messages to both endpoints but for the /enqueue endpoint we have a rule that suppresses the successful messages because we don’t need to see them. The failed deployment messages are not suppressed and do create incidents which are resolved by subsequent successful deployments.


However, I can’t actually find any suppressed alerts for successful deployments to the /enqueue endpoint so I don’t think we even need the rule to do the suppression.


We currently use a very simple dedup-key like <app>-<stage>-<region>. I’m thinking that means that the API can only really use the message if a corresponding incident is open using that dedup-key and it uses it to resolve the incident. Further, for all the times we send the “resolve” message but there is no open incident the message just gets silently dropped?


Hopefully that gives context to the short version above.

Hi Polski.


Additional “resolve” messages are ignored when there is no open alert or incident.


If your CI/CD system is sending events with “resolve” as the event_action when the build succeeds, they are dropped and you won’t see them in the alerts table.


That will work for what it sounds like you’re doing, but it’s kind of a blunt instrument. If you have a way to make the status of the previous build available to the current build, successful builds that follow other successful builds could send only the change event. Successful builds that follow a failed build could also send the resolution message.

Thanks @mandi walls - keep doing great work.


I’m sure we can look at our CI/CD solution but we didn’t want to introduce any statefulness.

Make sure the endpoint’s “resolved” message matches the open issues to avoid possible silence so you can go to toquesdecelular.com. The role of the deduct key in incident resolution warrants careful consideration for effective alarm handling.

Reply


Terms & ConditionsCookie settings