COMPOSITE ALARM in cloudwatch not triggering PD integration

Hi,

I’m using a new COMPOSITE alarm type introduced in March 2020 - https://aws.amazon.com/about-aws/whats-new/2020/03/amazon-cloudwatch-now-allows-you-to-combine-multiple-alarms/

I have the alarm triggering an SNS topic with 3 subscriptions: email, lambda & HTTPS to my Cloudwatch Integration URL which looks like this:
https://events.pagerduty.com/integration/XXXXXXXX/enqueue

I’m receving the lambda and email notifications, but a PD incident is not created. The HTTPS subscription is confirmed (using the auto-confirm option in Terraform) but it does not create an incident.

I am able to manually create an incident via the PD UI.

Is there perhaps an issue with the contents of the message coming from a COMPOSITE alarm that PD is failing to recognize?

I’ve also created a support request as I’m pretty sure I have this configured correctly - but figured someone here may know as well. We’re in our free-testing period, and this would definitely be a deal-breaker.

Thanks!

1 Like

I was able to log the SNS requests to cloudwatch, and confirmed that the message was delivered to PD, but still no incident:

{
    "notification": {
        "messageMD5Sum": "7b449add3c111111117bf1cdf93a6ac",
        "messageId": "81384914-1111-2222-3333-1e0d04abecc8",
        "topicArn": "arn:aws:sns:us-east-1:12341231234:mycompany-us-east-1-prod-infra_critical",
        "timestamp": "2020-04-22 00:47:55.776"
    },
    "delivery": {
        "deliveryId": "dec8f3b9-1111-2222-3333-a4a0f6ab38e0",
        "destination": "https://events.pagerduty.com/integration/XXXXXXX/enqueue",
        "providerResponse": "Accepted",
        "dwellTimeMs": 416,
        "attempts": 1,
        "statusCode": 202
    },
    "status": "SUCCESS"
}

still no incident created… this is on a new vanilla instance… all I’ve done so far is created a service, and added Cloudwatch Integration… This message is being created by a COMPOSITE alarm triggering SNS which then hits the URL.

HI Dan,

As this is a subject that requires more sensitive information, can you reach our directly to the support team on support@pagerduty.com with the details of what service you are sending it to on your Pagerduty account.

Once I have this I can take a closer look.

Kind regards,
John

Hi John, I’ve already sent one in: #253650

They’re telling me as I suspected, that the content of the message is different and not being accepted by PagerDuty.

I would imagine that a composite alarm should be accepted the same as a regular alarm, correct?

They have provided me with a workaround using a custom event type, but this will likely be quite a bit more difficult (and probably unreliable) for me to implement.

No word in the ticket that I see about any commitment or plans to update on the PagerDuty side.

Hi Dan,

Thanks for that - I have let them know about your community ticket also but will keep the back and forth in one place for now to ensure no context is lost!

Cheers,
John