Logz.io: parsing JSON keys in PagerDuty

Hello,

I’m using the Logz.io ‘PagerDuty’ integration to send alerts to PD. Everything works fine, but unfortunately the JSON sent by logz.io is sparse at best – they are encoding some metadata (aka, prometheus labels) in JSON keys, as shown below.

Is it possible to use dynamic field enrichment extraction or something else to extract metadata from a JSON key? My goal is to use Global Rulesets in PagerDuty to route alerts based on namespace or cluster or similar metadata. (See below for an example of the JSON that logz.io sends to PagerDuty.)

{
  "client": "Logz.io Metrics Alerts",
  "client_url": "<valid-logzio-url>,
  "contexts": [
    {
      "src": "https://s3-us-west-2.amazonaws.com/slack-files2/bot_icons/2015-11-19/14909590803_48.png",
      "type": "image"
    }
  ],
  "description": "Container Restarts",
  "event_type": "trigger",
  "incident_key": "i4s04lx696e28adhi6y49s981mzmv1mmpxrgs4na",
  "service_key": "kceeaq7e91nyb9lmhvrek6rvpd9g449v",
  "details": {
    "Account": "myAccount",
    "Description": "A container has restarted 3 or more times in the last 20 minutes.",
    "{cluster=\"eks-cluster-name\", container=\"my-container\", namespace=\"apps\", pod=\"my-pod-dfd8af8sf8-38ujd\"}": "1",
    "{cluster=\"eks-cluster-name\", container=\"coredns\", namespace=\"kube-system\", pod=\"coredns-dfd8af7ffy-8jfe8\"}": "1"
  }
}

Thanks,
Jeffrey

Yes, that’s exactly what DFEE is for to help you create normalized enriched alerts. If you have the ability, I’d recommend updating Prometheus AlertManager to send proper JSON values and all of our PD-CEF fields instead of the big string it does today by default so you don’t need to do as much regex in PD.

Thanks Doug. I was able to use DFEE to extract metadata directly from the details map using a regex (basically treating the entire data structure as a string). Seems like I could do something similar for Ruleset matching.

It’s ugly though, as you said. I’m also working with Logz.io (separately) to see if they can emit alert metadata in PD-CEF format. :crossed_fingers:

Thanks,
Jeffrey

It looks like you can create a new Notification Endpoint and define a custom payload in Event API v2 format and map in their parameters. (https://docs.logz.io/user-guide/integrations/custom-endpoints.html) Give it a try and see if it works!