PagerDuty Visibility Best Practices

howto
official
tips
(Hadley) #1

PagerDuty’s Visibility configuration presents multiple options that you can use to tailor metrics for teams across your organization. This guide includes some best practices and a common use-case to illustrate what you can do with Visibility and how to use it most efficiently.

Preparation

A primary goal of PagerDuty Visibility is to provide non-technical stakeholders with context when one of their business services is affected by an incident. To achieve this, it is crucial to set up your PagerDuty business services and impact metrics correctly so they are intelligible for all stakeholders regardless of their role.

When setting up your business services for the first time, ask yourself these questions:

  • Will this business service name and its impact metric make sense to executives and non-technical stakeholders?
  • Will this business service name and its impact metric answer questions that executives and non-technical stakeholders typically have about incident impact?

Example:

  • Problem: Executives are having difficulty receiving timely and clear information on the business impact of an incident.
  • Their question: “How many online customers were impacted by this incident?”
    • Business Service Name: Shopping Cart
    • Impact Metric Name: “Number of successful checkouts”

Walkthrough

Business Services

The first step in configuring Visibility is determining your Business Services. It’s important to consider what your core services are and where they’re kept. If you use a CMDB, ITSM or an internal/external status page, these are great places to look when deciding what you’re going to add to PagerDuty. Visibility will help you to align both technical and non-technical teams on the status of these core components.

Business Services Checklist

Your business services should meet the following criteria:

  • Must have unique names.
  • Must be recognizable to technical and non-technical teams.
  • Should represent a core component of your business.
  • May be owned by multiple teams.
  • Supported by one or more technical services represented in PagerDuty.
  • Must have defined health metrics that would be meaningful to non-technical stakeholders.

In this example, we will be using a Shopping Cart as our business service:

  1. From the Configuration menu, select Business Services.

  1. Enter your business service Name—ensure that it is a unique name that will be recognizable by non-technical teams. Click Create New Business Service.

  1. On the Edit Service Details screen, enter a Description and the Owner as yourself or a point of contact for the business service. (Optional)
  2. Read through the next section to continue on this screen.

Technical Services

After defining your Business Services, the next step is to link them to the Technical Services they depend on. A Technical Service refers to a traditional PagerDuty service that corresponds to the underlying components in your infrastructure. For information on how to create or edit a service, see our Services and Integration Guide.

Technical Services Checklist

Your technical services should meet the following criteria:

  • Must have unique names.
  • Directly responsible for the health of the Business Service.
  • Owned by one team. If two services could never be owned by two different teams, consider combining them into one service.

Walkthrough continued:

  1. Under Services this business service depends on, add any applicable service that would impact this business service. In our example, the Shopping Cart business service depends on the following Technical Services to remain functional: Customer Tickets, Database, App Server, and Stripe:

  1. Under Services relying on this Business Service, apply any (preferably business) services that this service would depend on and click Save Changes. In our example, the services Sales Retention, Customer Activity, Customer Testing and Customer Retention rely on the Shopping Cart business service:


7. Read through the next section to continue.

Impact Metrics

The final step is identifying the impact metrics that will be used, and the business services to which they are related. Each impact metric should be intelligible to a non-technical stakeholder and should help that stakeholder to understand one or more of the following:

  • Whether and to what extent the business service is disrupted vs. functioning properly.
  • The impact on customers from any disruption that is occurring.
  • The impact on the business from any disruption that is occurring.

Impact Metrics Checklist

Your impact metrics should meet the following criteria:

  • Names do not need to be unique (Example: Latency as a metric name could exist on multiple business services).
  • Metrics are measurable and clear.
  • They are consistently monitored—in order to update in real time, the metric should be frequently updated.
  • Must be directly correlated to the types of failures associate with your alerts (Example: A marketing metric containing the number of monthly sign-ups should not be used if you’re monitoring the service for availability).
  • Two to four impact metrics per business service is ideal.

To assess the functionality of our Shopping Cart, we will measure the total dollar amount of items in shopping carts across all visitors per second. This metric also allows us to see the impact to the company’s sales during wartime.

Walkthrough continued:

  1. Click on the Configuration menu and select Impact Metrics.
  2. Select Create New Impact Metric and input a meaningful name along with a description.
  3. Within the display settings, you can adjust how the impact metric graph will appear on the Visibility mobile dashboard.
    a. The y-axis minimum and maximum values define the range shown.
    b. The unit type is displayed with the current value in the title.
    c. The precision is used for values displayed on the y-axis and on data points.
    d. The aggregation determines the value used when one data point on the graph represents multiple observations over a given time interval.

  1. Under Select the Business Services where this Impact Metric should be displayed, add one or more business services for which the impact metric should appear. In our example, we have selected Shopping Cart as the business service:

  1. Once done, select Save Changes to this Metric. In return, you will be provided with a corresponding corresponding ID for each impact metric. You will need this ID in order to submit observations to the REST API.
  2. To send data to an impact metric you will first need a valid PagerDuty API token and the ID of the impact metric. For general information about interacting with the PagerDuty API, including information about API tokens and authentication, please refer to Using the REST API. The ID for the impact metric can be found in the Impact Metrics list on the web application.

Request Method

POST

Request URL

https://api.pagerduty.com/business_impact_metrics/<impact_metric_id>/observations

Request Body

{
  "observation": {
    "value": <number>,
    "observed_at": "<timestamp>"
  }
}

Note

  • value is a JSON number
  • observed_at is a string specifying a date and time in ISO 8601 format. The time, including seconds, is required. See the PagerDuty API docs on types for more details.

Response Code

HTTP 201 will be returned on success.

  1. Now that you have associated an impact metric with a business service, you will be able to see it on the Visibility mobile dashboard. To view, navigate to ☰ → Visibility and select the associated business service. In our example, the impact metric of total dollar amount of items in shopping carts across all visitors per second is listed with the Metrics for the Shopping Cart business service.

1 Like