Skip to main content

In the configuration as code space, I’m building infra structure as code and configuration that will manage and control our config in PD (as much as possible).  The main driver is to manage the configuration of this critical piece of operations more securely and without the human-in-the-loop ClickOps via the PD portal.

I’m running into trouble using the PagerDuty Terraform provider when you use it to create multiple  PD resources (users, services, schedules,etc) in bulk (i.e. using for-each in the terraform resource)

We get failures back from the PD provider along the lines of:


│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to
│ pagerduty_user.service_usersr"<redacted>@<email>"], provider
│ "providerr\"registry.terraform.io/pagerduty/pagerduty\"]" produced an
│ unexpected new value: Root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Question is - has anyone experienced something similar using the PagerDuty Terraform Provider and how did you work around it?

 

Sample Terraform Code

terraform {
  required_providers {
    pagerduty = {
      source  = "pagerduty/pagerduty"
      version = ">= 2.2.1"
    }
  }

  backend "azurerm" {
    use_msi = true
  }
}

variable "<redacted>_config" {
  type = string
}

variable "pagerduty_token" {
  type = string
}

provider "pagerduty" {
  token = var.pagerduty_token
}

locals {
  all_service_data = jsondecode(file(var.<redacted>_config))
  dev_oncall_users = local.all_service_data.serviceConfiguration.nonProduction.dev.onCallSchedule
  qa_oncall_users  = local.all_service_data.serviceConfiguration.nonProduction.qa.onCallSchedule
  service_oncall_users = setunion(
    local.dev_oncall_users,
    local.qa_oncall_users
  )
}

# Create a PagerDuty user
resource "pagerduty_user" "service_users" {
  for_each = { for user in local.service_oncall_users : user.email => user }
  name     = each.value.name
  email    = each.value.email
}


I note that all the PD terraform examples create resources one by one, which obviously can’t be used at scale in an enterprise.

Hi ​@3dmundbarton,

Thanks for reaching out and for the detailed context and code sample! I’m also sorry it took us a while to reply to your issue, and I will try to provide you with references if you or anyone runs into this issue in the future.

What you’re running into is a known issue with the PagerDuty Terraform provider (and other providers) when creating multiple resources in bulk. The root cause is usually eventual consistency in the PagerDuty API, meaning that after creating a resource, the API may not immediately return it on a read, so Terraform thinks it’s missing and throws the “Provider produced inconsistent result after apply” error.

A few things you can try:

  • Retry the apply: Often, just running terraform apply again will work, as the resources have had time to show up in PagerDuty.
  • Import resources: If a resource was created but Terraform lost track of it, you can use terraform import to bring it into state and then re-apply.
  • Manual cleanup: If a resource is stuck, you may need to manually delete it in PagerDuty and try again.
  • Batch your applies: If possible, try creating resources in smaller batches to reduce the chance of hitting this issue.

This is a known limitation and there are open issues and discussions about improving the provider’s retry logic (see this GitHub issue). For now, the above workarounds are your best bet.

Let us know if you need help with any of these steps or if you’re still stuck!

Have a nice day!

 

Reply


Terms & ConditionsCookie settings