pdagent urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]

Hello @guney, it’s likely that you need to update the certificate bundle on your server – here’s more info on this:

Customer Notification - Action Required - DigiCert root CA certificates

Background

On March 5, 2024, PagerDuty rotated the TLS certificates for PagerDuty subdomains (something.pagerduty.com) as well as for our REST and Events API endpoints (api.pagerduty.com and events.pagerduty.com). While this is a normal, annual operation, we received feedback from customers that older systems were unable to validate the new TLS certificates.

Investigations determined that these new certificates were signed by our certificate provider, DigiCert, using a different root CA certificate than the previous certificates. This root certificate, “DigiCert Global Root G2”, was issued in 2013. It was added to the Mozilla CA certificate bundle, widely used as a source by Linux distributions, in June 2014, after which distributions could add it to their own CA bundles. Digicert began using it for certificate issuance in March 2023.

Older systems are affected

Since systems and software using CA certificate bundles older than that time period would not have the DigiCert G2 root, they would be unable to verify the new TLS certificates which PagerDuty put in place on March 5.

This includes systems running PagerDuty’s pdagent service, in two scenarios:

1. Version 1.6 of pdagent and older were released in or before 2013, prior to inclusion of the G2 root, and which use a root certificate bundle distributed with pdagent, regardless of the system bundle; and
2. Versions newer than 1.6, where the system’s CA certificate bundle does not include the DigiCert G2 root.

While versions of pdagent newer than 1.6 are distributed with a root certificate bundle that contains the DigiCert G2 root, a bug in pdagent causes it to fall back to the system CA bundle.

Action required

If pdagent or any other API clients are unable to connect to PagerDuty because of certificate verification failures since the rotation on March 5, you must ensure that the CA bundle used by your software contains the DigiCert G2 root certificate.

If you are running pdagent 1.6 or older:

1. Obtain an updated root bundle from: https://raw.githubusercontent.com/PagerDuty/pdagent/update-certs/pdagent/root_certs/ca_certs.pem

2. For each of your pdagent instances:

3. Replace the existing file at “pdagent/root_certs/ca_certs.pem” with the downloaded one

4. Restart pdagent

If you are running a pdagent version newer than 1.6, update your operating system’s CA certificate bundle and restart pdagent.

• For Ubuntu, run: apt update && apt upgrade ca-certificates
• For CentOS, run: yum update ca-certificates

If you are affected because software other than pdagent is unable to validate the new certificates, you will need to update the CA bundle used by that software. Contact the vendor of that software for further information.

We do not recommend disabling certificate validation to work around this issue. We recommend updating your system CA bundles on a regular basis, as certificate authorities regularly make changes to the chain of trust used to sign their certificates.

We appreciate your patience during our investigation while we determined an appropriate response to this situation. We wanted to make sure we had accurate information about the overall impact to customers and appropriate recommendations for users of pdagent and other tools that reach out to PagerDuty but run on customer infrastructure.

We apologise for any inconvenience this has caused. ​⁠For any questions, comments, or concerns, please contact us at ​⁠support@pagerduty.com​⁠.

Thanks @xenda amici , I appreciated for your help !!! , After I update ca_certs.pem that works for us !!