CertificateError: hostname 'proxy' doesn't match either of '*.pagerduty.com', 'pagerduty.com'


(Richard Homewood) #1

When running pdagent behind and proxy and adding the relevant env variables to: /lib/systemd/system/pdagent.service and reloading the service and daemon, we get the following error in /var/log/pdagent/pdagentd.log and the send event doesn’t occur:

CertificateError: hostname 'spektrix-proxy' doesn't match either of '*.pagerduty.com', 'pagerduty.com'

spektrix-proxy refers to a proxy server we are behind:

My setup:
openssl 1.1.0
pdagent: 1.4
python: 2.7.9

the stack call from the DEBUG log:

2017-09-26 16:31:53,106 ERROR   SendEventTask        pdagent.sendevent    Certificate validation error while sending event: hostname 'spektrix-proxy' doesn't match either of '*.pagerduty.com', 'pagerduty.com'
2017-09-26 16:31:53,106 DEBUG   SendEventTask        pdagent.sendevent    Traceback:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pdagent/sendevent.py", line 90, in send_event
    response = self._urllib2.urlopen(request)
  File "/usr/local/lib/python2.7/site-packages/pdagent/thirdparty/httpswithverify.py", line 153, in urlopen
    return url_opener_cache[ca_certs].open(url, **kwargs)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/site-packages/pdagent/thirdparty/httpswithverify.py", line 136, in https_open
    return self.do_open(self._proxyHTTPSConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1195, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1082, in _send_request
  File "/usr/local/lib/python2.7/httplib.py", line 1038, in endheaders
  File "/usr/local/lib/python2.7/httplib.py", line 882, in _send_output
  File "/usr/local/lib/python2.7/httplib.py", line 844, in send
  File "/usr/local/lib/python2.7/site-packages/pdagent/thirdparty/httpswithverify.py", line 120, in connect
    match_hostname(self.sock.getpeercert(), self.host)
  File "/usr/local/lib/python2.7/site-packages/pdagent/thirdparty/ssl_match_hostname.py", line 149, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
CertificateError: hostname 'spektrix-proxy' doesn't match either of '*.pagerduty.com', 'pagerduty.com'
2017-09-26 16:31:53,107 INFO    SendEventTask        pdagent.pdqueue      Not processing any more events this time

is there something odd going on with python?

I can successfully openssl to the site and this is the response back:

openssl s_client --proxy spektrix-proxy:80 -connect events.pagerduty.com --port 443
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=19:self signed certificate in certificate chain
Certificate chain
 0 s:/OU=GT12858685/OU=See `www.rapidssl.com/resources/cps` (c)14/OU=Domain Control Validated - RapidSSL(R)/CN=*.pagerduty.com
   i:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G3
 1 s:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G3
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
Server certificate
subject=/OU=GT12858685/OU=See `www.rapidssl.com/resources/cps` (c)14/OU=Domain Control Validated - RapidSSL(R)/CN=*.pagerduty.com
issuer=/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G3
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
SSL handshake has read 3843 bytes and written 343 bytes
Verification error: self signed certificate in certificate chain
New, TLSv1.2, Cipher is E***
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
    Protocol  : TLSv1.2
    Cipher    : E***
    Session-ID: C3***
    Master-Key: 3***
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 68 ca 4f c8 25 9f d6 07-c5 72 90 e6 ec 55 0c f4   h.O.%....r...U..
    0010 - 79 b9 38 b4 8d b8 47 9f-4d 25 b3 66 45 e1 c0 8f   y.8...G.M%.fE...
    0020 - db 18 28 21 ab c3 31 bc-40 56 aa 61 c7 f5 f9 52   ..(!..1.@V.a...R
    0030 - a9 ac fe 7b ec 7e 14 e5-77 11 ce 4d b8 fe 93 f4   ...{.~..w..M....
    0040 - bd 58 ea db f7 23 a6 24-63 00 23 0a bc 8e b6 33   .X...#.$c.#....3
    0050 - dc a4 ee d5 fd 78 f9 4c-44 fa cf 7a 66 77 49 69   .....x.LD..zfwIi
    0060 - 64 7f cd d9 4a 69 94 88-ed 15 13 e9 ef 87 2d 8e   d...Ji........-.
    0070 - 63 77 12 d0 8d e5 69 a9-6b 85 bf e3 9e 55 79 c2   cw....i.k....Uy.
    0080 - 39 5d 33 dd eb bb d5 90-a3 31 9c ce 76 4b 56 db   9]3......1..vKV.
    0090 - 73 8e 80 07 4b 77 80 ab-6b 8c 13 90 e0 e5 95 0f   s...Kw..k.......
    00a0 - b2 86 0d 43 7b 82 b9 45-e4 8b 07 3d 32 d7 00 29   ...C{..E...=2..)

    Start Time: 1506440235
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no

Heads up: The forum admins have edited this post to remove some certificate information, and to correct a formatting issue with log output that made this post difficult to read.

(Richard Homewood) #2

Turns out this is a known bug in the underlying Python libraries and a fix has been submitted to the PagerDuty engineers hopefully for a fix in the next version of pdagent.

(Demitri Morgan) #3

Hi Richard,

You are correct. This is a known bug in the underlying Python libraries, and it’s one of the issues we’ll be addressing for a future release.

(Bohdan Sukhomlinov) #4

Hi, How soon this new release will take place?
Is there a way to walk around this bug?

(Demitri Morgan) #5

Hi @shellshock1953,

Here is a work-around we’ve used before with success. However, it is very brittle and inelegant; it requires modifying /etc/hosts, for which you will need root access to the server.

  1. Obtain the IP address of the proxy server.
  2. Add a new entry in /etc/hosts (let’s say the IP address is lan-proxy.pagerduty.com
  3. Update the PagerDuty Agent daemon environment configuration so that it uses the hostname lan-proxy.pagerduty.com as the proxy server host.
  4. Restart the agent to apply the configuration change.

To reiterate, the issue here is that the underlying library is comparing the remote server’s certificate’s common name (the wildcard *.pagerduty.com) against the proxy’s hostname, and this is not the correct / desired behavior. The above procedure should thus “trick” ssl_match_hostname to act as though it were connecting to a host whose name matches the wildcard.

(Naman Kaushal) #6

Is there a way to permanently fix this bug?

(Joe Pettit) #7

Hey Naman,

Thanks for following up here. For the time being, the workaround Demitri mentioned is the only solution we can offer. We are working towards a more permanent resolution but we do not have an ETA on when this will be completed.

Thank you.

(Demitri Morgan) #8

Hi Naman,

After some more research, I determined an adequate solution would be to use self._tunnel_host instead of self.host in pdagent.thirdparty.httpswithverify whenever a HTTP proxy is in use.

A pull request to fix the original issue is open and pending review:

(Demitri Morgan) #9

Also, I’d just like to add a bit more here, in case anyone was worried about whether the ugly workaround I shared last October might pose a security risk (it shouldn’t).

We enforce certificate validation in PagerDuty Agent by passing the keyword argument cert_reqs=ssl.CERT_REQUIRED to ssl.wrap_socket, and use an included certificate bundle for validation. These options ensure that the certificate’s validity is checked for a signature from our CA. If a certificate were not properly signed, ssl would terminate the connection and raise an exception.

Moreover, in the context of the host name validation: even though it looks phony and uses a FQDN that locally resolves to a LAN address, keep in mind that the common name of the certificate would still have to be *.pagerduty.com or lan-proxy.pagerduty.com. To reiterate: the signature must be issued from the CA included in the certificate bundle. Thus, it is unlikely that an attacker would be able to get a signature on such a certificate from DigiCert (the CA that signed the current certificate used in our Events API), or any legitimate, trusted certificate authority for that matter.

Something worth noting, however, if running Python tools in general: if your system runs on an earlier version of Python (including all of 2.6), ssl might not perform certificate validation by default (without setting cert_reqs) and so might be vulnerable to MITM attacks.

If your system uses Python 2 version 2.7.9 or later or Python 3 version 3.4.3 or later, certificate validity should be checked by default in all TLS wrapped network sockets. This was implemented after PEP-476 and issue 22417. See the 2.7.9 changelog / 3.4.3 release notes for further details.

(system) #10