Hipchat Server SSL Installation and Troubleshooting Guide

Still need help?

The Atlassian Community is here for you.

Ask the community

This Knowledge Base article was written specifically for the Atlassian Server platform. Due to the Functional differences in Atlassian Cloud, the contents of this article cannot be applied to Atlassian Cloud applications.

This guide is intended for use with Hipchat Server and explains SSL common configuration issues and problems. Most SSL certificate issues for Hipchat Server are caused by:

  • Root CA certificate included in the certificate chain (which is self signed most of the time).
  • Wrong or missing intermediate certificates in the certificate chain.
  • Self-signed certificate(s) located in certificate chain.
  • Badly formed .PEM files.

This page

To check your installed SSL certificate for errors, run this OpenSSL command from the Hipchat Server console:

echo | openssl s_client -showcerts -connect $(hostname):443 -CApath /etc/ssl/ && echo | openssl s_client -connect $(hostname):443 2>/dev/null | openssl x509 -noout -dates -text

This command checks the SSL certificate and chain(s) installed on Hipchat Server, much like a client would request SSL information to start a secure communication.

It details the public certificate chain and any errors that may cause problems. You can run this command either on the server's terminal / SSH console itself or externally:

  • Internal: Use the command provided above
  • Externally: Change the $(hostname) section to the hostname and domain of the server that you are running the check on. For example, hipchat.example.com

When using the comprehensive OpenSSL check, a correctly installed SSL certificate looks like this for example:

HipChat SSL Example
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
verify return:1
depth=0 C = US, ST = CA, L = San Francisco, O = "Example Inc.", CN = *.example.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=CA/L=San Francisco/O=Example, Inc./CN=*.example.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=CA/L=San Francisco/O=Example, Inc./CN=*.example.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
---
No client certificate CA names sent
---
SSL handshake has read 3277 bytes and written 421 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 9DB8577FB02B4CAD515ECE59E22194DC8F1D5BD7F19D6A016B15A445A2F2919F
    Session-ID-ctx:
    Master-Key: AAFA57013D6A42D45F6D36F36552887464832281FB409003DCCAE94A1235E46CCBA390827E10D41CBDA38A11C4C9BE42
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - b3 e6 08 37 ae f5 fa 66-e4 ac c6 a9 0d 97 d2 b8   ...7...f........
    0010 - 22 63 24 40 7b ea 1d 1f-9c ea cd 02 a6 dc 2f 22   "c$@{........./"
    0020 - 91 8c 9d 55 98 57 af 14-7b 61 60 93 d1 8c 44 14   ...U.W..{a`...D.
    0030 - 6f e1 9a e2 14 b0 d7 af-76 ba ad 70 d3 78 66 96   o.......v..p.xf.
    0040 - af 9d e2 c4 7d 8e 4f df-a1 88 48 38 7e 28 ca d2   ....}.O...H8~(..
    0050 - aa 37 61 1b 69 cf a0 ea-5e e1 0e 63 2d 79 ce a2   .7a.i...^..c-y..
    0060 - 9f d2 ea f6 8e bf b4 fe-60 15 9a aa 03 7e 4b 7d   ........`....~K}
    0070 - 87 55 92 27 60 b0 1d e9-fa ab fe 32 25 82 e8 be   .U.'`......2%...
    0080 - 6d 5a e4 db 60 c7 ae 9d-67 c3 96 b8 94 83 28 9c   mZ..`...g.....(.
    0090 - 8d a0 47 40 a0 88 12 ab-37 46 bf 70 4d 05 d1 0e   ..G@....7F.pM...
    00a0 - aa 76 e8 da ad 0c b1 61-56 c9 12 d2 0b 91 d7 11   .v.....aV.......

    Start Time: 1510550546
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
DONE
notBefore=Aug 31 00:00:00 2016 GMT
notAfter=Sep  5 12:00:00 2018 GMT
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:d8:87:50:83:2d:b7:58:48:50:72:7b:81:92:24:3c
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert SHA2 High Assurance Server CA
        Validity
            Not Before: Aug 31 00:00:00 2016 GMT
            Not After : Sep  5 12:00:00 2018 GMT
        Subject: C=US, ST=CA, L=San Francisco, O=Example, Inc., CN=*.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a6:04:43:0f:6a:27:eb:19:40:14:65:b1:2f:91:
                    d7:64:14:90:f3:75:f1:8a:78:59:b6:90:31:6a:ea:
                    ce:3b:fa:f4:a8:d9:74:af:01:41:6f:68:c8:8a:59:
                    38:1e:39:5f:0b:73:7c:51:90:81:52:b4:d2:8e:8f:
                    1f:89:c7:81:14:86:9c:77:4b:98:75:a1:e8:bf:21:
                    b3:95:60:e9:56:d9:9a:62:94:d1:35:e0:e1:f8:d4:
                    4a:8d:5f:61:4e:01:3d:94:23:74:64:a0:f8:db:de:
                    6f:75:c3:2d:54:15:f1:c6:ba:aa:2d:9d:6b:40:d9:
                    18:64:3c:96:a4:77:27:c5:df:c4:17:59:b6:18:95:
                    28:72:1d:64:45:9a:5f:f1:13:10:69:79:3a:be:8d:
                    2b:bf:72:b0:08:5c:b6:13:a3:2d:e5:52:be:6d:7e:
                    20:95:40:92:6e:bd:32:f4:0a:ca:09:5c:e1:3d:4a:
                    2b:2b:38:14:2c:8a:c1:a4:44:fa:cd:0d:d3:9b:64:
                    2f:95:6d:67:ae:25:5b:64:4c:5e:3c:50:c7:d7:0b:
                    42:bd:d5:06:97:32:12:b6:f5:66:af:62:d9:cd:db:
                    46:9b:42:29:09:2b:d6:f5:12:be:86:97:73:73:da:
                    02:b1:08:53:59:a3:0e:4f:74:7b:52:1a:d6:76:ab:
                    7d:71
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:51:68:FF:90:AF:02:07:75:3C:CC:D9:65:64:62:A2:12:B8:59:72:3B

            X509v3 Subject Key Identifier:
                03:E6:D2:4D:86:76:9B:5F:88:AD:C8:BE:39:BC:5B:8A:98:D9:85:55
            X509v3 Subject Alternative Name:
                DNS:*.example.com, DNS:example.com
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl3.digicert.com/sha2-ha-server-g5.crl

                Full Name:
                  URI:http://crl4.digicert.com/sha2-ha-server-g5.crl

            X509v3 Certificate Policies:
                Policy: 2.16.840.1.114412.1.1
                  CPS: https://www.digicert.com/CPS
                Policy: 2.23.140.1.2.2

            Authority Information Access:
                OCSP - URI:http://ocsp.digicert.com
                CA Issuers - URI:http://cacerts.digicert.com/DigiCertSHA2HighAssuranceServerCA.crt

            X509v3 Basic Constraints: critical
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         24:11:15:ca:a4:89:e7:dc:1b:2b:be:6b:c3:b4:8e:6b:60:aa:
         86:26:9d:3b:d0:44:43:21:bc:2e:71:ad:08:c0:aa:b8:30:fd:
         aa:99:0d:4e:34:d5:14:d4:f6:ca:6d:fe:f9:69:48:3b:df:da:
         66:22:74:a5:16:3f:53:e7:00:23:d1:17:ad:7b:ad:30:e8:ae:
         04:ef:99:3d:03:40:cd:ac:14:04:22:ee:11:86:14:f7:57:23:
         e2:14:07:83:37:23:42:07:04:2e:8d:dc:3e:fa:0d:83:f8:4c:
         47:8f:f6:d4:fc:9c:28:cb:00:7c:fa:91:f2:88:64:6a:c2:73:
         0f:96:69:e5:5c:7e:58:3f:6a:25:35:56:36:66:d5:92:5f:9f:
         86:c9:04:7a:ef:05:64:c1:60:c5:a8:a5:13:bd:fa:a6:af:73:
         41:e2:13:38:cf:d5:f6:8f:a7:9d:26:2a:83:23:f4:b3:6f:08:
         d3:25:72:3a:3e:5a:76:c5:b7:2b:46:f8:28:1e:05:25:8d:8e:
         6c:a0:ff:9a:e0:38:cb:e4:93:2a:d9:2a:62:70:f2:b0:72:ff:
         0e:8d:da:14:2b:d0:93:98:a2:ee:b9:d6:d8:36:63:1d:2b:47:
         bd:3b:bb:6c:62:25:35:ce:c2:a1:29:97:f7:5b:9a:ad:d4:96:
         4d:3d:36:93

In this example, there are two certificates listed in the certificate chain. Usually, certificate 0 is the primary certificate and can be easily identified by the CN which should list the fully qualified domain name (FQDN). Certificate 1 is the intermediate certificate. 

Root Cert

If a certificate is installed that has the same subject and issuer line (also called a self-signed certificate), then, more than likely, that is a root certificate and needs to be removed from the chain (because the root certificates are normally housed in the client trust-store).

Common OpenSSL Error Codes

Error Code  Description
verify error:num=27:certificate not trusted This normally comes together with the errors num=21 where the server doesn't have the intermediate certificates needed to verify the SSL certificate.
verify error:num=26:unsupported certificate purpose This means that the certificate being installed isn't a Server certificate and cannot be used for Server SSL handshaking.
verify error:num=21:unable to verify the first certificate This can mean that either some or all the intermediate certificates required to establish the chain of trust between the primary certificate (the one you buy from a CA vendor) and the Root certificate (housed and served by the client) is missing. See below for troubleshooting steps.

verify error:num=20:unable to get local issuer certificate

The -CApath is not specified or not found. Verify that the -CApath is pointed to the local certificate store. For Hipchat Server that is /etc/ssl/ but may be different depending on what console/terminal you are using to query the Server. See below for troubleshooting steps.
Verify return code=19:self signed certificate in certificate chain One or more of the certificates in the chain has a self-signed certificate. Root CA certificates are almost always self-signed. See below for troubleshooting steps.
Verify return code=18:self signed certificate  The default self signed certificate that comes together during the installation of Hipchat Server / generated through hipchat certificates --selfsign command is still being used in the instance. See below for troubleshooting steps.


Error - Unsupported Certificate Purpose

When running an OpenSSL check against a CSR or a CRT file, you may see this error:

verify error:num=26:unsupported certificate purpose

Troubleshooting

  1. Run the following command inside Hipchat Server command line interface:

    Check against installed SSL certificate
    openssl s_client -showcerts -connect localhost:443 </dev/null 2>/dev/null|openssl x509 -text |grep v "$(grep -E -A1 "Key Usage")"

    or run a check against your .crt file:

    Check against a .crt file
    openssl x509 -text -noout -in <filename for crt>.crt
  2. In the output under X509v3 Extended Key Usage should be the following line: TLS Web Server Authentication
  3. If this line is missing, then the certificate isn't specifically setup for Server handshaking. You will need to reach out to your CA and ask them to generate a Server certificate.
  4. If the command returns grep: : No such file or directory then you do not have the certificate installed (are you using a self-signed certificate?)

Cause

This error specifically states that the intended purpose of the certificate installed does not include the role to act as a Server certificate thus cannot be used for handshaking when a client attempts to connect.

Error - Unable to verify the first certificate

When running OpenSSL check, an installed SSL cert from trusted CA vendor shows the following error:

verify error:num=21:unable to verify the first certificate

Troubleshooting

  1. Verify that you have an intermediate certificate installed.
  2. If you have an intermediate certificate installed, it's possible that you may need more than one to establish trust. This is common with vendors such as Entrust. Check with your vendor regarding how many intermediate certificates you need.
  3. You can use the GeoCerts SSL Checker tool to visually see your chain and check if any intermediate certs are missing.
  4. If you've recently installed a chain with an intermediate certificate, first snapshot or back up your instance. (This will disconnect any user that's currently connected to the service.) Then restart the services (HipChat service -restart) or rebooting the Server (sudo reboot). 

Cause

The primary certificate (the one you purchase from a vendor) cannot establish the chain of trust between itself and the Root certificate stored locally. Usually installing the intermediate certificates fixes this.

Error - Unable to get local issuer certificate

When running OpenSSL check, an installed SSL cert from trusted CA vendor shows:

verify error:num=20:unable to get local issuer certificate

Troubleshooting

  1. Verify that you are using the -CApath argument when running the OpenSSL check.
  2. Verify that the path to the local certificate storage is correct (Hipchat Server local certificate trust-store is /etc/ssl).

Cause

This error code is thrown when a local certificate path is not found during the OpenSSL check is used. This error is specific to the OpenSSL check. Generally, it can be ignored and doesn't affect the trust of a certificate chain.

Error - Self-signed certificate in certificate chain

When running OpenSSL check, an installed SSL cert from trusted CA vendor shows:

Verify return code=19:self signed certificate in certificate chain

Troubleshooting

  1. Install a trusted CA signed primary SSL certificate and an intermediate certificate.
  2. Verify that the Root CA is not included in the certificate chain (as this is almost always self-signed).
  3. If you've removed the Root CA cert, first snapshot or back up your instance. (This will disconnect any user that's currently connected to the service.) Then restart the services (HipChat service -restart) or rebooting the Server (sudo reboot). 

Cause

This error is thrown whenever there is a self-signed certificate in the chain. When Hipchat Server is booted for the first time, it generates a self-signed certificate for testing purposes. We highly recommend replacing this certificate with a certificate chain from a trusted vendor.

Error - The certificate and key do not match

When uploading your SSL private key and certificate through the Hipchat Server administrative interface or command-line interface, the following error occurs:

Unable to update: The certificate and key do not match

Troubleshooting

Use the following commands to generate an MD5 sum for the private key, certificate signing request (CSR), and certificate files:

openssl rsa -noout -modulus -in domain.key | openssl md5
openssl x509 -noout -modulus -in domain.crt | openssl md5
openssl req -noout -modulus -in domain.csr | openssl md5

Verify that the output string from each command is identical to verify that the files are matching. If the outputs are identical but problem persists, try to re-install the SSL certificate you get from CA as cleanly as possible. Start with reconfiguring the server to use self-signed certificate and then re-import the trusted SSL certificate into the server again:

hipchat certificates --selfsign
cat hipchat-example-com.crt > new.pem
cat hipchat-example-com.key >> new.pem
hipchat certificates --import new.pem

If problem persists even after cleanly importing the certificate, there could be a problem with the certificate that the CA generated. Reach out to the CA to re-issue the SSL certificate followed by a re-installation of the new certificate on the server. On a side note, the wrong order of your SSL certificate could also produced similar error. Double check the SSL certificate format following the recommended format here: Creating or Obtaining an SSL Key and Certificate

Cause

The private key and certificate being uploaded do not match and are incompatible with each other. It will be necessary to re-form the private key, CSR, and/or certificate in such a way that they do match, at which point they can be uploaded to the Hipchat Server.

Error - self signed certificate

Consider purchasing a trusted CA signed SSL certificate to tackle this issue permanently.

SSL Certificate Chain Ideal Setup

As listed in the Creating or Obtaining an SSL Key and Certificate, when installing the SSL certificate chain in Hipchat Server, please make sure that your .pem file includes the RSA private key, the primary certificate and an intermediate certificate. This gives the certificate chain the best chance of minimizing problems with trust in Hipchat Server.

ECC Certificates

At this time, Hipchat Server is not compatible with ECC (Elliptic Curve Cryptography) or ECDH (Elliptic Curve Diffie–Hellman) generated SSL certificates.

Trust Issues

The chain of trust extends from the root certificate (also called the anchor certificate) to the intermediate certificate then to the primary certificate. If there are any errors or issues with trust throughout the chain, the following issues may happen:

Self-Signed Certificates

Hipchat Server generates a self-signed certificate and private key when it is first deployed. This certificate is meant for testing purposes and is not recommended that you use it in a production environment.

To ensure that the functionalities of Hipchat Server are not affected in any way, we always recommend you to obtain a trusted CA signed full certificate chain applied on your instance.


Internally Signed SSL Certificates

If your group is using an SSL certificate signed by an internal CA (opposed to a vendor like DigiCert), the internal CA will need to provide the corresponding RSA private key, primary, and intermediate certs to install in Hipchat Server. Once the SSL chain is installed in Hipchat Server, install the root certificate into the trust-store of the OS running the Hipchat client. This adds the certificate as 'trusted' which is needed when connecting to the Hipchat Server certificate chain. If the root certificate is not installed in the OS running the Hipchat client, then the trust will not be established and you may have problems connecting the client.

Last modified on Nov 8, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.