Problem

No Hipchat clients are able to connect to Hipchat Server. There are several symptoms:

1. Web application fails to load with the following:
   
   Below errors are seen /var/log/hipchat/coral.log:

   2016-11-25T02:11:41.502474+00:00 hipchat coral-1: /#REQmWDwCq =====> New API request: #630, uri=/, method=HEAD, from=127.0.0.1, job_handle=none
   2016-11-25T02:11:41.502988+00:00 hipchat coral-1: /#REQmWDwCq WARN: Sending API error: 404 - <p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
   2016-11-25T02:11:41.503627+00:00 hipchat coral-1: /#REQmWDwCq WARN: =====> API request failed: #630, in-progress=0, code=404
   2016-11-25T02:11:41.504933+00:00 hipchat coral-0: /#REQnC1YlL =====> New API request: #632, uri=/, method=HEAD, from=127.0.0.1, job_handle=none
   2016-11-25T02:11:41.505821+00:00 hipchat coral-0: /#REQnC1YlL WARN: Sending API error: 404 - <p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
   2016-11-25T02:11:41.506482+00:00 hipchat coral-0: /#REQnC1YlL WARN: =====> API request failed: #632, in-progress=0, code=404
   

2. Mac OSX Client fails to connect to the server with the following message:
   
   From the Mac OSX client logs, these errors are seen:

   2016-11-24 17:33:24:941-0800 [1_1@test.hipchat.com] [AccountView] onAuthenticating() called
   2016-11-24 17:33:24:941-0800 [1_1@test.hipchat.com] [AccountView] Destroying chat view controller
   2016-11-24 17:33:25:007-0800 [1_1@test.hipchat.com] [Auth] Access token acquired:
   [expires_in: 3599, access_token: Ff..., group_name: hipchat, token_type: bearer, refresh_token: MeQ1..., group_id: 1, scope: admin_room manage_rooms send_message send_notification view_group view_messages view_room]
   2016-11-24 17:33:25:023-0800 [1_1@test.hipchat.com] [AccountContext] Current state updated: Authenticated
   2016-11-24 17:33:25:023-0800 [1_1@test.hipchat.com] [AccountView] onAuthenticated() called
   2016-11-24 17:33:25:023-0800 [1_1@test.hipchat.com] [AccountView] Destroying chat view controller
   2016-11-24 17:33:25:023-0800 [1_1@test.hipchat.com] [AccountView] Creating chat view controller
   2016-11-24 17:33:25:024-0800 [SystemState] Starting idle monitor - idle seconds: 300.000000
   2016-11-24 17:33:25:024-0800 [1_1@test.hipchat.com] [AccountContext] Current state updated: Loading
   2016-11-24 17:33:25:173-0800 [1_1@test.hipchat.com] Generated chat html from template in 0.147s
   2016-11-24 17:33:25:199-0800 [1_1@test.hipchat.com] [AccountView] onLoading() called
   2016-11-24 17:33:27:417-0800 [1_1@test.hipchat.com] [Web] Web Core log since 5:33:25 PM (3 messages)
   2016-11-24 17:33:25:784-0800 [1_1@test.hipchat.com] [WebCore] Configuring local logger JS error reporting
   2016-11-24 17:33:25:784-0800 [1_1@test.hipchat.com] [WebCore] Skipping Sentry JS error reporting for BTF host
   2016-11-24 17:33:25:799-0800 [1_1@test.hipchat.com] [WebCore] HC instance ready
   2016-11-24 17:35:25:191-0800 [1_1@test.hipchat.com] [AccountContext] Current state updated: Error
   2016-11-24 17:35:25:191-0800 [1_1@test.hipchat.com] [AccountView] onError() called
   2016-11-24 17:35:25:191-0800 [1_1@test.hipchat.com] [AccountView] Received error: LoadingTimeout
   2016-11-24 17:35:25:192-0800 [1_1@test.hipchat.com] [AccountView] Destroying chat view controller

Diagnosis

• Verify the clients' connection to the server following this Knowledge Base article - Unable to connect Hipchat apps to Hipchat Server and make sure it's not a network connection issue.

  • Verify your Hipchat Server's fully qualified domain name (FQDN) using ping:

    ping hipchat.example.com 

    Where  hipchat.example.com  is the URL of your Hipchat Server

    • If the FQDN is valid, you receive a static IP address response. 
    • If the FQDN is invalid, run the following command to reset it:

    hipchat network --hostname=yourHostName --domainname=yourDomainName

    It's also possible that you may be affected by the following JIRA issue:

    
    

      HCPUB-2302 - Getting issue details... STATUS

  • Verify Inbound Connectivity via telnet. On Windows OS, you have to enable telnet before you can use it. See the instructions for enabling telnet for Windows 8 and Windows Vista and Windows 7.

    Open a command prompt and enter the following command:

     echo | telnet hipchat.example.com portNumber

    Where: 

    • hipchat.example.com  is the URL to your Hipchat Server
    • portNumber is 443 - For Hipchat desktop clients. After 2.0.7, XMPP was disabled by default. For more information check this documentation. 
    • portNumber is 443, 5222, and 5223 - all three should ports should be open to allow connectivity from third-party XMPP Hipchat clients (E.g: Adium).

    If the port is closed,  you'll see a message like “Unable to connect to remote host: Connection refused” or “Could not open connection to the host on port: 5222”. You can also use the static IP address to isolate routing and DNS issues. For example, enter telnet 10.0.1.1 443

• Verify that the server's URL that the clients are connecting to matches the hostname and domain configured in  Server admin > Network.  If the Hipchat Server does not have hostname and domain registered in the DNS server yet,  edit the local client machine's host file and the correct FQDN and IP address for the server. 

• 
  

• Verify the correct server's hostname and domain are configured in the configuration files per the below table if there is a mismatch proceed to the Resolution section:

  The values in this table are using an example FQDN named hipchatiscool.example.com please change these accordingly.

  /etc/localenv.sh: 

  hc_host="hipchatiscool"
  hc_domain="example.com"

  
  /etc/chef/solo.json:

  "fqdn": "hipchatiscool.example.com",
  "web_host": "hipchatiscool.example.com",

  /etc/hosts:

  127.0.0.1       hipchatiscool.example.com hipchatiscool localhost localhost.localdom

  /etc/hostname:

  hipchatiscool.example.com

  /etc/monit.d/host.monitrc:

  check system hipchatiscool.example.com

  hostname:

  hipchatiscool.example.com

  /etc/ovfenv.sh:

  The settings you see here are taken specifically from the vApps settings. If you do use vApps and have the hostname registered on the vApps settings, update the hostname accordingly in there too.

  hc_domain='example.com'
  hc_host='hipchatiscool' 

  /hipchat/config/site.json: 

  "external": {
      "web_host": "hipchatiscool.example.com",
      "chat_host": "chat.btf.hipchat.com",
      "api_host": "hipchatiscool.example.com",
      "muc_host": "conf.btf.hipchat.com",
      "cookie_domain_scope": "hipchatiscool.example.com",
      "bosh_host": "https://hipchatiscool.example.com/http-bind/",
      "xmpp_ports_enabled": false,
      "xmpp_route": "xmpp:hipchatiscool.example.com:5222",
      "xmpp_route_proxy_proto": "xmpp:hipchatiscool.example.com:5522",

Cause

The server URL cannot be resolved correctly by the clients. The clients are trying to connect to e.g. test.hipchat.com however internally it's resolving to a different hostname test2.hipchat.com

Resolution

1. On the Hipchat Server, edit the incorrect server's hostname located in any of the following files:
   
   • /etc/localenv.sh
   • /etc/chef/solo.json
   • /etc/hosts
   • /hipchat/config/site.json
   • /etc/monit.d/host.monitrc
   • /etc/ovfenv.sh
   • /etc/hostname

   • hostname

     The sed command can be used to replace the old values with the new values. Be sure to substitute the correct values for incorrect_address, correct_address, and /file_path in the command below:

     sudo dont-blame-hipchat -c "sed -i 's/incorrect_address/correct_address/g' file_path"

2. Run a complete service restart of Hipchat Server and the clients should be able to connect to the server successfully:

   hipchat upgrade --restart 

The publicly available bug is tracked here:  HCPUB-2302 - Getting issue details... STATUS .