Problem

Attempting to sync Hipchat Server with an external directory fails with the following error in the /var/log/hipchat/atlassian-crowd.log:

2017-10-24 17:23:28,921 scheduler_Worker-9 WARN [hipchat.sync.crowd.IncrementalCrowdSync] There were errors connecting to remote services: [com.atlassian.hipchat.sync.issue.SyncIssues$ImmutableSyncIssue@47349db0]
2017-10-24 17:23:28,930 scheduler_Worker-9 INFO [hipchat.sync.crowd.CrowdSyncManager] Sync completed with the following issues:
2017-10-24 17:23:28,930 scheduler_Worker-9 ERROR [hipchat.sync.crowd.CrowdSyncManager] Could not connect to Coral (HipChat API) at: http://127.0.0.1:8080/v2\\ 2017-10-24 17:23:28,930 scheduler_Worker-9 ERROR [hipchat.sync.crowd.CrowdSyncManager] The cause of this issue is:
Unexpected response '400' with message 'Bad Request'

Diagnosis

Environment

• Hipchat Server 2.2.6 (this can also affect older releases)
• Hipchat Server is connected to an external directory for user management. 

Diagnostic Steps

• Coral and all other chat services are up and running.  There are no issues with chat functionality.  Only the directory sync seems to fail. 

• The /var/log/hipchat.coral.err.log contains many instances of the following errors:

  2018-01-10T20:23:27.236370-08:00 chat coral-2: [/sync/users#REQLas58j] ERROR: Not able to hydrate 851 from db
  
  2018-01-10T20:36:20.273207-08:00 chat coral-2: [/user#REQOr4Wrl] WARN: Sending API error: 400 - Incorrect JSON value: Expecting value: line 1 column 1 (char 0)

• The "851" in the above example refers to a user id within the Hipchat Database's users table. 

Cause

• The "metadata" value for the user id in question was invalid.  
• This was due to a server administrator making changes directly in the Hipchat database to a user account. 

• The metadata value was set to parenthesis instead of curly brackets.  Here is an example of what the account looked like in the database.  Note the metadata value:

  select * from users where id=851\G
  
  
  *************************** 850. row ***************************
            id: 851
       version: WTJJ0WIA
   external_id: {hcs}joe.user
          name: User, Joe 
      nickname: NULL
  mention_name: JoeUser
         email: joe.user@example.com
      password: p
         title: 
     photo_url: NULL
     confirmed: 0
       created: 2016-07-21 23:57:19
    is_deleted: 0
    is_enabled: 1
      is_guest: 0
      timezone: UTC
      metadata: ()

Resolution

Fix the metadata value to be curly braces instead of parenthesis, then attempt a full sync of the user directory following the steps below:

1. Take a snapshot of the Hipchat Serever virtual machine before making any changes to the databse. 

2. Log in to the Hipchat command line and access the database:

   DBPASS=$(awk '/"pass"/ {gsub(/[",]/,"");print $2}' /hipchat/config/site.json) && mysql -uroot -p$DBPASS hipchat

3. Correct the metadata value for the user id in question.  In this example, we changed it for user id 851.  Please substitute the user id accordingly for your situation:

   update users set metadata='{}' where id=851;

4. Log in to the Hipchat Server's Web User Inteface as an admin user.
5. Navigate to Group Admin → Authentication.
6. Click on the "Configure" button for the External Directory.
7. Locate the directory.
8. Under the Options column for the directory, click "Disable"
9. When the link changes to "Enable", click "Enable" 
10. Steps 8 and 9 will initiate a full synchronization with the external directory. 
11. Wait a couple of minutes then check the /var/log/hipchat/atlassian-crowd.log to ensure the synchronization completes without any errors. 
12. Also, if no new users were being synced into Hipchat because of the error, please check Group Admin → Users and ensure that the accounts are now in Hipchat.