Hipchat Support

Purpose

This article explains how to use the Hipchat Data Center migration tool. This tool exports data from Hipchat Server version 2.2.3 or later, and imports it into Hipchat Data Center.

This tool is separate from, and incompatible with current export feature available in Hipchat Server. Data exported using the hipchat export CLI cannot be migrated using this tool.

What is migrated

Migrated	Not migrated
Users Rooms room message history room file attachments private message file attachments custom emoticons autojoins - the rooms and people open in the client's left sidebar 'General' and 'Notifications' client preferences	'Appearance' client preferences chatView - include or exclude user avatars in chat density - how close together content is rendered nameDisplay - display users by name or @mention name theme - normal or dark color scheme user passwords API tokens installed integrations/addons/apps

Process

Check your Hipchat Server version

This tool exports data from Hipchat Server version 2.2.3 or later only. You may need to upgrade your Hipchat Server instance to proceed.

Check your Hipchat Server utilization

Before you start the migration process, you should make sure you know how many files, and how many messages you have in your existing Hipchat Server instance. The migration tool requires enough memory on the Hipchat Server instance that you're running it on to process thousands of messages and gigabytes of files.

In general:

For every 1000 messages, the migration tool will consume an additional 2.6 MB of memory.
For every 1GB of files, the migration tool will consume an additional 500 KB of memory.

Once you know your file store size and how many messages you have, you can calculate the additional memory requirements. You might need to provision your Hipchat Server instance with more memory to complete the migration tool run. Once the tool has run, you can reduce the instance's memory back to the original amount, if needed.

(Note that this tool runs an all-or-nothing export. If you continue to use your Hipchat Server instance after running the export phase of the migration, you will naturally have some chat history that is not included in the export.)

How do I find out how large my file store is? (Click for instructions)

To get an accurate file store size, connect to the Hipchat Server deployment and run a disk utilization command.

Connect to the Hipchat Server deployment (with the admin account), using your terminal or shell.
Check the disk utilization on the file store path by running the following command:
```
du -h -d 1 /file_store/local/
```

The command runs, and outputs the size of the /file_store/local/ directory.

How do I find out how many messages I have? (Click for instructions)

Hipchat Server tracks how many messages have been sent on your deployment on the Status page in the Server admin section. However, retention settings and message deletion can make this number inaccurate.

To get an accurate message count, connect to the Hipchat Server deployment and run a script.

Connect to the Hipchat Server deployment (with the admin account), using your terminal or shell.

Copy the script below to the server.
You can either create the file locally as message_counter.sh and then use scp to copy the file to the Hipchat Server machine, or you can create a new file directly on the Hipchat Server machine and then paste the script contents into the file.

message_counter.sh

#!/bin/bash
# This script counts the message entries directly from the database
# 06-21-17
# Must run inside Hipchat Server command line interface

counts=$(curl 'localhost:9200/_cat/indices?v' | awk 'FNR > 1 {print $6}')

total=0

for number in $counts
do
   total=$(($total+$number))
done

echo "Total messages: $total"

Once you have the script on the server, make it executable using the following command:
chmod +x message_counter.sh
Finally, execute (run) the script using the following command:
./message_counter.sh

The script runs and returns the number of messages in your deployment.

Install Hipchat Data Center

Deploy Hipchat Data Center with at least one node.
Use the Setup Wizard to configure the FQDN, License, etc.
Make sure no users will be using the system during the import process.
Configure the email server on the destination server by going to System > Email server in the admin UI.
(Unless your organization uses an external authentication source, all users will receive a password reset after migration, so email is important!)

Install the migration utility

Download the migration utility to your local computer: https://s3.amazonaws.com/hipchat-server-stable/utils/migrate/hc-migrate
v3.3.2, last updated May 24th 2019. SHA512 sum:
```
you may need to scroll -->
0f37ecb30774ac903f46bde055d24b2b6c320dc263b0928f46187ea93e5ae2f0d72df9efb3569a1f8f7837fb59b0080500359a4edfc2a631d0cbccc7d6fb3b90
```

Copy the migration utility to the source Hipchat Server.

scp -i /path/to/sshkey.pem hc-migrate admin@<hcserver.example.com>:/home/admin/hc-migrate

Copy the migration utility to a node on the destination Hipchat Data Center instance.
```
scp -i /path/to/sshkey.pem hc-migrate admin@<hcdcnode.example.com>:/home/admin/hc-migrate
```
Depending on your environment, you may need to copy the migration utility to your data center's jumpbox (or management portal) then copy it to the node.

Export from Hipchat Server

Reminder! This tool exports data from Hipchat Server version 2.2.3 or later only. You may need to upgrade to a compatible version of Hipchat Server to proceed.

Connect to the Hipchat Server command-line interface using a terminal or SSH
```
ssh -i /path/to/sshkey.pem admin@<hcserver.example.com>
```
Escalate your user permissions to root
```
sudo dont-blame-hipchat
```
Switch to the directory containing the migration utility
```
cd /home/admin/
```

Update the permissions for the migration utility so it can be run, verify the checksum and the version.

chmod +x /home/admin/hc-migrate
sha512sum hc-migrate
/home/admin/hc-migrate -version
  Hipchat Migration Utility version 3.1.4 (2018-02-01)

Retrieve the password to the source Hipchat Server MariaDB database. You'll use this (without the quotation marks in the next step.
```
cat /hipchat/config/site.json | jq -r .databases.hipchat.pass
```

Copy, edit, and run the following example command to start the export.

By default, the export artifact is saved in the directory from which you run the command. You can use the the --export flag to specify another location.

/home/admin/hc-migrate --export "exportFileLocation.tar.gz.aes" --aes_password "passwordToExportFile" \
--db_password "databasePassword" --verbose |& tee desiredLogFile

A sample command would look something like this:

/home/admin/hc-migrate --export "/home/admin/20171115_migrate.tar.gz.aes" --aes_password "hipchat" --db_password "reallysecurepassword" --verbose |& tee /home/admin/newmigrate_export.log

When the export is complete the logs will stop in the terminal and the command will exit. The end of the log output for a successful export looks like this:

DEBUG 2017/11/15 01:59:04 Writing rooms/711/history.json
DEBUG 2017/11/15 01:59:04 Writing rooms/712/history.json
DEBUG 2017/11/15 01:59:04 Writing rooms/713/history.json
DEBUG 2017/11/15 01:59:04 Writing rooms/714/history.json
DEBUG 2017/11/15 01:59:04 Writing rooms/715/history.json
DEBUG 2017/11/15 01:59:04  ======== *** Export completed successfully! *** ========

Copy the resulting export artifact to a workstation or other system with access to the Hipchat Data Center node

Import into Hipchat Data Center

Copy the export artifact to a Hipchat Data Center node
Connect to the Hipchat Data Center node command-line interface using a terminal or SSH
```
ssh -i /path/to/sshkey.pem admin@<hcdcnode.example.com>
```
Escalate your user permissions to root
```
sudo dont-blame-hipchat
```
Switch to the directory containing the migration utility
```
cd /home/admin/
```

Update the permissions for the migration utility so it can be run, verify the checksum and the version.

chmod +x /home/admin/hc-migrate
sha512sum hc-migrate
/home/admin/hc-migrate -version
  Hipchat Migration Utility version 3.1.4 (2018-02-01)

You'll need your Redis and Postgres connection details for the next steps. If you don't already have them, run the following commands get them.

Postgres connection details

jq -r '.databases.hipchat_postgres' /hipchat/config/site.json

Redis connection details

pwd=$(jq -r '.databases.hipchat_postgres.pass' /hipchat/config/site.json) 
usr=$(jq -r '.databases.hipchat_postgres.user' /hipchat/config/site.json) 
host=$(jq -r '.databases.hipchat_postgres.servers[0]' /hipchat/config/site.json | cut -d: -f1) 
schema=$(jq -r '.databases.hipchat_postgres.schema' /hipchat/config/site.json) 
PGPASSWORD=$pwd psql -h $host -U $usr -d $schema -t -c "SELECT * FROM configurations WHERE key LIKE 'redis%';"

Run the command below to clear the Redis cache.

This will remove all key:value pairs from the Redis environment specified, regardless of whether or not they are associated with Hipchat.

redis-cli -h <RedisHost> -a <Password> flushdb

Sample Output

root@ip-10-10-10-12:/home/admin# jq -r '.databases.hipchat_postgres' /hipchat/config/site.json
{
  "user": "hcdc",
  "servers": [
    "10.10.10.13:5432"
  ],
  "schema": "hipchat_postgres",
  "pass": "super-secret-pw"
}
root@ip-10-10-10-12:/home/admin# pwd=$(jq -r '.databases.hipchat_postgres.pass' /hipchat/config/site.json)
root@ip-10-10-10-12:/home/admin# usr=$(jq -r '.databases.hipchat_postgres.user' /hipchat/config/site.json)
root@ip-10-10-10-12:/home/admin# host=$(jq -r '.databases.hipchat_postgres.servers[0]' /hipchat/config/site.json | cut -d: -f1)
root@ip-10-10-10-12:/home/admin# schema=$(jq -r '.databases.hipchat_postgres.schema' /hipchat/config/site.json)
root@ip-10-10-10-12:/home/admin# PGPASSWORD=$pwd psql -h $host -U $usr -d $schema -t -c "SELECT * FROM configurations WHERE key LIKE 'redis%';"
 redishostname | 10.10.10.13
 redisport     | 6379
 redispass     | supersecretredispassword

root@ip-10-10-10-12:/home/admin# redis-cli -h 10.10.10.13 -a supersecretredispassword flushdb
OK

Copy, edit, and run the example command below to start the import process.

Running the import with the --forceclean flag will delete all existing user accounts in the database (including the owner account created during the setup wizard).

/home/admin/hc-migrate --import "exportFileLocation.tar.gz.aes" --aes_password "passwordToExportFile" \
--postgres_url "dbname=hipchat_postgres host=postgresIpAddressorDomain user=postgresUsername password=postgresPassword port=postgresPort sslmode=disable" \
--redis_url "redisIpOrHostname:RedisPort" --redis_password "redisPassword" \
--forceclean --verbose |& tee <desiredLogFile>

A sample command would look something like this:

/home/admin/hc-migrate --import "hcs_export_05-12-2017.tar.gz.aes" --aes_password "asdf9324lkJasdf" \
--postgres_url "dbname=hipchat_postgres host=10.10.10.7 user=hcdc password=supersecretpostgres port=5432 sslmode=disable" \
--redis_url "10.10.10.6" --redis_password "supersecretredis" \
--forceclean --verbose |& tee <desiredLogFile>

When the import finishes, the terminal window stops displaying new log updates, and the command exits. Look for the fancy ASCII art at the end of the log output which indicates a successful export:

DEBUG 2017/11/15 02:41:19 Counted 3 messages in muc-2017.11
DEBUG 2017/11/15 02:41:19 == Extracting and processing rooms/714/history.json
DEBUG 2017/11/15 02:41:19 Counted 3 messages in muc-2017.11
DEBUG 2017/11/15 02:41:19 == Extracting and processing rooms/715/history.json
DEBUG 2017/11/15 02:41:19 Counted 3 messages in muc-2017.11
DEBUG 2017/11/15 02:41:19 total message count 16688
DEBUG 2017/11/15 02:41:19 Done processing the source
DEBUG 2017/11/16 21:38:27  ======== *** Import completed successfully! *** ========

Run the command below to restart Hipchat. Run this command on each node in the HCDC cluster.
```
hup
```
Request a password reset for each user that will be logging in to HCDC using the internal authentication system at https://example.hcdc.com/forgot_password
(If your deployment uses an external user directory, such as LDAP or SAML, you don't need to do this.)
Complete any necessary set up for Hipchat Data Center and start chatting.

Performance profiling for the migration tools

If your migration job is not completing successfully, or is having other issues, you might want to collect performance data on your Hipchat deployment to better understand what's going on. Before you begin either the the export or the import process, you can open a new terminal window with a new SSH session so you can monitor and log performance data while other operations are taking place.

Open a new terminal or SSH session into the deployment you'll be working on.
Collect a baseline of system utilization (wait ~1 minute for this to finish).
Replace "impbase" in these commands with a filename that indicates to you if the file is from an import or an export process.
```
date > /home/admin/impbase_vmstat.log && vmstat -w 3 20 | tee -a /home/admin/impbase_vmstat.log
```
Start collecting system utilization (this will run until cancelled).
Replace "import" in these commands with a filename that indicates to you if the file is from an import or an export process.
```
date > /home/admin/import_vmstat.log && vmstat -w 3 | tee -a /home/admin/import_vmstat.log
```
Run the export or import task in the other terminal or SSH session. When it completes, return to the profiling session.
Press Ctrl + C to cancel the vmstat run.
Append a stop stamp:
```
date >> /home/admin/import_vmstat.log
```

Expected Errors


"AES password incorrect, cannot decrypt archive. Please try again"	This most frequently occurs when the incorrect password is used to import an export artifact This may also occur if the actual export artifact was corrupted, but this is far less likely
"Cannot begin import, destination database contains user data. Use --forceclean to overwrite"	This error will occur during any import attempt without using the `--forceclean` option because the Hipchat Data Center Setup Wizard creates one user and one room by default If in doubt about cleaning the Postgres database during import, please check your database to verify its contents prior to running the command

Migration tool flag reference

Global

Flag

What does it do

Default

--version,
-v

Output the current version of the migrate tool

false

--verbose;

-j

Enable verbose logging

false

--help

-h

Shows help message

false

Import

Flag	What does it do	Default
--import -i	The location to the exported artifact that will be imported	none
--aes_password	The password to use to decrypt the export artifact	none
--forceclean	Clean tables before starting the import. Ensures the database is in a good state for importing	false
--hcs	Import to HCS, default is to HCDC	false

Export

Flag	What does it do	Default
--export -e	The location of where to save the exported artifact	none
--aes_password	The password to use to encrypt the export artifact	none
--hcdc	Export from HCDC, default is from HCS	false
--start_date	The earliest date to fetch history in the RFC3339 format If set, messages before this date will be skipped.	empty
--end_date	The latest date to fetch history in the RFC3339 format If set, messages after this date will be skipped.	empty
--skip_attachments	Skip attachments in the export	false

RFC339 format yyyy-mm-ddThh:mm:ss+00:00, for example: 2018-05-01T00:00:00+00:00

Data Access

Flag	What does it do	Default
--elasticsearch	The HCS Elasticsearch endpoint for export	http://localhost:9200/
--db_user	The HCS MariaDB/MySQL DB username	root
--db_password	The HCS MariaDB/MySQL DB password	empty
--postgres_url	The HCDC Postgres database to connect to	postgres://localhost
--redis_url	Redis instance to connect to	localhost:6379
--redis_password	Redis instance password	empty
--redis_stats_url	Redis Stats instance to connect to	localhost:6380
--redis_stats_password	Redis Stats instance password	empty

Hipchat Support

Knowledge base

Products

Jira Software

Jira Service Management

Jira Work Management

Confluence

Bitbucket

Resources

Documentation

Community

Suggestions and bugs

Marketplace

Billing and licensing

How to migrate from Hipchat Server to Hipchat Data Center

Still need help?

Purpose

What is migrated

Process

Check your Hipchat Server version

Check your Hipchat Server utilization

Install Hipchat Data Center

Install the migration utility

Export from Hipchat Server

Import into Hipchat Data Center

Performance profiling for the migration tools

Expected Errors

Migration tool flag reference

Global

Import

Export

Data Access

In this section

Page

Viewport

Confluence

How to migrate from Hipchat Server to Hipchat Data Center

Related content

Still need help?

Purpose

What is migrated

Process

Check your Hipchat Server version

Check your Hipchat Server utilization

Install Hipchat Data Center

Install the migration utility

Export from Hipchat Server

Import into Hipchat Data Center

Performance profiling for the migration tools

Expected Errors

Migration tool flag reference

Global

Import

Export

Data Access

In this section

Related content