How to validate Hipchat Server/Data Center imports and exports
The following functions are built into the
data branch of the Toolkit for use during a migration.
Each command's output can be captured as a log file through redirection. For example:
./hipchat_toolkit.run data import_log_stats -l /var/log/hipchat/hcapp.log > import_stats.out
Outputs statistics from a Hipchat Server/DC export process.
Use this function after exporting Hipchat data. Please note that HCS/HCDC has an automatic log rotation mechanism, so the correct log file may be zipped in the /var/log/hipchat (HCS) or /file_store/shared/logs (HCDC) directory.
$ ./hipchat_toolkit.run data export_log_stats Usage: -l <full location of hcapp.log> : log file which contains the export messages (must contain entry "Starting export of" and following)
$ ./hipchat_toolkit.run data export_log_stats -l /var/log/hipchat/hcapp.log ... [2017-10-24T14:35:18] [INFO] - Parsing logs for export with ID: 249ddddb-3124-4b42-9ca8-51f79b035e28 Start timestamp: 2017-10-20T21:03:34.341134+00:00 Finish timestamp: 2017-10-20T21:06:35.224271+00:00 Duration: 3 minutes and 1 seconds Users: 5643 Rooms: 398 No attachment failure detected in log.
Outputs counts of data contained in an export archive file.
Use this function after exporting Hipchat data.
$ ./hipchat_toolkit.run data export_archive_stats Usage: -a <absolute path to export archive file> : .tar.gz.aes file generated by the HipChat export service Optional: -d <absolute path to expansion directory>: the directory where the export archive will be decrypted and decompressed If expansion directory is not specified, /file_store/shared/tmp/export_archive_stats will be used.
$ ./hipchat_toolkit.run data export_archive_stats -a /home/admin/hipchat-2017-10-24_15-38-21.tar.gz.aes [2017-10-24T19:45:00] [INFO] - Counting complete Users: 5643 Rooms: 398 (26 archived) Room messages: 269591 (60 GuestAccessMessages) 1-1 messages: 0 Room attachments: 0 (0.0B) 1-1 attachments: 0 (0) [2017-10-24T19:45:00] [INFO] - Cleaning up working directory.
The number of archived rooms and GuestAccessMessages both factor into the total number of messages that will be imported. At this stage, these numbers are not an indication of success or failure.
Outputs statistics from a Hipchat Server/DC import process.
Use this function after importing Hipchat data. Please note that HCS/HCDC has an automatic log rotation mechanism, so the correct log file may be zipped in the /var/log/hipchat (HCS) or /file_store/shared/logs (HCDC) directory.
$ ./hipchat_toolkit.run data import_log_stats Usage: -l <full location of hcapp.log> : log file which contains the import messages (must contain entry "Starting import job" and following)
$ ./hipchat_toolkit.run data import_log_stats -l /var/log/hipchat/hcapp.log ... [2017-10-24T19:39:16] [INFO] - Parsing logs for import with ID: e949d7c0-6cdf-4fff-aac7-b9cb3b505d11 Start timestamp: 2017-10-24T18:46:42.880169+00:00 Finish timestamp: 2017-10-24T19:24:31.605364+00:00 Duration: 37 minutes and 49 seconds Users: 5643 (1 on server before import) Rooms: 398 Room import errors: Failed messages: 16 Failed attachments: 0
If an unacceptably high number of messages fail during an import, please reach out to support. Please be sure to include the versions of HCS/HCDC involved in the export and import, in addition to the following log files from each server involved in the export/import:
This will allow for a quicker start on troubleshooting the issue.
Outputs counts of data contained in a HCDC database.
Use this function after importing Hipchat data into Hipchat Data Center.
$ ./hipchat_toolkit.run data database_stats
$ ./hipchat_toolkit.run data database_stats ... [2017-10-24T19:48:28] [INFO] - Tallying values Users: 5643 Rooms: 399 (12 archived) Room messages: 269586 1-1 messages: 0 Room attachments: 1054 1-1 attachments: 0
When exporting and importing data between HCS/HCDC environments, the number of room messages in the database will not directly match the number in the export archive file based on the following factors:
- GuestAccessMessages in the archive file will fail to import
- a duplicate message will be posted for each archived room in the export archive (the duplicate message is sent to the archived room stating that the room is archived)
- any failed messages in the import logs will not be counted in the message total
As a result, the number of room messages listed in the export archive stats can be calculated with the following equation:
export_archive_stats:room messages - export_archive_stats:GuestAccessMessages + export_archive_stats:archived rooms - import_log_stats:failed messages
- The number of rooms listed may be incremented based on how many rooms that existed prior to importing an archive file.
For example, in a freshly installed HCS/HCDC instance, there should only be one room called "Default", so there will be one more room in
- Placeholders for files sent through Hipchat are created during an import whether or not they are included in an export, so there will be some number of Room and/or 1-1 attachments counted whether or not attachments are included in the export archive.