Unable to complete Hipchat Server export due to insufficient disk space
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.
This is for an outdated version of Hipchat Server
This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.
Problem
Hipchat Server exports fail regardless of whether they are run from the Web UI, from Server admin > Export, from the command line, or from the backup script.
The export runs successfully until the exporter tool attempts to encrypt it, which is one of the very last steps in the process.
The following appears in the /var/log/hipchat/hcapp.log:
2017-05-23T14:25:28.090600+00:00 hipchat scissortail-worker:20579:INFO GID: 1 JID: 751809fb-d1b8-4626-9209-cdb52c272da6 - Couldn't encrypt the file: Command '['openssl', 'aes-256-cbc', '-in', '/file_store/scissortail/tmpvq8/hipchat-2017-05-23_13-54-01.tar.gz', '-out', '/file_store/scissortail/tmpvq8/hipchat-2017-05-23_13-54-01.tar.gz.aes', '-pass', 'env:1118030fc4874dc5af47f885fc51a6b2']' returned non-zero exit status 1
2017-05-23T14:25:28.824028+00:00 hipchat scissortail-worker:20579:ERROR GID: 1 JID: 751809fb-d1b8-4626-9209-cdb52c272da6 - Job 751809fb-d1b8-4626-9209-cdb52c272da6 failedThe following errors may also be seen towards the end of the /var/log/hipchat/runtime.log:
+ grep monit /var/log/hipchat/daemon.log
+ tail -n 10
2017-05-23T22:58:28.792370+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:00:30.673295+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:02:32.555099+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:04:34.433574+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:06:36.322431+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:08:38.200033+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:10:40.088419+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:12:41.967497+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:14:43.855672+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
2017-05-23T23:16:45.735505+00:00 esgdachat monit[12127]: 'file_store_vg-file_store' space usage 91.0% matches resource limit [space usage>90.0%]
Diagnosis
Review /var/log/hipchat/runtime.log and specifically look for this section, which deals with disk usage:
+ uname -a
Linux hipchat.atlassiansupport.com 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
+ df -m
Filesystem 1M-blocks Used Available Use% Mounted on
udev 3980 1 3980 1% /dev
tmpfs 799 1 798 1% /run
/dev/sda6 11617 4403 6617 40% /
none 1 0 1 0% /sys/fs/cgroup
none 5 0 5 0% /run/lock
none 3993 0 3993 0% /run/shm
none 100 0 100 0% /run/user
/dev/sda1 704 104 564 16% /boot
/dev/sda7 2960 1118 1689 40% /var/log
/dev/mapper/file_store_vg-file_store 151058 137506 6388 96% /file_store
/dev/mapper/chat_history_vg-chat_history 9948 496 8940 6% /chat_historyAlternatively, from the server's terminal/command-line run df -h which provides similar output. From the example above, /dev/mapper/file_store_vg-file_store, mounted on /file_store, is using 96% of the total 151 GB allocated for the partition. This specific disk is does not have enough space to allow the exporter to successfully compress and encrypt the export archive.
Cause
/file_store is running out of disk space.
Resolution
Increase the size of the virtual disk for the /file_store drive. For reference, see Increasing disk capacity.
Also, be aware of HCPUB-3031 - Getting issue details... STATUS . Double check if the Hipchat Server is affected by the problem described in the bug report and remove any unwanted export files from the local filesystem to free up space, if necessary.