Hipchat Server disk space is full
This article only applies to the Atlassian server platform. Learn more about the differences between cloud and server.
This is for an outdated version of Hipchat Server
This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.
The following versions have been deprecated:
- Hipchat Server 1.3 (EOL Date: Aug 17, 2017)
- Hipchat Server 2.0 (EOL Date: Jun 17, 2018)
- Hipchat Server 2.1 (EOL Date: Dec 8, 2018)
The following version will be deprecated soon:
- Hipchat Server 2.2 (EOL Date: May 30, 2019)
You can read more about Atlassian's End of Life policy here.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.
Please be sure to contact Hipchat Server support in the event that you experience this complication so that we can investigate and file any necessary bug reports to bolster the application's resiliency against this type of problem.
One, or more, Hipchat Server storage volume(s) is full or has almost filled up. This can result in a variety of problems, such as users being unable to log in.
The following alert, or similar, may be emailed to the Hipchat Server administrative user:
Desc: space usage 95.1% matches resource limit [space
Event: Resource limit matched
Date: Mon, 08 Jun 2015 15:37:52
Log into the Hipchat Server through the virtual machine/AWS interface or SSH into the server and run the following command:
This command provides a readout of the filesystem, including space usage for each mount point. Here's a sample output from an OVA server:
Filesystem Size Used Avail Use% Mounted on /dev/sda6 12G 11.5G 0.5G 96% / udev 2.0G 12K 2.0G 1% /dev tmpfs 395M 396K 394M 1% /run none 5.0M 0 5.0M 0% /run/lock none 2.0G 0 2.0G 0% /run/shm /dev/sda1 720M 62M 622M 10% /boot /dev/sda7 3.9G 346M 3.4G 10% /var/log /dev/mapper/file_store_vg-file_store 40G 193M 38G 1% /file_store /dev/mapper/chat_history_vg-chat_history 9.9G 151M 9.2G 2% /chat_history
In the example above, we can see that the '/' (root) directory is at 96% use, which is cause for concern and informs where to look to resolve the problem.
There are a number of reasons that storage on Hipchat Server will fill up, including regular use. The steps provided in the Resolution will likely indicate a root cause for the space utilization.
There are two possible resolutions for this problem:
- Increase the size of the storage volumes per our documentation for Increasing disk capacity
Investigate the disk space utilization on the server and remove large files/directories:
After accessing the Hipchat Server via SSH or direct access, run the following command:
find / -path /sys -prune -o -path /proc -prune -o -printf "%s - %p\n" 2>/dev/null | sort -n | tail -10Expand for details on command above...
find <directory>## Defines the starting directory at which a search will be run
-path /sys -prune -o -path /proc -prune## Prevents searching in the /sys and /proc directories
-printf "%s - %p\n"## Print output in size (bytes), then file name, then start a new line
2>/dev/null## Prevent errors from showing in the output
sort -n## Sort results numerically (smallest at the top, largest at the bottom)
tail -10## Truncate results to the last ten items
Please note that the command above will not scan the entire filesystem without first gaining root access to the server. Doing so may not be necessary, depending on the files that need to be removed to clear space.
Here is a sample output of the command above:
admin@hipchat:~$ find / -path /sys -prune -o -path /proc -prune -o -printf "%s - %p\n" 2>/dev/null | sort -n | tail -10 54614896 - /opt/atlassian/hipchat/clients/linux/yum/x86_64/hipchat-2.2.1328-1.x86_64.rpm 56958754 - /opt/atlassian/hipchat/clients/linux/apt/pool/main/h/hipchat/hipchat_2.2.1328_i386.deb 57307932 - /opt/atlassian/hipchat/clients/linux/apt/pool/main/h/hipchat/hipchat_2.2.1328_amd64.deb 69024481 - /var/log/hipchat/nginx.log 77806738 - /var/cache/apt/archives/hc-crowd_1.2.5-2015.04.23.1917491.2_all.deb 90494991 - /var/log/cumulus.log 118769604 - /var/cache/apt/archives/hc-crappie_1.2.5-2015.04.23.1917491.2_all.deb 267113614 - /var/cache/apt/archives/hc-btf_1.2.5-2015.04.23.1917491.2_all.deb 383039922 - /var/cache/apt/archives/hc-client-bundle_1.2.3-2015.04.08.2130141.2_all.deb 650153536 - /home/admin/bigfile.uhoh admin@hipchat:~$
In the example above,
/home/admin/bigfile.uhohis the largest file found on the server and will need to be addressed.
We recommend that files be copied away from the server (see the man page for 'scp' to securely copy the file off the server), then truncated to clear out their contents. We do not recommend that files on Hipchat Server be deleted outright as this may produce unexpected results/complications. To truncate the file, please use the following command:
echo "" > /absolute/path/to/file/causing/problems.foo
This will cause the offending file to be emptied.