Hipchat Server disk space is full

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.

This is for an outdated version of Hipchat Server

 This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.

When will my version be deprecated?

The following versions have been deprecated:

  • Hipchat Server 1.3 (EOL Date: Aug 17, 2017)
  • Hipchat Server 2.0 (EOL Date: Jun 17, 2018)
  • Hipchat Server 2.1 (EOL Date: Dec 8, 2018)

The following version will be deprecated soon:

  • Hipchat Server 2.2 (EOL Date: May 30, 2019)

You can read more about Atlassian's End of Life policy here.

You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.

Please be sure to contact Hipchat Server support in the event that you experience this complication so that we can investigate and file any necessary bug reports to bolster the application's resiliency against this type of problem.

Problem

One, or more, Hipchat Server storage volume(s) is full or has almost filled up. This can result in a variety of problems, such as users being unable to log in.

The following alert, or similar, may be emailed to the Hipchat Server administrative user:

Action: alert
Desc: space usage 95.1% matches resource limit [space
usage>75.0%]
Event: Resource limit matched
Service: sda6
Date: Mon, 08 Jun 2015 15:37:52
Manage: https://hipchat.domain.dom/

Diagnosis

Log into the Hipchat Server through the virtual machine/AWS interface or SSH into the server and run the following command:

df -h

This command provides a readout of the filesystem, including space usage for each mount point. Here's a sample output from an OVA server:

Filesystem                                Size  Used Avail Use% Mounted on
/dev/sda6                                  12G  11.5G  0.5G  96% /
udev                                      2.0G   12K  2.0G   1% /dev
tmpfs                                     395M  396K  394M   1% /run
none                                      5.0M     0  5.0M   0% /run/lock
none                                      2.0G     0  2.0G   0% /run/shm
/dev/sda1                                 720M   62M  622M  10% /boot
/dev/sda7                                 3.9G  346M  3.4G  10% /var/log
/dev/mapper/file_store_vg-file_store       40G  193M   38G   1% /file_store
/dev/mapper/chat_history_vg-chat_history  9.9G  151M  9.2G   2% /chat_history

In the example above, we can see that the '/' (root) directory is at 96% use, which is cause for concern and informs where to look to resolve the problem.

Cause

There are a number of reasons that storage on Hipchat Server will fill up, including regular use. The steps provided in the Resolution will likely indicate a root cause for the space utilization.

Resolution

There are two possible resolutions for this problem:

  • Increase the size of the storage volumes per our documentation for Increasing disk capacity

  • Investigate the disk space utilization on the server and remove large files/directories:

    After accessing the Hipchat Server via SSH or direct access, run the following command: 

    find / -path /sys -prune -o -path /proc -prune -o -printf "%s - %p\n" 2>/dev/null | sort -n | tail -10
    Expand for details on command above...

    find ## Defines the starting directory at which a search will be run

    -path /sys -prune -o -path /proc -prune ## Prevents searching in the /sys and /proc directories

    -printf "%s - %p\n" ## Print output in size (bytes), then file name, then start a new line

    2>/dev/null ## Prevent errors from showing in the output

    sort -n ## Sort results numerically (smallest at the top, largest at the bottom)

    tail -10 ## Truncate results to the last ten items

    (info) Please note that the command above will not scan the entire filesystem without first gaining root access to the server. Doing so may not be necessary, depending on the files that need to be removed to clear space.

    Here is a sample output of the command above:

    admin@hipchat:~$ find / -path /sys -prune -o -path /proc -prune -o -printf "%s - %p\n" 2>/dev/null | sort -n | tail -10
54614896 - /opt/atlassian/hipchat/clients/linux/yum/x86_64/hipchat-2.2.1328-1.x86_64.rpm
56958754 - /opt/atlassian/hipchat/clients/linux/apt/pool/main/h/hipchat/hipchat_2.2.1328_i386.deb
57307932 - /opt/atlassian/hipchat/clients/linux/apt/pool/main/h/hipchat/hipchat_2.2.1328_amd64.deb
69024481 - /var/log/hipchat/nginx.log
77806738 - /var/cache/apt/archives/hc-crowd_1.2.5-2015.04.23.1917491.2_all.deb
90494991 - /var/log/cumulus.log
118769604 - /var/cache/apt/archives/hc-crappie_1.2.5-2015.04.23.1917491.2_all.deb
267113614 - /var/cache/apt/archives/hc-btf_1.2.5-2015.04.23.1917491.2_all.deb
383039922 - /var/cache/apt/archives/hc-client-bundle_1.2.3-2015.04.08.2130141.2_all.deb
650153536 - /home/admin/bigfile.uhoh
admin@hipchat:~$

    In the example above, /home/admin/bigfile.uhoh is the largest file found on the server and will need to be addressed.

    We recommend that files be copied away from the server (see the man page for 'scp' to securely copy the file off the server), then truncated to clear out their contents. We do not recommend that files on Hipchat Server be deleted outright as this may produce unexpected results/complications. To truncate the file, please use the following command:

     echo "" > /absolute/path/to/file/causing/problems.foo

    This will cause the offending file to be emptied.

Last modified on Jan 19, 2018

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.