"Couldn't upload the file" error due to low disk space on Hipchat Server
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.
Problem
When uploading attachments to Hipchat Server through a Hipchat client, the error Couldn't upload the file is thrown.
Diagnosis
The following error is thrown in the /var/log/hipchat/coral.log log file:
2017-05-23T14:32:42.354529+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] =====> New API request: #27579, uri=/user/1/share/file, method=POST, from=127.0.0.1, job_handle=none
2017-05-23T14:32:42.355373+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Fetching an oauth session for token XXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxx
2017-05-23T14:32:42.356255+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Success loading oauth session from cache for token XXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxx
2017-05-23T14:32:42.357518+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Access token retrieved from cache for oauth_session:42511:accesstoken
2017-05-23T14:32:42.357771+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Authenticated session 42511 by access token XXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxx
2017-05-23T14:32:42.359488+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Auth (internal) successful for uid-1
2017-05-23T14:32:42.359595+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Authenticated 1 via user oauth session
2017-05-23T14:32:42.363007+00:00 hipchat coral-1: [HiRedisClient,client] Traceback (most recent call last):
2017-05-23T14:32:42.363059+00:00 hipchat coral-1: [HiRedisClient,client] Failure: exceptions.IOError: [Errno 28] No space left on device
2017-05-23T14:32:42.363153+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] ALERT: 500 error processing request /user/1/share/file : IOError(28, 'No space left on device')
2017-05-23T14:32:42.363224+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] ERROR: Sending API error: 500 - An internal error occurred. Please try again later.
2017-05-23T14:32:42.363650+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Traceback (most recent call last):
2017-05-23T14:32:42.363706+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] Failure: exceptions.IOError: [Errno 28] No space left on device
2017-05-23T14:32:42.363808+00:00 hipchat coral-1: [/user/1/share/file#XXXXxxxx] WARN: =====> API request failed: #27579, in-progress=0, code=500
Cause
The server does not have any available disk space to store new attachments.
Resolution
Determine which volume is out of space by running the following command on the Hipchat Server terminal / SSH console
df -hFor example, /dev/xvda1 is out of space in the output below and needs to be expanded or freed up:
admin@hipchat:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 1.9G 12K 1.9G 1% /dev tmpfs 377M 460K 377M 1% /run /dev/xvda1 59G 59G 0G 100% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 1.9G 0 1.9G 0% /run/shm none 100M 0 100M 0% /run/user- With reference to the Hipchat Server disk space is full, perform any of the following steps:
- Increase the size of the storage volumes per our knowledge base article for Increasing disk capacity
- Investigate the disk space utilization on the server and remove large files / directories
There is a specific case related to Overflow filesystem mounted as /tmp in linux where this problem may occur even though df -h shows sufficient disk space. Once the system has an appropriate amount of available disk space, remounting the /tmp volume will resolve the issue.