Problem

Running the following command on HipChat Server fails with the [Errno 30] Read-only file system error:

$ hipchat service
Traceback (most recent call last):
File "/opt/atlassian/hipchat/sbin/_service.py", line 184, in <module>
format='%(asctime)s %(levelname)s %(message)s', )
File "/usr/lib/python2.7/logging/__init__.py", line 1532, in basicConfig
hdlr = FileHandler(filename, mode)
File "/usr/lib/python2.7/logging/__init__.py", line 903, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib/python2.7/logging/__init__.py", line 928, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 30] Read-only file system: '/var/log/hipchat/hipchat-server-cli.log'

$ hipchat stats -w
Could not connect to the database: [Errno 30] Read-only file system: '/var/log/hipchat/mymysql.log'
Active Users: 11 today, 13 yesterday
CPU: 0.7% of 4 cores
Chat Sessions: 29 now, 32 peak yesterday
Clock: 2018-03-22 15:51:42 UTC
Configuration Manager: OK
File Attachments: 31111 MB, 82.8% of capacity
Logs: 287 MB, 7.1% of capacity
Memory: 78.0% of 4 GB total, 0 GB cache
Rooms: 0 active, 0 archived, 0 deleted
Total Messages: 158167
Total Users: 0
Version: Atlassian HipChat 2.2.9 (2018.01.23.110010)

Diagnosis

Environment

• HipChat Server 2.2.9

Diagnostic Steps

• Reboot the server and take note of the following error messages:

  util.py[WARNING]: No instance datasource found! Likely bad things to come!

Cause

The disk being used for the data center deployment is corrupted.

Workaround

1. Reboot the VM instance normally and press S to continue the booting without mounting the problematic partition

2. Stop all services

3. Unmount /file_store from the system

4. Check for errors using the -n parameter (to avoid the attempt to repair any problems, but simply report such problems to stdout)

   fsck -n -v /dev/chat_history_vg/chat_history

5. Correct the errors:

   fsck -v /dev/chat_history_vg/chat_history

6. Perform the same procedure for other file systems:

   fsck -n -v /dev/file_store_vg/file_store
   fsck -v /dev/file_store_vg/file_store

7. Reboot the server again

8. Confirm that messages are sent properly through the client
9. Double check the chat history search through the web interface