Problem

When searching Hipchat message history while connected to Hipchat Server, the follow message is displayed in the interface:

We are temporarily unable to load chat history for this room.

The following appears in the /home/admin/logs/tetra.log and/or /home/admin/logs/tetra.err.log files:


Unable to read history from Elasticsearch: Connection was refused by other side: 111: Connection refused.

Diagnostic Steps

Potential Triggers

• Hipchat Server was recently upgraded to Hipchat Server v1.3.1 or newer.
• The server ran low on memory —less than 5% free— and the ElasticSearch process crashed. 

Diagnosis

Log into the Hipchat Server command line interface and run the following command:

ps aux | grep elastic

If the output does not show an active Java application process, this knowledge base article applies:

 admin    29369  0.0  0.0   8108   940 pts/0    S+   22:55   0:00 grep --color=auto elastic

If the output does show an active Java application process, another problem may be present - please contact Hipchat Server Support for assistance.

9002      2159  0.1  6.5 2125668 247796 ?      Sl   Sep11  13:19 /usr/bin/java -server -Djava.net.preferIPv4Stack=true -Des.config=/usr/local/etc/elasticsearchelasticsearch.yml -Xms739m -Xmx739m -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+UseCondCardMark -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -Delasticsearch -Des.pidfile=/usr/local/var/run/elasticsearch/shanye_impen_com.pid -Des.path.home=/usr/local/elasticsearch -cp :/usr/local/elasticsearch/lib/*:/usr/local/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.ElasticSearch
admin    29369  0.0  0.0   8108   940 pts/0    S+   22:55   0:00 grep --color=auto elastic

Cause

The ElasticSearch service has stopped and is no longer available to query for or store chat history.

Workaround

The fail-over chat history log will house messages (in JSON format) that did not write to Elasticsearch when it was down. It is located at:

• /hipchat/tetra-app/es_history_write_failures.log for 1.4.0 and above. 

• /hipchat/tetra-app/elasticsearch_write_failures.log versions older than 1.4.0. 

Restoring messages from elasticsearch_write_failure.log

The process below outlines the steps to import the message history (complete with timestamps) back into Elasticsearch

1) Inside the HCS console, elevate privileges

 sudo dont-blame-hipchat

2) Upload the attached elastic_import_prereqs.sh to the server under /home/admin and run it: 

bash elastic_import_prereqs.sh

3) Navigate to the directory where fabfile.py is located

cd /hipchat-scm/tetra/tools/history

4) Verify that Elasticsearch is running

Please see the Diagnosis section above.

If it is not running, please start it by running:

sudo service elasticsearch start

5) Import the fabfile.puy script and call the restore function passing in the path to the ES flat file.

If you're on Hipchat Server 1.4.0 and up, run:

python -c "import fabfile; fabfile.import_local_elasticsearch_file('/hipchat/tetra-app/es_history_write_failures.log','es_history')"

else:

python -c "import fabfile; fabfile.import_local_elasticsearch_file('/hipchat/tetra-app/elasticsearch_write_failures.log')"

This will start the import process. An example of the output will look like:

20% - 1/5 - u'b6f21bb3-8cf8-42e1-8061-cbc933f7dc57'
40% - 2/5 - u'1cc6abbe-cb2f-4801-86d6-836301a97b56'
60% - 3/5 - u'5e3ca9fa-a8ad-4b60-9df8-a494b19fc9ca'
80% - 4/5 - u'52d8c32b-9e0a-4005-bf55-0471bc46a0ef'
100% - 5/5 - u'037c3466-e8bb-4c7e-8260-b97e029508f4'

If you have any issues, please file a support ticket at support.atlassian.com.