Problem

Elasticsearch, a service running on Hipchat Server, has an incompatibility with OpenJDK v1.7.0_55 and fails to start with the following error appearing in /home/admin/logs/elasticsearch.log/elasticsearch.log:

2015-10-17T13:21:35+00:00 hipchat.example.com [ERROR][bootstrap                ] [hces]Exception
2015-10-17T13:21:35+00:00 hipchat.example.com java.lang.RuntimeException: Java version: 1.7.0_55 suffers from critical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause data corruption.
2015-10-17T13:21:35+00:00 hipchat.example.com Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.
2015-10-17T13:21:35+00:00 hipchat.example.com If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JVM_OPTS environment variable.
2015-10-17T13:21:35+00:00 hipchat.example.com Upgrading is preferred, this workaround will result in degraded performance.

Diagnosis

Environment

• Hipchat Server v1.3.x upgraded from a pre-beta deployment

Diagnostic Steps

• Long-term chat history (more than 75 messages prior to current) may fail to load
• Hipchat Server exports may fail

Cause

Pre-beta versions of Hipchat Server (~ v1.0.x) came bundled with OpenJDK v1.7.0_55, which is not upgraded as part of Hipchat Server application updates up to Hipchat Server v1.3.5.

Workaround

In an emergency, the following steps can be used to restart Elasticsearch:

Add Parameter Emergency Workaround

1. Log into the Hipchat Server command line interface
2. Run the following command to gain root access to the system:
   sudo dont-blame hipchat
3. Open the /usr/local/etc/elasticsearch/elasticsearch-env.sh file with nano or vim to add "-XX:UseSuperWord" under the "ES_JAVA_OPTS=" section
4. Run the following command to start Elasticsearch:
   service elasticsearch start

A healthier long-term option, is to patch OpenJDK with the following steps:

Patch to Upgrade OpenJDK

1. Log into the command-line interface

2. Run the following command to gain root access to the system: 

    sudo dont-blame-hipchat

3. Add a new directory for the package download/unpack

   mkdir /tmp/jdk 

4. Change to the new temporary directory

    cd /tmp/jdk

5. Download the necessary files for installing the Java

    wget https://hipchat-server-stable.s3.amazonaws.com/patch/openjdk/openjdk-7-7u65-2.5.1-4ubuntu1~0.12.04.2.tar

6. Unpack the resulting tar file

    tar -xvf openjdk-7-7u65-2.5.1-4ubuntu1~0.12.04.2.tar

7. Install the downloaded Java packages

    dpkg -i *.deb

8. Update apt for the installation

    apt-get -f -y install

9. Restart the database-related services

    hipchat service -u db

10. Restart the remaining services

     hipchat service -u

11. Check that Java is up to 1.7.0_65

     java -version

Resolution

Upgrade to the latest version of Hipchat Server to tackle this issue permanently.