Elasticsearch Fails to Start due to Incompatible Java Version
Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.
This version of Hipchat Server is no longer supported
This article applies to a version of Hipchat Server which is beyond the Atlassian End of Life policy, and is no longer supported.
The following versions have been deprecated:
- Hipchat Server 1.3 (EOL Date: Aug 17, 2017)
The following versions will be deprecated soon:
- Hipchat Server 2.0 (EOL Date: Jun 17, 2018)
- Hipchat Server 2.1 (EOL Date: Dec 8, 2018)
- Hipchat Server 2.2 (EOL Date: May 30, 2019)
You can read more about Atlassian's End of Life policy here
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes. If possible, you should also consider deploying Hipchat Data Center instead.
Elasticsearch, a service running on Hipchat Server, has an incompatibility with OpenJDK v1.7.0_55 and fails to start with the following error appearing in /home/admin/logs/elasticsearch.log/elasticsearch.log:
2015-10-17T13:21:35+00:00 hipchat.example.com [ERROR][bootstrap ] [hces]Exception 2015-10-17T13:21:35+00:00 hipchat.example.com java.lang.RuntimeException: Java version: 1.7.0_55 suffers from critical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause data corruption. 2015-10-17T13:21:35+00:00 hipchat.example.com Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations. 2015-10-17T13:21:35+00:00 hipchat.example.com If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JVM_OPTS environment variable. 2015-10-17T13:21:35+00:00 hipchat.example.com Upgrading is preferred, this workaround will result in degraded performance.
- Hipchat Server v1.3.x upgraded from a pre-beta deployment
- Long-term chat history (more than 75 messages prior to current) may fail to load
- Hipchat Server exports may fail
Pre-beta versions of Hipchat Server (~ v1.0.x) came bundled with OpenJDK v1.7.0_55, which is not upgraded as part of Hipchat Server application updates up to Hipchat Server v1.3.5.
In an emergency, the following steps can be used to restart Elasticsearch:
Add Parameter Emergency Workaround
- Log into the Hipchat Server command line interface
- Run the following command to gain root access to the system:
sudo dont-blame hipchat
- Open the /usr/local/etc/elasticsearch/elasticsearch-env.sh file with nano or vim to add "-XX:UseSuperWord" under the "ES_JAVA_OPTS=" section
- Run the following command to start Elasticsearch:
service elasticsearch start
A healthier long-term option, is to patch OpenJDK with the following steps:
The following steps require restarting components within Hipchat Server and will result in temporary disruption of service.
Patch to Upgrade OpenJDK
- Log into the command-line interface
Run the following command to gain root access to the system:
Add a new directory for the package download/unpack
Change to the new temporary directory
Download the necessary files for installing the Java
Unpack the resulting tar file
tar -xvf openjdk-7-7u65-2.5.1-4ubuntu1~0.12.04.2.tar
Install the downloaded Java packages
dpkg -i *.deb
Update apt for the installation
apt-get -f -y install
Restart the database-related services
hipchat service -u db
Restart the remaining services
hipchat service -u
Check that Java is up to 1.7.0_65
Upgrade to the latest version of Hipchat Server to tackle this issue permanently.