Bundled Elasticsearch/OpenSearch does not start after installing or upgrading Bitbucket Data Center using binaries
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bundled Elasticsearch/OpenSearch does not start after installing or upgrading Bitbucket Data Center using Linux binaries.
Environment
- Bitbucket Data Center
- Linux
- Embedded Elasticsearch/OpenSearch
Diagnosis
Running the ps command below shows only one java process running for Bitbucket and no java process is shown for Elasticsearch/OpenSearch:
> ps aux | grep java
user 6113 10.2 25.4 4450132 1036100 ? Sl 13:57 2:09 /opt/atlassian/bitbucket/7.4.0/jre/bin/java -classpath /opt/atlassian/bitbucket/7.4.0/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/var/atlassian/application-data/bitbucket -Dbitbucket.install=/opt/atlassian/bitbucket/7.4.0 -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/var/atlassian/application-data/bitbucket/tmp -Djava.library.path=/opt/atlassian/bitbucket/7.4.0/lib/native;/var/atlassian/application-data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start
user 6580 0.0 0.0 17676 660 pts/0 S+ 14:18 0:00 grep --color=auto java
The output above only shows the Bitbucket process, but it is expected to have a second java process for Elasticsearch/OpenSearch, such as in the example below:
For ElasticSearch
user 7441 115 30.5 4096488 1247196 pts/0 Sl 14:35 0:12 /opt/atlassian/bitbucket/7.4.0/jre/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=COMPAT -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/var/atlassian/application-data/bitbucket/tmp/search -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:MaxDirectMemorySize=536870912 -Des.path.home=/opt/atlassian/bitbucket/7.4.0/elasticsearch -Des.path.conf=/var/atlassian/application-data/bitbucket/shared/search -Des.distribution.flavor=oss -Des.distribution.type=zip -Des.bundled_jdk=true -cp /opt/atlassian/bitbucket/7.4.0/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/atlassian/application-data/bitbucket/log/search/elasticsearch.pid
For Opensearch
atlbitb+ 13091 1 93 11:25 pts/1 00:00:02 /opt/atlassian/bitbucket/7.21.3/jre/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,JRE -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/atlassian/application-data/bitbucket_new/tmp/search -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:MaxDirectMemorySize=536870912 -Dopensearch.path.home=/opt/atlassian/bitbucket/7.21.3/opensearch -Dopensearch.path.conf=/var/atlassian/application-data/bitbucket_new/shared/search/config -Dopensearch.distribution.type=tar -Dopensearch.bundled_jdk=true -cp /opt/atlassian/bitbucket/7.21.3/opensearch/lib/* org.opensearch.bootstrap.OpenSearch -d -p /var/atlassian/application-data/bitbucket_new/log/search/opensearch.pid
During the installation or the upgrade to Bitbucket data center either the Data Center instance or Upgrade an existing Data Center instance and Install Bitbucket as a service options were chosen
|
|
|
Cause
When selecting Data Center instance or Upgrade an existing Data Center instance options during the installation process, the --no-search switch will be added to the service startup script, causing only Bitbucket to start.
Workaround
Edit /etc/init.d/atlbitbucket script and remove the --no-search from the following line:
atlbitbucket servicestart() { /<bitbucket_install-path>/bin/start-bitbucket.sh --no-search }To:
atlbitbucket servicestart() { /<bitbucket_install-path>/bin/start-bitbucket.sh }- Save the atlbitbucket file.
- Restart Bitbucket service.
Solution
Bitbucket data center requires an external Elasticsearch/OpenSearch instance, as detailed in the Install Bitbucket Data Center guide:
- A remote Elasticsearch/OpenSearch instance with only one remote connection to Bitbucket is required. The instance may be a standalone Elasticsearch/OpenSearch installation or a clustered installation behind a load balancer.
- For help installing and configuring a remote Elasticsearch/OpenSearch instance see our documentation.