Summary

While starting up the bundled OpenSearch service in Bitbucket, it fails with below error in the atlassian-bitbucket.log.

Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read C:\Atlassian\Bitbucket\7.21.3\opensearch\config\bitbucket.pem

The environmental variable BITBUCKET_HOME is set and the bitbucket service functions perfectly.

Environment

Bitbucket Server 7.21.0+ Installed on Windows Server.

Diagnosis

Confirm the OpenSearch configuration path from the bitbucket_search.log. It can be found from a line similar to this.

[2022-08-23T09:35:42,197][INFO ][o.o.n.Node] [bitbucket_bundled] JVM arguments [-Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=C:\Users\Prathima.G\AppData\Local\Temp\3\opensearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:MaxDirectMemorySize=536870912, -Dopensearch, -Dopensearch.path.home=C:\Atlassian\Bitbucket\7.21.3\opensearch, -Dopensearch.path.conf=C:\Atlassian\Bitbucket\7.21.3\opensearch\config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=false, exit, abort, -Xms1024m, -Xmx1024m, -Xss1024k]

Here the path that's being passed to the service is -Dopensearch.path.conf=C:\Atlassian\Bitbucket\7.21.3\opensearch\config. Cross verify this path with the Bitbucket home directory in your instance.

Cause

The root cause is currently not known. While doing the upgrade to the version 7.21.x, one of the processes in the installer is resetting the path of the OpenSearch service to the OpenSearch configuration folder in the Bitbucket Installation directory, instead of the one in Bitbucket home directory. So we will have to manually change this in the service configuration. 

So far this issue has been reported only with Bitbucket installed on Windows Environment. 

Solution

Follow the below steps to change the path of the OpenSearch service.

1. Confirm all OpenSearch SSL security configuration files(root-ca.pem,bitbucket.pem and bitbucket-key.pem) are located in the right place, <bitbucket-home-directory>\shared\search\config 
2. Confirm the exact Bitbucket Search service name, by going to the Services console and checking the Atlassian Bitbucket Search.
3. Now open a command prompt and go to the OpenSearch binaries directory <bitbucket-installation-directory>\opensearch\bin
4. Execute the command opensearch-service-mgr.exe //ES//<BitbucketSearch Service Name>
5. In the pop-up, go to the Java tab.
6. Go to Java Options and find the line -Dopensearch.path.conf, it is near to the bottom.
7. Confirm the path is <bitbucket-home-directory>\shared\search\config
8. If the path is not the one above, change it accordingly and click Apply to confirm the changes.
9. Try to restart the service and confirm it's working fine.