Code Search Troubleshooting Guide for Bitbucket Server and DataCenter

Still need help?

The Atlassian Community is here for you.

Ask the community

Bitbucket Data Center and Server allow you to search through the code to find exactly what you’re looking for, right from the search bar using Elasticsearch / OpenSearch. Bitbucket Data Center requires a remote Elasticsearch server / OpenSearch server for multi-node Data Center deployments. The Data Center can have only one remote connection to a search service for your cluster. This may be a standalone search service installation or a clustered installation behind a load balancer. For more details on code search, refer to Administer code search



General FAQ

Can I use the bundled search server with Bitbucket Data Center?

If you're installing a single-node Bitbucket Data Center, you can use the bundled search server. The bundled search gets automatically configured with its scripts out of the box.

If you’re installing a multi-node Bitbucket Data Center, you can't use a bundled search server. For a clustered Data Center (Data Center installation type), you would need to install it separately on a remote server.

Can I use a clustered installation of a search server with Bitbucket Data Center?

Yes, however, Bitbucket Data Center can have only one remote connection to the shared search server for your cluster. This may be a search server installation or a clustered installation behind a load balancer. For guidance on using a clustered search server installation with Bitbucket Data Center, see the page Use a clustered search server with Bitbucket Data Center.

Does Bitbucket Data Center include a license to use remote search?

No. The versions of Elasticsearch above a certain version (7.10) may require the purchase of a separate license from Elastic. You may need to contact Elastic for more information.

OpenSearch is open source and should not require a license.

From https://opensearch.org/faq/
All of the software in the OpenSearch project is released under the Apache License, Version 2.0 (ALv2). The ALv2 license grants you well-understood usage rights for OpenSearch. You can use, modify, extend, embed, monetize, resell, and offer OpenSearch as part of your products and services. We have also published permissive usage guidelines for the OpenSearch trademark, so you can use the name to promote your offerings.

Is a search server required to run Bitbucket Server/Data Center?

No. You can run a Bitbucket Server / Data Center without a search server. Bitbucket itself will run as expected, but some features (such as code search, advanced repository management etc.) will not work.

How can I disable code search functionality entirely?

If you need to turn off code search for all of your Bitbucket users, you can disable code search entirely from within the bitbucket.properties file. See the page Bitbucket Server config properties - Search for details.

I can't configure search server settings from the UI because the fields are greyed out

Your search server was configured using the bitbucket.properties file. If a parameter is set in the bitbucket.properties file, it can't be edited later from the admin UI. Any changes that need to be made to the search server configuration must be made within the bitbucket.properties file.
:info: The changes to the bitbucket.properties file will take effect only after the restart of all the nodes in Bitbucket Data Center.

Following are the parameter which need to be set in bitbucket.properties file.

PropertyParameter name for properties file
URLplugin.search.config.baseurl
Usernameplugin.search.config.username
Passwordplugin.search.config.password
My remote search server is using up a lot of disk space

You can configure the search indexes to exclude forked repositories, and personal repositories, which can save disk space when using a fork-based workflow for development. See the page Bitbucket Server config properties - Search for the options available to change what is indexed for code search.
:info: Note that excluding content from being indexed can prevent such content from ever appearing in code search results, but will not prevent the repositories themselves from appearing in search results.

Code search is unavailable

There are a few ways the search server may not have been configured properly. To troubleshoot the bundled search server that comes with Bitbucket Server, the page Troubleshooting steps for Bitbucket Server Code Search is a good place to start. If you're still stuck, you could post a question in the Atlassian Community (for a Starter license or if your configuration is unique), or contact Atlassian Support for assistance (you'll need to log in to raise a support request).

How do I secure my remote search server?

Atlassian strongly recommends you secure access to your remote search server with a username and password, and a minimum of basic HTTP authentication. See Secure your search server for details.

How to enable DEBUG level logs for bundled search service?

If additional troubleshooting is required and the debug log level of the Bitbucket Server to Search server is required, the following command will enable it:

curl -u <ADMIN_USERNAME> -v -X PUT -d "" -H "Content-Type: application/json" <BASE_URL>/rest/api/latest/logs/logger/com.atlassian.bitbucket.search.internal.indexing/debug 

The additional log lines will be added to the atlassian-bitbucket.log file. Refer to the Enable debug logging page for more information and alternative methods to enable debug logging.

How to check search index progress?

You can see the index progress by navigating to below URL:

<BITBUCKET_BASEURL>/rest/indexing/latest/status

Example results looks like the following:

{
"queues": {
"delay": 0,
"event": 0
},
"status": "IDLE"
} 

Result Definitions:

  • delay = Minutes in which the index queue is delayed. Either by failures, retries, or other performance reasons
  • event = Number of events triggered for index. There are ~2 events per repository. Existing event queue size for indexing is 50k per node.

This provides a look into the current queue. If for some reason there's an event that fails to queue, or there's an error in indexing, it'll be placed onto a queue that delays by 5 minutes. When it sits at zero for a while, we can be confident that indexing is complete (for now). This is a moving target as new events coming in will be added to the queue, but is a good guide for when indexing is done. Overall, as long as search is working, that delayed queue will eventually complete and the status will change to idle.


Elasticsearch

Elasticsearch is a supported search server distribution for Bitbucket Data Center besides OpenSearch. Bitbucket 8. x will no longer support Elasticsearch 5. x, Elasticsearch 6. x, or Elasticsearch versions lower than 7.10.2 due to licensing change. Customers running an external search server will need to use a supported version of Elasticsearch 7.10.2 or higher, or OpenSearch 1.2.4 or higher with Bitbucket 8.0. Refer to Supported platforms for compatible search versions with Bitbucket. Note that, starting Bitbucket 7.21, OpenSearch is bundled with Bitbucket for Elasticsearch due to a change in Elastic Licensing.

The support for Elasticsearch will be deprecated and removed in Bitbucket 9.0. In Bitbucket 9.0 and later, Opensearch will be the only supported search service.


Elasticsearch FAQ

How can I install remote Elasticsearch service?

:info: Before proceeding with the installation of your search engine, check the Supported platforms for compatibility with Bitbucket version.
:warning: Installing an incompatible search version with Bitbucket might cause issues with search functionality.

To setup and install single node remote Elasticsearch, you can refer to Install and configure a remote Elasticsearch server

Where can I find Elasticsearch service logs?

Bundled Elasticsearch
log location: $BITBUCKET_HOME/log/search
Data directory where the indexes are stored: $BITBUCKET_HOME/shared/search/data

Remote Elasticsearch:
Verify the log location in $ES_HOME/config/elasticsearch.yml.
You can configure the log location as per the article https://discuss.elastic.co/t/where-does-elasticsearch-store-logs/200466

Check the Search server and Bitbucket Server logs for error messages

You might have some obvious log messages that could help you address the issue.

These are logs you should check:

  • $BITBUCKET_HOME/log/atlassian-bitbucket.log
  • $BITBUCKET_HOME/log/search/bitbucket_search.log
How to increase pagination in Elasticsearch since every search is limited to 1000 results?

When searching for a word which happens to be present in more than 1000 files in all of Bitbucket repositories, results can be seen for only the first 1000. Subsequent results are not fetched and we are instead presented with the following alert: “The server couldn’t process the request. Please try again later“

You can configure pagination settings by referring to the article How to increase Pagination in Elasticsearch since every search is limited to 1000 results

How to resolve common Elasticsearch issues?

The article How to resolve common Elasticsearch issues helps to resolve Elasticsearch issues related to GC overhead, Low max file descriptors, Low  vm.max_map_count


Elasticsearch Troubleshooting

In many cases, we noticed the search functionality doesn’t work as expected after an upgrade of Bitbucket / Elasticsearch version or migration of Bitbucket from one node to another. The possible reasons may be the Elasticsearch service startup failure, connectivity problems with Bitbucket or indexing issues.

The below checklist can be referred to identify the problem.

Checklist

Check if the Elasticsearch service is started

Run the below command on Bitbucket server (if you are using a bundled search server)

On Linux,

ps -ef | grep search

For the remote Elasticsearch server, run the command on the remote search server

On Windows,

Navigate to Windows Task Manager > Details to verify if the search service is running.

:info: If the search service is down, start the service. Refer to Bundled Elasticsearch/OpenSearch does not start after installing or upgrading Bitbucket Data Center using binaries

Check version compatibility with Bitbucket

Bundled Elasticsearch:

If the Elasticsearch service fails to start, check version compatibility with Bitbucket

Check if the Elasticsearch version is compatible with Bitbucket. Refer to Supported platforms

Remote Elasticsearch:
The search service starts but the functionality may not work as expected in case of incompatible Elasticsearch versions.

Check if environment variables (JRE_HOME, JAVA_HOME) are set correctly

If the Elasticsearch service fails to start:

check if environment variables are set correctly

The below error can be seen when the Elasticsearch startup fails as it could not find Java. This can happen if the JAVA_HOME, and JRE_HOME environment variables are not set correctly.

 Starting the bundled search server                                                                                                                                                                  
        Hint: Run start-bitbucket.sh --no-search to skip starting the bundled search server                                                                                                         
could not find java in bundled jdk at <ES_HOME>                                                                                                                   
There was a problem starting the bundled search server   

To check the current environment variables configuration:

On Linux,
Run the below commands to verify the JAVA_HOME, JRE_HOME configuration

echo $JAVA_HOME
echo $JRE_HOME


On Windows,

Windows 7 – Right-click My Computer and select Properties > Advanced > Environment Variables
Windows 8 – Go to Control Panel > System > Advanced System Settings > Environment Variables
Windows 10 – Search for Environment Variables then select 'Edit the system environment variables'


To set the variables:

To set the JAVA_HOME, and JRE_HOME to the correct path, refer to

Check if search service is started with non-root user

If the Elasticsearch service fails to start:

check if the search service is started with the right user.

The search service can't be started with the root user. For more details, refer to When using the root user, Search server cannot be started due to "don't run <search server> as root"

To verify the user with which the service is started, please run:

On Linux,

service <search-service-name> status

ps -ef|grep <service-name>

On Windows,

Navigate to Windows Task Manager and verify the process information

Check if Elasticsearch service has required file permissions

check if the Elasticsearch service has required file permissions

The user with which the search service is running requires read and write permissions to ES_HOME (Elasticsearch HOME DIR). The default location for Bundled Elasticsearch is $BITBUCKET_HOME/shared/search.

Check network connectivity between Bitbucket nodes and remote Elasticsearch works

Verify network connectivity

To check if the connectivity between Bitbucket and remote Elasticsearch works as expected, run the below curl command from Bitbucket instance

curl -v <elasticsearch-url>

The curl output returns 200 if Bitbucket successfully connects to Elasticsearch

(info) For more details, refer to Troubleshooting Connection Issues for Elasticsearch

Test search service authentication

The Elasticsearch service is up but search functionality is unavailable

Test service authentication by navigating to Administration :gear: > Server Settings > Search > Test in Bitbucket application.

If the authentication fails, refer to Test button in Search server for Bitbucket server results in the access being denied to resolve the problem.


Troubleshooting bundled Elasticsearch

Startup issues

The below KBs can be referred to fix bundled Elasticsearch and OpenSearch startup issues post-Bitbucket upgrade.

Troubleshooting Indexing issues

Please refer to the following KBs to resolve indexing problems.

SSL & connection troubleshooting

Please refer to the following KBs for resolve SSL and connectivity problems.

Known Bugs

Troubleshooting Remote Elasticsearch

A series of troubleshooting solutions can be found below to help fix problems that a remote Elasticsearch deployment might encounter.


OpenSearch

OpenSearch is an open-source search and analytics engine derived from Elasticsearch (fork of Elastic Search version 7.10.2).

Starting with Bitbucket version 7.21, OpenSearch is bundled with Bitbucket instead of Elasticsearch. Note that if you are upgrading to Bitbucket 7.21 and using bundled Elasticsearch, it will be automatically upgraded to bundled OpenSearch 1.2.4 and result in a full search re-index.

Customers running their own external search server have the option to either continue using Elasticsearch or alternatively use OpenSearch.


OpenSearch FAQ

How can I install remote OpenSearch service?

To install single node remote OpenSearch, you can refer to Install and configure a remote OpenSearch server

:info: Before proceeding with the installation of your search engine, check the Supported platforms for compatible search versions with Bitbucket.
:warning: Installing an incompatible search version with Bitbucket might cause issues with search functionality.

Where can I find OpenSearch service logs?

Bundled OpenSearch:

  • logs location: ${BITBUCKET_HOME}/log/search/

  • logs pattern: bitbucket_search.log or bitbucket_search_%date%.log

  • data directory (where indexes are stored): ${BITBUCKET_HOME}/shared/search/data
    :info: The above locations are configured in $BITBUCKET_HOME/shared/search/config/opensearch.yml

Remote OpenSearch:
Verify the log location in $OPENSEARCH_HOME/config/opensearch.yml
For more details, refer to https://opensearch.org/docs/latest/monitoring-your-cluster/logs/

What versions of OpenSearch are supported with Bitbucket?

Please refer to Bitbucket Supported platforms for supported OpenSearch versions.

How to resolve common issues with OpenSearch?

To resolve some common issues with OpenSearch, refer to https://opensearch.org/docs/2.9/troubleshoot/index/ from OpenSearch documentation.

How to Generate OpenSearch certificates if they were not automatically created

Check that the following files are present under $BITBUCKET_HOME/shared/search/config:

$ ls -lrt *.pem
-rw------- 1 atlbitbucket atlbitbucket  1009 Mar  3 09:12 root-ca.pem
-rw------- 1 atlbitbucket atlbitbucket   981 Mar  3 09:12 bitbucket.pem
-rw------- 1 atlbitbucket atlbitbucket  1704 Mar  3 09:12 bitbucket-key.pem

There may be some cases where those files were not generated during an upgrade/installation process. In such cases, those files needs to be manually created.

To do that, just execute the following steps (making sure that you are still located at $BITBUCKET_HOME/shared/search/config):


# Creating the Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/CN=BITBUCKET" -out root-ca.pem -days 3650
# Creating OpenSearch certs
openssl genrsa -out bitbucket-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in bitbucket-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out bitbucket-key.pem
openssl req -new -key bitbucket-key.pem -subj "/CN=BITBUCKET" -out bitbucket.csr
openssl x509 -req -in bitbucket.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha512 -out bitbucket.pem -days 3650
# Cleanup
rm bitbucket-key-temp.pem
rm bitbucket.csr
rm root-ca-key.pem
rm root-ca.srl

Once those files have been created, start Bitbucket again and make sure that you are now able to do a code search. You can also test it:

  • Go to the Administration area (Gear Icon at the top bar)
  • Click on Server Settings
  • Under the Search area click on Test.

Case an "Access Denied" is displayed, we need to reset the password on the Search server. Follow the procedures from "Test button in Search server for Bitbucket server results in the access denied" to resolve that issue.


OpenSearch Troubleshooting

In many cases, we noticed the search functionality doesn’t work as expected after an upgrade of Bitbucket / OpenSearch version or migration of Bitbucket from one node to another. The possible reasons may be the OpenSearch service startup failure, connectivity problems with Bitbucket or indexing issues.

The below checklist can be referred to identify the problem.

Checklist

Check if OpenSearch service is started

Run the below commands to verify if the search service is running

On Linux,

ps -ef | grep opensearch


On Windows,

Navigate to Windows Task Manager > Details to verify if the search service is running.

(info) If the service is not running, try to manually start the service.

Check the Search server and Bitbucket Server logs for error messages

You might have some obvious log messages that could help you address the issue.

These are logs you should check:

  • $BITBUCKET_HOME/log/atlassian-bitbucket.log
  • $BITBUCKET_HOME/log/search/bitbucket_search.log
Check version compatibility with Bitbucket

Bundled OpenSearch:

If the OpenSearch service fails to start, check version compatibility with Bitbucket

Check if the OpenSearch version and Java version are compatible with Bitbucket. Refer to Supported platforms

Remote OpenSearch:

The search service starts but the functionality may not work as expected in case of incompatible OpenSearch versions.

Check if environment variables (JRE_HOME, JAVA_HOME) are set correctly

If the OpenSearch service fails to start, check if environment variables are set correctly

Bundled OpenSearch:

OpenSearch fails to start if the service can't find java i.e if JAVA_HOME, JRE_HOME environment variables are not set properly.

To verify if environment variables are set:

On Linux,
Run the below commands to verify the JAVA_HOME, JRE_HOME configuration

echo $JAVA_HOME
echo $JRE_HOME


On Windows,

Windows 7 – Right-click My Computer and select Properties > Advanced > Environment Variables
Windows 8 – Go to Control Panel > System > Advanced System Settings > Environment Variables
Windows 10 – Search for Environment Variables then select Edit the system environment variables

Starting 7.21.1, Bitbucket installer will always install bundled JRE for Bitbucket and bundled search. The bundled JRE is used by default or can be customised by setting the JRE_HOME.

Remote OpenSearch:
Set OPENSEARCH_JAVA_HOME or JAVA_HOME environment variable to the right Java installation directory. The supported Java versions with OpenSearch:


To set the variables:

To set the JAVA_HOME, and JRE_HOME to the correct path, refer to

Check if OpenSearch service is started with non-root user and has required permissions

If the OpenSearch service fails to start, check if the OpenSearch service has required file permissions

OpenSearch service can't be started as root user

[root@xxx bin]# uncaught exception in thread [main]                                                                                                                                   
java.lang.RuntimeException: can not run opensearch as root                                                                                                                                          
        at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:124)                                                                                                                 
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191)                                                                                                                             
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)                                                                                                                              
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)                                                                                                                            
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)                                                                                                                         
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)                                                                                                     
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)                                                                                                                    
        at org.opensearch.cli.Command.main(Command.java:101)                                                                                                                                        
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)                                                                                                                            
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)                                                                                                                            
For complete error details, refer to the log at /data/atlbitbucket-home/log/search/bitbucket_search.log
root@xxx:/opt/atlassian/bitbucket/bin# Exception in thread "main" java.lang.RuntimeException: don't run opensearch as root.

Verify if the search service is started with the right user. This can be validated by checking the username and group details present over the server directories or files.

Bundled OpenSearch:

Directory for Search home path: <Bitbucket_Installation>/current/opensearch

Directory for search config directory: <Bitbucket_Home>/shared/search/config

Remote OpenSerach:

Directory content in which Remote OpenSearch installed or tar content has been extracted:

Directory content of <OpenSearch_Untar_Directory>/bin from where OpenSearch can be started
To start OpenSearch → Execute → ./opensearch

Refer to When using the root user, Search server cannot be started due to "don't run <search server> as root" for more details

Check network connectivity between Bitbucket and OpenSearch

To check the connectivity between Bitbucket and remote OpenSearch, run the below curl command from the Bitbucket instance

Bundled OpenSearch:

curl -v <opensearch-url>

Check if we can connect to port 7992

Either browse following URL in the Browser -> http://localhost:7992
Or
curl -u username:password http://localhost:7992


Remote OpenSearch:

Check the status of the Search Server

curl -v <opensearch-url>
# For example
curl -v -u username:password <Server_Server_URL>:9200

The result should be similar to:

Test search service authentication

The OpenSearch service is up but search functionality is unavailable, check for service authentication
By navigating to Administration :gear: > Server Settings > Search > Test in Bitbucket application.

If the authentication fails, refer to How to reset the bundled OpenSearch password post first start in Bitbucket Server


Troubleshooting bundled OpenSearch

Startup issues:


Indexing Issues

SSL & connection troubleshooting

Known Bugs


Troubleshooting remote OpenSearch

A series of troubleshooting solutions can be found below to help fix problems that a remote OpenSearch deployment might encounter.


For further troubleshooting, please contact Atlassian support


Last modified on Dec 20, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.