Code Search Troubleshooting Guide for Bitbucket Server and DataCenter
Related content
- No related content found
Bitbucket Data Center and Server allow you to search through the code to find exactly what you’re looking for, right from the search bar using Elasticsearch / OpenSearch. Bitbucket Data Center requires a remote Elasticsearch server / OpenSearch server for multi-node Data Center deployments. The Data Center can have only one remote connection to a search service for your cluster. This may be a standalone search service installation or a clustered installation behind a load balancer. For more details on code search, refer to Administer code search
General FAQ
If you're installing a single-node Bitbucket Data Center, you can use the bundled search server. The bundled search gets automatically configured with its scripts out of the box.
If you’re installing a multi-node Bitbucket Data Center, you can't use a bundled search server. For a clustered Data Center (Data Center installation type), you would need to install it separately on a remote server.
Yes, however, Bitbucket Data Center can have only one remote connection to the shared search server for your cluster. This may be a search server installation or a clustered installation behind a load balancer. For guidance on using a clustered search server installation with Bitbucket Data Center, see the page Use a clustered search server with Bitbucket Data Center.
No. The versions of Elasticsearch above a certain version (7.10) may require the purchase of a separate license from Elastic. You may need to contact Elastic for more information.
OpenSearch is open source and should not require a license.
From https://opensearch.org/faq/
All of the software in the OpenSearch project is released under the Apache License, Version 2.0 (ALv2). The ALv2 license grants you well-understood usage rights for OpenSearch. You can use, modify, extend, embed, monetize, resell, and offer OpenSearch as part of your products and services. We have also published permissive usage guidelines for the OpenSearch trademark, so you can use the name to promote your offerings.
No. You can run a Bitbucket Server / Data Center without a search server. Bitbucket itself will run as expected, but some features (such as code search, advanced repository management etc.) will not work.
If you need to turn off code search for all of your Bitbucket users, you can disable code search entirely from within the bitbucket.properties file. See the page Bitbucket Server config properties - Search for details.
Your search server was configured using the bitbucket.properties file. If a parameter is set in the bitbucket.properties file, it can't be edited later from the admin UI. Any changes that need to be made to the search server configuration must be made within the bitbucket.properties file.
:info: The changes to the bitbucket.properties file will take effect only after the restart of all the nodes in Bitbucket Data Center.
Following are the parameter which need to be set in bitbucket.properties file.
Property | Parameter name for properties file |
---|---|
URL | plugin.search.config.baseurl |
Username | plugin.search.config.username |
Password | plugin.search.config.password |
You can configure the search indexes to exclude forked repositories, and personal repositories, which can save disk space when using a fork-based workflow for development. See the page Bitbucket Server config properties - Search for the options available to change what is indexed for code search.
:info: Note that excluding content from being indexed can prevent such content from ever appearing in code search results, but will not prevent the repositories themselves from appearing in search results.
There are a few ways the search server may not have been configured properly. To troubleshoot the bundled search server that comes with Bitbucket Server, the page Troubleshooting steps for Bitbucket Server Code Search is a good place to start. If you're still stuck, you could post a question in the Atlassian Community (for a Starter license or if your configuration is unique), or contact Atlassian Support for assistance (you'll need to log in to raise a support request).
Atlassian strongly recommends you secure access to your remote search server with a username and password, and a minimum of basic HTTP authentication. See Secure your search server for details.
If additional troubleshooting is required and the debug log level of the Bitbucket Server to Search server is required, the following command will enable it:
curl -u <ADMIN_USERNAME> -v -X PUT -d "" -H "Content-Type: application/json" <BASE_URL>/rest/api/latest/logs/logger/com.atlassian.bitbucket.search.internal.indexing/debug
The additional log lines will be added to the atlassian-bitbucket.log
file. Refer to the Enable debug logging page for more information and alternative methods to enable debug logging.
You can see the index progress by navigating to below URL:
<BITBUCKET_BASEURL>/rest/indexing/latest/status
Example results looks like the following:
{
"queues": {
"delay": 0,
"event": 0
},
"status": "IDLE"
}
Result Definitions:
- delay = Minutes in which the index queue is delayed. Either by failures, retries, or other performance reasons
- event = Number of events triggered for index. There are ~2 events per repository. Existing event queue size for indexing is 50k per node.
This provides a look into the current queue. If for some reason there's an event that fails to queue, or there's an error in indexing, it'll be placed onto a queue that delays by 5 minutes. When it sits at zero for a while, we can be confident that indexing is complete (for now). This is a moving target as new events coming in will be added to the queue, but is a good guide for when indexing is done. Overall, as long as search is working, that delayed queue will eventually complete and the status will change to idle.
Elasticsearch
Elasticsearch is a supported search server distribution for Bitbucket Data Center besides OpenSearch. Bitbucket 8. x will no longer support Elasticsearch 5. x, Elasticsearch 6. x, or Elasticsearch versions lower than 7.10.2 due to licensing change. Customers running an external search server will need to use a supported version of Elasticsearch 7.10.2 or higher, or OpenSearch 1.2.4 or higher with Bitbucket 8.0. Refer to Supported platforms for compatible search versions with Bitbucket. Note that, starting Bitbucket 7.21, OpenSearch is bundled with Bitbucket for Elasticsearch due to a change in Elastic Licensing.
The support for Elasticsearch will be deprecated and removed in Bitbucket 9.0. In Bitbucket 9.0 and later, Opensearch will be the only supported search service.
Elasticsearch FAQ
:info: Before proceeding with the installation of your search engine, check the Supported platforms for compatibility with Bitbucket version.
:warning: Installing an incompatible search version with Bitbucket might cause issues with search functionality.
To setup and install single node remote Elasticsearch, you can refer to Install and configure a remote Elasticsearch server
Bundled Elasticsearch
log location: $BITBUCKET_HOME/log/search
Data directory where the indexes are stored: $BITBUCKET_HOME/shared/search/data
Remote Elasticsearch:
Verify the log location in $ES_HOME/config/elasticsearch.yml.
You can configure the log location as per the article https://discuss.elastic.co/t/where-does-elasticsearch-store-logs/200466
You might have some obvious log messages that could help you address the issue.
These are logs you should check:
$BITBUCKET_HOME/log/atlassian-bitbucket.log
$BITBUCKET_HOME/log/search/bitbucket_search.log
A good place to start is Elastic documentation. A few references are below:
Security issues for Elasticsearch can be found in the below links
Bitbucket Elasticsearch CVE-related information can be found in the security advisories below
When searching for a word which happens to be present in more than 1000 files in all of Bitbucket repositories, results can be seen for only the first 1000. Subsequent results are not fetched and we are instead presented with the following alert: “The server couldn’t process the request. Please try again later“
You can configure pagination settings by referring to the article How to increase Pagination in Elasticsearch since every search is limited to 1000 results
The article How to resolve common Elasticsearch issues helps to resolve Elasticsearch issues related to GC overhead, Low max file descriptors, Low vm.max_map_count
Elasticsearch Troubleshooting
In many cases, we noticed the search functionality doesn’t work as expected after an upgrade of Bitbucket / Elasticsearch version or migration of Bitbucket from one node to another. The possible reasons may be the Elasticsearch service startup failure, connectivity problems with Bitbucket or indexing issues.
The below checklist can be referred to identify the problem.
Checklist
Run the below command on Bitbucket server (if you are using a bundled search server)
On Linux,
ps -ef | grep search
For the remote Elasticsearch server, run the command on the remote search server
On Windows,
Navigate to Windows Task Manager > Details to verify if the search service is running.
:info: If the search service is down, start the service. Refer to Bundled Elasticsearch/OpenSearch does not start after installing or upgrading Bitbucket Data Center using binaries
Bundled Elasticsearch:
If the Elasticsearch service fails to start, check version compatibility with Bitbucket
Check if the Elasticsearch version is compatible with Bitbucket. Refer to Supported platforms
Remote Elasticsearch:
The search service starts but the functionality may not work as expected in case of incompatible Elasticsearch versions.
If the Elasticsearch service fails to start:
check if environment variables are set correctly
The below error can be seen when the Elasticsearch startup fails as it could not find Java. This can happen if the JAVA_HOME, and JRE_HOME environment variables are not set correctly.
Starting the bundled search server
Hint: Run start-bitbucket.sh --no-search to skip starting the bundled search server
could not find java in bundled jdk at <ES_HOME>
There was a problem starting the bundled search server
To check the current environment variables configuration:
On Linux,
Run the below commands to verify the JAVA_HOME, JRE_HOME configuration
echo $JAVA_HOME
echo $JRE_HOME
On Windows,
Windows 7 – Right-click My Computer and select Properties > Advanced > Environment Variables
Windows 8 – Go to Control Panel > System > Advanced System Settings > Environment Variables
Windows 10 – Search for Environment Variables then select 'Edit the system environment variables'
To set the variables:
To set the JAVA_HOME, and JRE_HOME to the correct path, refer to
Setting JAVA_HOME for Linux
Setting the JAVA_HOME Variable in Windows for Windows
If the Elasticsearch service fails to start:
check if the search service is started with the right user.
The search service can't be started with the root user. For more details, refer to When using the root user, Search server cannot be started due to "don't run <search server> as root"
To verify the user with which the service is started, please run:
On Linux,
service <search-service-name> status
ps -ef|grep <service-name>
On Windows,
Navigate to Windows Task Manager and verify the process information
check if the Elasticsearch service has required file permissions
The user with which the search service is running requires read and write permissions to ES_HOME (Elasticsearch HOME DIR). The default location for Bundled Elasticsearch is $BITBUCKET_HOME/shared/search.
Verify network connectivity
To check if the connectivity between Bitbucket and remote Elasticsearch works as expected, run the below curl command from Bitbucket instance
curl -v <elasticsearch-url>
The curl output returns 200 if Bitbucket successfully connects to Elasticsearch
For more details, refer to Troubleshooting Connection Issues for Elasticsearch
The Elasticsearch service is up but search functionality is unavailable
Test service authentication by navigating to Administration :gear: > Server Settings > Search > Test in Bitbucket application.
If the authentication fails, refer to Test button in Search server for Bitbucket server results in the access being denied to resolve the problem.
Troubleshooting bundled Elasticsearch
Startup issues
The below KBs can be referred to fix bundled Elasticsearch and OpenSearch startup issues post-Bitbucket upgrade.
ElasticSearch fails to start (java.lang.RuntimeException: starting java failed) in Bitbucket Server
Bundled Elasticsearch throws node validation exception after Bitbucket upgrade
Bundled ElasticSearch Windows service fails on switching to use Java 11
Troubleshooting Indexing issues
Please refer to the following KBs to resolve indexing problems.
Rebuild the Search server indexes for Bitbucket Server | Bitbucket Data Center and Server
Bitbucket Server Code Search is unable to find results from some or all repositories
Elasticsearch index stuck with Unable to enqueue event warning message
SSL & connection troubleshooting
Please refer to the following KBs for resolve SSL and connectivity problems.
Known Bugs
BSERV-12267 - Getting issue details... STATUS
BSERV-12056 - Getting issue details... STATUS
BSERV-13368 - Getting issue details... STATUS
BSERV-12519 - Getting issue details... STATUS
Troubleshooting Remote Elasticsearch
A series of troubleshooting solutions can be found below to help fix problems that a remote Elasticsearch deployment might encounter.
Several troubleshooting issues can be diagnosed using the health API.
Indices mix index allocation filters with data tiers node roles to move through data tiers
Total number of shards for an index on a single node exceeded
For more, please refer to: Elasticsearch Guide [ 7.17 ] | Elastic
OpenSearch
OpenSearch is an open-source search and analytics engine derived from Elasticsearch (fork of Elastic Search version 7.10.2).
Starting with Bitbucket version 7.21, OpenSearch is bundled with Bitbucket instead of Elasticsearch. Note that if you are upgrading to Bitbucket 7.21 and using bundled Elasticsearch, it will be automatically upgraded to bundled OpenSearch 1.2.4 and result in a full search re-index.
Customers running their own external search server have the option to either continue using Elasticsearch or alternatively use OpenSearch.
OpenSearch FAQ
To install single node remote OpenSearch, you can refer to Install and configure a remote OpenSearch server
:info: Before proceeding with the installation of your search engine, check the Supported platforms for compatible search versions with Bitbucket.
:warning: Installing an incompatible search version with Bitbucket might cause issues with search functionality.
Bundled OpenSearch:
logs location: ${BITBUCKET_HOME}/log/search/
logs pattern: bitbucket_search.log or bitbucket_search_%date%.log
data directory (where indexes are stored): ${BITBUCKET_HOME}/shared/search/data
:info: The above locations are configured in $BITBUCKET_HOME/shared/search/config/opensearch.yml
Remote OpenSearch:
Verify the log location in $OPENSEARCH_HOME/config/opensearch.yml
For more details, refer to https://opensearch.org/docs/latest/monitoring-your-cluster/logs/
To secure the OpenSearch server, refer to How to configure HTTPS for remote OpenSearch instance in Bitbucket Data Center
The below links from OpenSearch documentation can give information about CVE
Please refer to Bitbucket Supported platforms for supported OpenSearch versions.
A good place to start is https://opensearch.org/docs/latest/.
For configuration settings, refer to https://opensearch.org/docs/2.9/install-and-configure/install-opensearch/index/#important-settings
To resolve some common issues with OpenSearch, refer to https://opensearch.org/docs/2.9/troubleshoot/index/ from OpenSearch documentation.
Check that the following files are present under $BITBUCKET_HOME/shared/search/config
:
$ ls -lrt *.pem
-rw------- 1 atlbitbucket atlbitbucket 1009 Mar 3 09:12 root-ca.pem
-rw------- 1 atlbitbucket atlbitbucket 981 Mar 3 09:12 bitbucket.pem
-rw------- 1 atlbitbucket atlbitbucket 1704 Mar 3 09:12 bitbucket-key.pem
There may be some cases where those files were not generated during an upgrade/installation process. In such cases, those files needs to be manually created.
To do that, just execute the following steps (making sure that you are still located at $BITBUCKET_HOME/shared/search/config
):
# Creating the Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/CN=BITBUCKET" -out root-ca.pem -days 3650
# Creating OpenSearch certs
openssl genrsa -out bitbucket-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in bitbucket-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out bitbucket-key.pem
openssl req -new -key bitbucket-key.pem -subj "/CN=BITBUCKET" -out bitbucket.csr
openssl x509 -req -in bitbucket.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha512 -out bitbucket.pem -days 3650
# Cleanup
rm bitbucket-key-temp.pem
rm bitbucket.csr
rm root-ca-key.pem
rm root-ca.srl
Once those files have been created, start Bitbucket again and make sure that you are now able to do a code search. You can also test it:
- Go to the Administration area (Gear Icon at the top bar)
- Click on Server Settings
- Under the Search area click on Test.
Case an "Access Denied" is displayed, we need to reset the password on the Search server. Follow the procedures from "Test button in Search server for Bitbucket server results in the access denied" to resolve that issue.
OpenSearch Troubleshooting
In many cases, we noticed the search functionality doesn’t work as expected after an upgrade of Bitbucket / OpenSearch version or migration of Bitbucket from one node to another. The possible reasons may be the OpenSearch service startup failure, connectivity problems with Bitbucket or indexing issues.
The below checklist can be referred to identify the problem.
Checklist
Run the below commands to verify if the search service is running
On Linux,
ps -ef | grep opensearch
On Windows,
Navigate to Windows Task Manager > Details to verify if the search service is running.
If the service is not running, try to manually start the service.
You might have some obvious log messages that could help you address the issue.
These are logs you should check:
$BITBUCKET_HOME/log/atlassian-bitbucket.log
$BITBUCKET_HOME/log/search/bitbucket_search.log
Bundled OpenSearch:
If the OpenSearch service fails to start, check version compatibility with Bitbucket
Check if the OpenSearch version and Java version are compatible with Bitbucket. Refer to Supported platforms
Remote OpenSearch:
The search service starts but the functionality may not work as expected in case of incompatible OpenSearch versions.
If the OpenSearch service fails to start, check if environment variables are set correctly
Bundled OpenSearch:
OpenSearch fails to start if the service can't find java i.e if JAVA_HOME, JRE_HOME environment variables are not set properly.
To verify if environment variables are set:
On Linux,
Run the below commands to verify the JAVA_HOME, JRE_HOME configuration
echo $JAVA_HOME
echo $JRE_HOME
On Windows,
Windows 7 – Right-click My Computer and select Properties > Advanced > Environment Variables
Windows 8 – Go to Control Panel > System > Advanced System Settings > Environment Variables
Windows 10 – Search for Environment Variables then select Edit the system environment variables
Starting 7.21.1, Bitbucket installer will always install bundled JRE for Bitbucket and bundled search. The bundled JRE is used by default or can be customised by setting the JRE_HOME.
Remote OpenSearch:
Set OPENSEARCH_JAVA_HOME or JAVA_HOME environment variable to the right Java installation directory. The supported Java versions with OpenSearch:
To set the variables:
To set the JAVA_HOME, and JRE_HOME to the correct path, refer to
Setting JAVA_HOME for Linux
Setting the JAVA_HOME Variable in Windows for Windows
If the OpenSearch service fails to start, check if the OpenSearch service has required file permissions
OpenSearch service can't be started as root user
[root@xxx bin]# uncaught exception in thread [main]
java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:124)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
For complete error details, refer to the log at /data/atlbitbucket-home/log/search/bitbucket_search.log
root@xxx:/opt/atlassian/bitbucket/bin# Exception in thread "main" java.lang.RuntimeException: don't run opensearch as root.
Verify if the search service is started with the right user. This can be validated by checking the username and group details present over the server directories or files.
Bundled OpenSearch:
Directory for Search home path: <Bitbucket_Installation>/current/opensearch
Directory for search config directory: <Bitbucket_Home>/shared/search/config
Remote OpenSerach:
Directory content in which Remote OpenSearch installed or tar content has been extracted:
Directory content of <OpenSearch_Untar_Directory>/bin from where OpenSearch can be started
To start OpenSearch → Execute → ./opensearch
Refer to When using the root user, Search server cannot be started due to "don't run <search server> as root" for more details
To check the connectivity between Bitbucket and remote OpenSearch, run the below curl command from the Bitbucket instance
Bundled OpenSearch:
curl -v <opensearch-url>
Check if we can connect to port 7992
Either browse following URL in the Browser -> http://localhost:7992
Or
curl -u username:password http://localhost:7992
Remote OpenSearch:
Check the status of the Search Server
curl -v <opensearch-url>
# For example
curl -v -u username:password <Server_Server_URL>:9200
The result should be similar to:
The OpenSearch service is up but search functionality is unavailable, check for service authentication
By navigating to Administration :gear: > Server Settings > Search > Test in Bitbucket application.
If the authentication fails, refer to How to reset the bundled OpenSearch password post first start in Bitbucket Server
Troubleshooting bundled OpenSearch
Startup issues:
Please refer to the article links mentioned in the section Troubleshooting Elasticsearch startup issues
Indexing Issues
For specific repository corruption: Reindex a specific repository for code search
For some or all repositories corruption: Bitbucket Server Code Search is unable to find results from some or all repositories
Rebuild Search Server index: Rebuild the Search server indexes for Bitbucket Server
SSL & connection troubleshooting
For SSL and connectivity issues: Troubleshooting steps for Bitbucket Server Code Search
Known Bugs
- BSERV-13648 - Getting issue details... STATUS
- BSERV-13328 - Getting issue details... STATUS
- BSERV-13302 - Getting issue details... STATUS
Troubleshooting remote OpenSearch
A series of troubleshooting solutions can be found below to help fix problems that a remote OpenSearch deployment might encounter.
Common issues related to OpenSearch https://opensearch.org/docs/latest/troubleshoot/index/
For TLS issues https://opensearch.org/docs/latest/troubleshoot/tls/
For further troubleshooting, please contact Atlassian support
Related content
- No related content found