Troubleshooting steps for Bitbucket Server Code Search

Still need help?

The Atlassian Community is here for you.

Ask the community

Purpose

This page contains the troubleshooting steps that can be performed to understand the reasons leading to the Bitbucket Server Code Search functionality not being available.

The steps below apply for the Embedded search server. If you have "Bitbucket Data Center" it means that you do not have Embedded search server but your own self-managed search server instance and therefore some of the troubleshooting steps below might not apply to your environment.

For Bitbucket 7.20 and below, Elasticsearch was bundled. Starting with 7.21, OpenSearch is the bundled search server

Troubleshooting steps

Check the status of Search server

You can see the index progress by going to:

<baseUrl>/rest/indexing/latest/status

Example results looks like the following:

{
"queues": {
"delay": 0,
"event": 0
},
"status": "IDLE"
} 

Result Definitions:

  • delay = Minutes in which the index queue is delayed. Either by failures, retries, or other performance reasons
  • event = Number of events triggered for index. There are ~2 events per repository. Existing event queue size for indexing is 50k per node.


This provides a look into the current queue. If for some reason there's an event that fails to queue, or there's an error in indexing, it'll be placed onto a queue that delays by 5 minutes. When it sits at zero for a while, we can be confident that indexing is complete (for now). This is a moving target as new events coming in will be added to the queue, but is a good guide for when indexing is done. Overall, as long as search is working, that delayed queue will eventually complete and the status will change to idle.

Check that Search server is starting with the atlbitbucket user

Ideally you want the search server to start through the services that we deploy when you set your environment up by using the installer.

On Linux, you should run the following commands as root (the service will make sure to change into the atlbitbucket user):

# service atlbitbucket status
# service atlbitbucket start
# service atlbitbucket stop

Bitbucket Server 4.14

 For Bitbucket Server 4.14 and earlier versions, it should be atlbitbucket_search service. Elasticsearch will be started as a result of starting the atlbitbucket_search service.

Check the username/password for Search server

Check if you're hitting this: Test button in Search server for Bitbucket server results in the access being denied

For Elasticsearch

Make sure that you did not update the Elasticsearch username and password in the Bitbucket Server user interface, if this is the case you'll need to reset them by following these steps:

  1. locate the $BITBUCKET_HOME/shared/search/buckler/buckler.yml file
  2. Retrieve the username (auth.basic.username attribute) and the password (auth.basic.password attribute) and update the ones available to Bitbucket Server in the user interface

If you did not update them, then there is no need to change anything.

For Opensearch

Use the steps detailed in this document: Test button in Search server for Bitbucket server results in the access being denied to change the Opensearch username/password.

Configure details of your Search server instance within Bitbucket Server

To configure the details of your search server instance within the Bitbucket Server UI, go to the Administration settings page, then click Server Settings. At the bottom of the page is where you can configure the details of your search server instance.

Important:

Another way to configure connection settings is via bitbucket.properties file (resides in $BITBUCKET_HOME/shared).

If a parameter is set in the properties file, it cannot be edited later from the admin UI. Any changes that need to be made to the search server configuration must be made within the bitbucket.properties file (which requires an application restart to take effect).

  1. In the $BITBUCKET_HOME directory, create a new directory called shared.
  2. In the $BITBUCKET_HOME/shared directory, create a text file named bitbucket.properties.

Add these Elasticsearch properties to the file.

PropertyParameter name for properties file
URLplugin.search.config.baseurl
Usernameplugin.search.config.username
Passwordplugin.search.config.password

(info) This is applicable to both Elasticsearch and Opensearch. For more details refer to Install and configure a remote Elasticsearch server and Install and configure a remote OpenSearch server

Check the Search server is writing to its own logs

In a very specific situation, we saw Elasticsearch completely hang and not write in the logs or bind to its port as defined in Elasticsearch on $BITBUCKET_HOME/shared/search/elasticsearch.yml or for Opensearch on $BITBUCKET_HOME/shared/search/opensearch.yml.

The symptoms are:

  • tail -f $BITBUCKET_HOME/log/search/bitbucket_search.log
    • Nothing happens
  • ps -ef | grep search
    • reveals search server is up and running
  • netstat -tulnp 
    • as atlbitbucket user does not show the port 7992 on listen. You can't see the PID of your search server listened
  • telnet localhost 7992
    • Connection is refused

If you run as atlbitbucket user:

For Elasticsearch

jstack $ELASTIC_SEARCH_PID > elasticsearch_threads.`date +%s`.txt

You will get the following thread dump as a result:

"main" #1 prio=5 os_prio=0 tid=0x00007f4a6800a800 nid=0x6710 runnable [0x00007f4a6e25a000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
	at sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
	at sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
	at sun.nio.fs.UnixFileStore.devFor(UnixFileStore.java:55)
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:70)
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:48)
	at sun.nio.fs.LinuxFileSystem.getFileStore(LinuxFileSystem.java:112)
	at sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(UnixFileSystem.java:213)
	at sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(UnixFileSystem.java:224)
	- locked <0x00000000c0b3c088> (a sun.nio.fs.UnixFileSystem$FileStoreIterator)
	at org.apache.lucene.util.IOUtils.getFileStore(IOUtils.java:515)
	at org.apache.lucene.util.IOUtils.spinsLinux(IOUtils.java:459)
	at org.apache.lucene.util.IOUtils.spins(IOUtils.java:448)
	at org.elasticsearch.env.ESFileStore.<init>(ESFileStore.java:57)
	at org.elasticsearch.env.Environment.<clinit>(Environment.java:90)
	at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:81)
	at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107)
	at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100)
	at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:226)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

For Opensearch

jstack $OPEN_SEARCH_PID > opensearch_threads.`date +%s`.txt

You will get the following thread dump as a result:

"opensearch[bitbucket_bundled][transport_worker][T#16]" #136 daemon prio=5 os_prio=31 tid=0x00007fc945a2c000 nid=0xae07 runnable [0x000070000521f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007da37ead8> (a sun.nio.ch.Util$3)
        - locked <0x00000007da37eaf0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007da44eba0> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:748)


According to the discussion below:

A NFS mount that is unreachable, for example, could cause such scenario. As the Elasticsearch dev explains, Elasticsearch checks all mount points on your machine during startup.

If you facing that it also means:

That a df on your system won't return the shell line; it will hang indefinitely.

Solution:

You really need to check which one is the failing mount point. After that, restarting ES works.

Enable DEBUG level logs

If additional troubleshooting is required and the debug log level of the Bitbucket Server to Search server is required, the following command will enable it:

curl -u <ADMIN_USERNAME> -v -X PUT -d "" -H "Content-Type: application/json" <BASE_URL>/rest/api/latest/logs/logger/com.atlassian.bitbucket.search.internal.indexing/debug

The additional log lines will be added to the atlassian-bitbucket.log file. Refer to the Enable debug logging page for more information and alternative methods to enable debug logging.

Check the Search server and Bitbucket Server logs for error messages

You might have some obvious log messages that could help you address the issue.

These are logs you should check:

  • $BITBUCKET_HOME/log/atlassian-bitbucket.log
  • $BITBUCKET_HOME/log/search/bitbucket_search.log
Check the service scripts for Search server are installed

For Bitbucket 5.0 versions and above, the Search server will be started as a result of starting the atlbitbucket service. Verify if the atlbitbucket has been created:

ll /etc/init.d/atlbitbucket

If the file exists, but the process is not up and running, perform the following command as root:

# service atlbitbucket start

Bitbucket Server 4.14

 For Bitbucket Server 4.14 and earlier versions, it should be atlbitbucket_search service. The Search server will be started as a result of starting the atlbitbucket_search service.

Check that the Search server configuration files are present

 Check that the following files are present under BITBUCKET_HOME/shared/search:

For Elasticsearch

├── buckler
│   └── buckler.yml
├── elasticsearch.yml
└── logging.yml

For Opensearch

$ ls
config data
$ cd config
$ ls -lrta
total 52
-rw------- 1 atlbitbucket atlbitbucket  1540 Mar  1 14:03 opensearch.yml
-rw------- 1 atlbitbucket atlbitbucket 11646 Mar  1 14:03 log4j2.properties
-rw------- 1 atlbitbucket atlbitbucket  2297 Mar  1 14:03 jvm.options
drwx------ 2 atlbitbucket atlbitbucket  4096 Mar  3 09:12 jvm.options.d
-rw------- 1 atlbitbucket atlbitbucket  1009 Mar  3 09:12 root-ca.pem
-rw------- 1 atlbitbucket atlbitbucket   981 Mar  3 09:12 bitbucket.pem
-rw------- 1 atlbitbucket atlbitbucket  1704 Mar  3 09:12 bitbucket-key.pem
-rw------- 1 atlbitbucket atlbitbucket     1 Mar  3 09:12 .version
drwxrwxr-x 4 atlbitbucket atlbitbucket  4096 Mar  3 09:12 ..
-rw-rw---- 1 atlbitbucket atlbitbucket   196 Mar  3 09:12 opensearch.keystore
drwx------ 3 atlbitbucket atlbitbucket  4096 Mar  3 09:12 .
$ cd ../data
$ ls -lrta
total 16
-rw-rw-r-- 1 atlbitbucket atlbitbucket    1 Mar  3 09:12 .version
drwxrwxr-x 4 atlbitbucket atlbitbucket 4096 Mar  3 09:12 ..
drwxr-xr-x 3 atlbitbucket atlbitbucket 4096 Mar  3 09:13 nodes
drwxrwxr-x 3 atlbitbucket atlbitbucket 4096 Mar  3 09:13 .

If these files are missing, do the following:

  1. Stop atlbitbucket service
  2. Copy the contents of $BITBUCKET_INSTALL/elasticsearch/config-template or $BITBUCKET_INSTALL/opensearch/config (based on the Search server in use) into BITBUCKET_HOME/shared/search
  3. Start the atlbitbucket service

    # service atlbitbucket status
    # service atlbitbucket start
    # service atlbitbucket stop

    Bitbucket Server 4.14

    For Bitbucket Server 4.14 and earlier versions, it should be atlbitbucket_search service. Search server will be started as a result of starting the atlbitbucket_search service.

  4. Follow the steps in the "Check the username/password for Search server" to retrieve the password and add it to the Search settings on Bitbucket UI.
Check that you can connect to port 7992 locally

The user/password required by Search server can be obtained by following the steps on the expand box above "Check the username/password for Search server".

The steps below should be run from the server where your Bitbucket Server/Search server instance are running. The reason why we reinforce it has to be a local connection is due to the configuration that we ship for Elasticsearch on $BITBUCKET_HOME/shared/search/elasticsearch.yml or for Opensearch on $BITBUCKET_HOME/shared/search/opensearch.yml.

You won't be able to browse Search server from a remote machine based on the search server configuration we ship. If you're installing your own remote Elasticsearch, you should configure the network.host: 0.0.0.0 on $Bitbucket-installation-dir/elasticsearch/<Elasticsearch_Config_dir>/elasticsearch.yml or for Opensearch on $Bitbucket-installation-dir/opensearch/<Opensearch_Config_dir>/opensearch.yml.

Please refer to Elasticsearch documentation to understand why: Special values for network.host

Click to see the configuration we ship with the bundled Elasticsearch on elasticsearch.yml...
[root@mybbs search]# cat elasticsearch.yml
cluster.name: bitbucket_search
node:
  name: bitbucket_bundled
  local: true
 
network.host: _localhost_
 
discovery.zen.ping.multicast.enabled: false
 
action.auto_create_index: false
index.mapper.dynamic: false
http.port: 7992

Please refer to Opensearch documentation to understand why: Step 3: Bind a cluster to specific IP addresses

Click to see the configuration we ship with the bundled Opensearch on opensearch.yml...
[root@mybbs /var/atlassian/application-data/bitbucket/7.21.0/shared/search]# cat opensearch.yml
cluster.name: bitbucket_search
node:
  name: bitbucket_bundled

network.host: _local_
discovery.type: single-node

path:
  logs: ${BITBUCKET_HOME}/log/search
  data: ${BITBUCKET_HOME}/shared/search/data

action.auto_create_index: false

http.port: 7992
transport.tcp.port: 7993

# The OpenSearch security plugin stores its configuration in an index in the cluster itself. On startup if the
# security index doesn't exist yet, sitting this to true will cause the security plugin to read the yml files and
# configure the index using the contents of the files.
plugins.security.allow_default_init_securityindex: true

# Using the yml files with default initialisation, we create a bitbucket user and give it the all_access in-built role.
# However, access to the REST API is disabled by default even for the all_access role so we need to explicitly give
# it permission here so that the bitbucket user can access the OpenSearch REST API.
plugins.security.restapi.roles_enabled: ["all_access"]

# Mandatory TLS setup for transport layer
plugins.security.authcz.admin_dn:
  - CN=BITBUCKET
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.pemcert_filepath: bitbucket.pem
plugins.security.ssl.transport.pemkey_filepath: bitbucket-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

# Logs audit events to bitbucket_search_server.json
plugins.security.audit.type: log4j
plugins.security.audit.config.log4j.logger_name: audit
plugins.security.audit.config.log4j.level: INFO


Access the Search server user interface at the Search server bundled URL. From a browser window running locally on the server where Bitbucket is hosted, hit the following URL: http://localhost:7992

If the following login dialog is displayed, Search server is up and running:

 


If you don't have a local browser on the server that is running Bitbucket Server, you should also be able to run the following command successfully:

$ curl -u username:password http://localhost:7992
{
  "name" : "bitbucket_bundled",
  "cluster_name" : "bitbucket_search",
  "cluster_uuid" : "s8DTMsNGRzSEaMrgo9x23Q",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.4",
    "build_type" : "tar",
    "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
    "build_date" : "2022-01-14T03:38:06.881862Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Or

$ telnet localhost 7992
Connected to localhost.
Escape character is '^]'.
 
 $ netstat -an | grep 7992


Notes:

  • As a result on your browser, you should expect to see something similar to that on your browser:
{
  "name" : "bitbucket_bundled",
  "cluster_name" : "bitbucket_search",
  "cluster_uuid" : "s8DTMsNGRzSEaMrgo9x23Q",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.4",
    "build_type" : "tar",
    "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
    "build_date" : "2022-01-14T03:38:06.881862Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
Generate OpenSearch certificates if they were not automatically created

 Check that the following files are present under $BITBUCKET_HOME/shared/search/config:

$ ls -lrt *.pem
-rw------- 1 atlbitbucket atlbitbucket  1009 Mar  3 09:12 root-ca.pem
-rw------- 1 atlbitbucket atlbitbucket   981 Mar  3 09:12 bitbucket.pem
-rw------- 1 atlbitbucket atlbitbucket  1704 Mar  3 09:12 bitbucket-key.pem

There may be some cases where those files were not generated during an upgrade/installation process. In such cases, those files needs to be manually created.

To do that, just execute the following steps (making sure that you are still located at $BITBUCKET_HOME/shared/search/config):


# Creating the Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/CN=BITBUCKET" -out root-ca.pem -days 3650
# Creating OpenSearch certs
openssl genrsa -out bitbucket-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in bitbucket-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out bitbucket-key.pem
openssl req -new -key bitbucket-key.pem -subj "/CN=BITBUCKET" -out bitbucket.csr
openssl x509 -req -in bitbucket.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha512 -out bitbucket.pem -days 3650
# Cleanup
rm bitbucket-key-temp.pem
rm bitbucket.csr
rm root-ca-key.pem
rm root-ca.srl

Once those files have been created, start Bitbucket again and make sure that you are now able to do a code search. You can also test it:

  • Go to the Administration area (Gear Icon at the top bar)
  • Click on Server Settings
  • Under the Search area click on Test.

Case an "Access Denied" is displayed, we need to reset the password on the Search server. Follow the procedures from "Test button in Search server for Bitbucket server results in the access denied" to resolve that issue.


Check the Search server process is up and running
pgrep -fl search

or its equivalent

ps -ef | grep search

This would reveal if your server is listening on port 7992 on Linux and which PID is doing it:

$ netstat -tulnp

Last modified on Dec 6, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.