"Access Denied" when testing connection to search server in Bitbucket Data Center

Related content

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

During the installation of Bitbucket, Bitbucket will hash a random password, which will be used between Bitbucket and bundled OpenSearch for authentication. This password will be stored in the Bitbucket database and OpenSearch security index. 

Due to the nature of the provisioning, the password is unknown. If this goes out of sync between Bitbucket and bundled OpenSearch, we will need to set the password for bundled OpenSearch manually.

Environment

  • Bitbucket 8 with bundled OpenSearch v1
  • Bitbucket 9 with bundled OpenSearch v2

Diagnosis

  1. Navigate to Administration > Server Settings.
  2. Click on the Test button under the Search.
  3. The result would be "Access to the search server was denied. Check your settings.".
  4. During startup, Bitbucket application log; $BITBUCKET_HOME/log/atlassian-bitbucket.log , shows the following error messages: 
    2025-03-04 03:14:16,621 INFO  [Caesium-1-1]  c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2025-03-04 03:14:17,793 ERROR [Caesium-1-1]  c.a.b.i.s.i.IndexingSynchronizationService Failed to create indexes: ''[Unsuccessful search response: 401. Bitbucket did not have authorization to request data from the search server.]''
2025-03-04 03:14:17,793 ERROR [Caesium-1-1]  c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in the search server
2025-03-04 03:14:17,794 WARN  [Caesium-1-1]  c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob  Job id: c2ffe4a6-5d69-4e4e-8d6b-fda3afd54010
...
  5. Bundled OpenSearch log; $BITBUCKET_HOME/log/search/bitbucket_search.log , show authentication warnings: 
    [2025-03-04T03:14:17,384][WARN ][o.o.s.a.BackendRegistry  ] [bitbucket_bundled] Authentication finally failed for bitbucket from 127.0.0.1:57880
[2025-03-04T03:14:17,774][WARN ][o.o.s.a.BackendRegistry  ] [bitbucket_bundled] Authentication finally failed for bitbucket from 127.0.0.1:57880

Cause

The password expected by the Search server is out of sync with the one that Bitbucket uses to authenticate.

Solution

INFO

  • Please make sure to have a backup of the $BITBUCKET_HOME/shared/search before proceeding.
  • Please replace the $BITBUCKET_INSTALL and $BITBUCKET_INSTALL accordingly.
  • For Bitbucket 8 (bundled with OpenSearch v1), the path is 
    $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
    instead of 
    $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml
    This is due to the changes in OpenSearch v2.
  1. Log in to the Bitbucket machine (SSH and any remote connection).
  2. Stop Bitbucket.
  3. Hash a new password by running the following command: 
    # /bin/bash +x $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/tools/hash.sh
WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java
[Password:] <qwerty>
$2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC
  4. Update the hash value in $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml. 
    # vi $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml
# cat $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml
_meta:
  type: "internalusers"
  config_version: 2
bitbucket:
  # Default "bitbucket-changeit" password which will be updated by automatic provisioning
  hash: "$2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC"
  backend_roles:
    - "admin"
  description: "Admin user"
  5. Update $BITBUCKET_HOME/shared/search/config/opensearch.yml (Skip this step for Bitbucket 8). 
    # vi $BITBUCKET_HOME/shared/search/config/opensearch.yml
# cat $BITBUCKET_HOME/shared/search/config/opensearch.yml
<existing content>
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: bitbucket.pem
plugins.security.ssl.http.pemkey_filepath: bitbucket-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: false

  6. Start Bitbucket.

  7. Run securityadmin.sh. 
    # /bin/bash +x $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/tools/securityadmin.sh --port 7992 \
  -i .bitbucket_security_index_1 \
  -f $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml \
  -t internalusers \
  -icl \
  -nhnv \
  -cacert $BITBUCKET_HOME/shared/search/config/root-ca.pem \
  -cert $BITBUCKET_HOME/shared/search/config/bitbucket.pem \
  -key $BITBUCKET_HOME/shared/search/config/bitbucket-key.pem
WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java
Security Admin v7
Will connect to localhost:7992 ... done
Connected as "CN=BITBUCKET"
OpenSearch Version: 2.18.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: bitbucket_search
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.bitbucket_security_index_1 index already exists, so we do not need to create one.
Populate config from /root
Force type: internalusers
Will update '/internalusers' with $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
SUCC: Expected 1 config types for node {"updated_config_types":["internalusers"],"updated_config_size":1,"message":null} is 1 (["internalusers"]) due to: null
Done with success
    (info) For Bitbucket 8, please use --port 7993  instead.
  8. For Bitbucket 8, test and update the search password in the Server Settings page. For Bitbucket 9, please proceed to the next step.
  9. Stop Bitbucket.
  10. Revert the changes on the $BITBUCKET_HOME/shared/search/config/opensearch.yml.
  11. Start Bitbucket.
  12. Test and update the search password in the Server Settings page.

Workaround (with data loss)

WARNING

  • The following step involves deleting all the bundled OpenSearch dat,a which means you will lose all the index and search will be unavailable in Bitbucket while bundled OpenSearch is reindexing.
  • There is no definitive estimation for the reindexing duration.
  1. Log in to the Bitbucket machine (SSH and any remote connection).
  2. Stop Bitbucket.
  3. Hash a new password by running the following command: 
    # /bin/bash +x $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/tools/hash.sh
WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java
[Password:] <qwerty>
$2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC
  4. Update the hash value in $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml. 
    # vi $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml
# cat $BITBUCKET_HOME/shared/search/config/opensearch-security/internal_users.yml
_meta:
  type: "internalusers"
  config_version: 2
bitbucket:
  # Default "bitbucket-changeit" password which will be updated by automatic provisioning
  hash: "$2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC"
  backend_roles:
    - "admin"
  description: "Admin user"
  5. Delete all the contents of $BITBUCKET_HOME/shared/search. 
    # rm -rf $BITBUCKET_HOME/shared/search
  6. Start Bitbucket.
  7. Test and update the search password in the Server Settings page.


Unsupported platforms

INFO

  • The following steps involve platforms that are no longer supported.
  • These steps are not maintained by us, so please use them with caution.

For Elasticsearch

Updating the credentials on the buckler.yml and restarting the Elasticsearch service should fix this issue.

  1. Shutdown Elasticsearch and Bitbucket Server

    # service atlbitbucket stop

  2. Edit the buckler.yml file. Back it up first

    # vi /var/atlassian/application-data/bitbucket/shared/search/buckler/buckler.yml
  3. Right after the text "auth.basic.password:" remove the current default password and replace it with any password of your choice.

  4. Next, start up Bitbucket Server and ensure you're able to access Elasticsearch at http://localhost:7992 in the browser.

    # service atlbitbucket start
    • Perform a test and enter the credentials on the pop-up dialog with the updated credentials to see if you're able to log in.
  5. If you're able to log in, attempt to test with the same credentials from the Bitbucket Server Administration area
  6. If you are not able to log in:
    1. Navigate to Administration  > Server Settings > Search 
    2. Paste the chosen password from Step 3 into the "Search server password" field 

For OpenSearch on Windows

Step 1: Create a new password hash

Passwords are stored in a hash format in a file named internal_users.yml file. As the name suggests this file stores user credentials including the admin user. Therefore, the first step is to create the hash. Opensearch provides a script to create a password hash.

  1. For Windows, you have to enter into the <Bitbucket-installation-directory>\opensearch\plugins\opensearch-security\tools> and run the hash.bat file
cd C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0\opensearch\plugins\opensearch-security\tools>
C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0\opensearch\plugins\opensearch-security\tools> .\hash.bat
[Password:]
$2y$12$zjrg.uQxhyOAbpZ1WPBCyeCjPB60K5k3U/p9PmxlllfI8vBgahhga

You will be prompted to enter a password and outputs its hash. You can use bitbucket-changeit for example. Copy the output as we are going to store it in internal_users.yml

Here C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0 is our installation directory for Bitbucket 7.21.0.

Step 2: Update internal_users.yml
  1. Go to <Bitbucket-installation-directory>\opensearch\plugins\opensearch-security\securityconfig
  2. Open the internal_users.yml file in a text editor
  3. Find the “admin” user section and update the hash value with what you got in the previous step:
_meta:
  type: "internalusers"
  config_version: 2

bitbucket:
  # Default "bitbucket-changeit" password which will be updated by automatic provisioning
  hash: "$2y$12$zjrg.uQxhyOAbpZ1WPBCyeCjPB60K5k3U/p9PmxlllfI8vBgahhga"
  backend_roles:
    - "admin"
  description: "Admin user"
Step 3: Apply security changes
  1. Run the below command with your custom paths for Bitbucket home and installation directory. In the below command, the home path is D:\bitbucket-home\ and the installation directory path is C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0.
  2. Open a Powershell Window or Command Prompt in the C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0\opensearch\plugins\opensearch-security\tools and run the below command:
.\securityadmin.bat --port 7993 -f C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0\opensearch\plugins\opensearch-security\securityconfig\internal_users.yml -t internalusers -icl -nhnv -cacert D:\bitbucket-home\shared\search\config\root-ca.pem -cert D:\bitbucket-home\shared\search\config\bitbucket.pem -key D:\bitbucket-home\shared\search\config\bitbucket-key.pem

       3. This is the output you will get:

Security Admin v7
WARNING: Seems you want connect to the OpenSearch HTTP port.
         securityadmin connects on the transport port which is normally 9300.
Will connect to localhost:7993 ... done
Connected as CN=BITBUCKET
OpenSearch Version: 1.2.4
OpenSearch Security Version: 1.2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: bitbucket_search
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0\opensearch\plugins\opensearch-security\tools
Force type: internalusers
Will update '_doc/internalusers' with C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0\opensearch\plugins\opensearch-security\securityconfig\internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Done with success

 Lastly, attempt to test with the same credentials from the Bitbucket Server Administration area under Search




Last modified on Mar 6, 2025

Was this helpful?

Yes
No
Provide feedback about this article

Related content

Powered by Confluence and Scroll Viewport.