Cloud
Data Center 9.3
Versions

Upgrade a Bitbucket cluster through the API without downtime

Upgrade Bitbucket without downtime

On this page

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

This document provides guidance on how to initiate and finalize a rolling upgrade through API calls. This upgrade method is suitable for admins with the skills and automation tools to orchestrate maintenance tasks (like upgrades).

tip/resting Created with Sketch.

For an overview of rolling upgrades (including planning and preparation information), see Upgrade Bitbucket without downtime.

On this page:

API reference

The entire rolling upgrade process is governed by the following API:

https://<host>:<port>/rest/zdu/cluster

This API has the following calls:

GET /rest/zdu/cluster

Get an overview of the cluster's status.
Enable upgrade mode.
Get the status of the cluster. 
Get an overview of a node's status.
Disable upgrade mode. You can only use this call if the upgrade progress is not MIXED.

Once all nodes are upgraded, finalize the rolling upgrade. This will automatically disable upgrade mode.

For detailed information about each API call, see Bitbucket REST API Documentation.

Authenticating REST API calls

In a secure environment, you'll need to authenticate your REST API calls. See Personal access tokens and Bitbucket Server REST API Example - Basic Authentication for more information.

Downloading upgrade files

Before you start the upgrade, download the right Bitbucket version first. You'll be installing this on each node. Remember, you can only upgrade to a higher bug fix version (for example, from Bitbucket 7.9.0 to 7.9.4). Download the file directly from here:

https://www.atlassian.com/software/bitbucket/download

Alternatively, you can also use the Pre-upgrade planning tool to help you download a compatible bug fix version. Choose   > Administration > Plan your upgrade to open the tool.

Initiating a rolling upgrade

To initiate a rolling upgrade, enable rolling upgrade first. To do this, use:

https://<host>:<port>/rest/zdu/start

Enabling upgrade mode allows your cluster to accept nodes running a later bug fix version. This lets you upgrade one node and let it rejoin the cluster (along with the other non-upgraded nodes). Both upgraded and non-upgraded active nodes work together in keeping Bitbucket available to all users.

You can disable upgrade mode as long as you haven’t upgraded any nodes yet.

Upgrading each node individually

In general, upgrading a node during a rolling upgrade consists of four phases:

Disconnecting the node from the load balancer
tip/resting Created with Sketch.

Start with the least busy node

We recommend that you start upgrading the node with the least number of running tasks and active users. This will typically be the node with the lowest amount of CPU usage. 

When you disconnect a node from the load balancer, user requests will no longer be routed to the node. The following table provides guidance how to do so for popular load balancers:
NGINX

NGINX defines groups of cluster nodes through the upstream directive . To prevent the load balancer from connecting to a node, delete the node's entry from its corresponding upstream group. Learn more about the upstream directive in the ngx_http_upstream_module module

HAProxy

With HAProxy, you can disable all traffic to the node by putting it in a maint state:

set server <node IP or hostname> state maint

Learn more about forcing a server's administrative state

Apache

You can disable a node (or "worker") by setting its activation member attribute to disabled. Learn more about advanced load balancer worker properties in Apache

Azure Application Gateway

We provide a deployment template for Bitbucket Data Center on Azure; this template uses the Azure Application Gateway as its load balancer. The Azure Application Gateway defines each node as a target within a backend pool. Use the Edit backend pool interface to remove your node's corresponding entry. Learn more about adding (and removing) targets from a backend pool

Shutting down Bitbucket gracefully on the node

With upgrade mode enabled, you can now upgrade your first node. Start by shutting down Bitbucket gracefully on the node:

  1. Access the node through a command line or SSH.

  2. Shut down Bitbucket gracefully on the node. This will provide Bitbucket with some time to finish all of its tasks first before going offline. If you installed Bitbucket manually, run the bin/stop-bitbucket.sh script to gracefully shut down Bitbucket. Learn more about gracefully shutting down Bitbucket

  3. Wait for the node to go offline. You can monitor its status on the Node status column of the Rolling upgrade page’s Cluster overview section.

Extracting the downloaded files into a new directory

Once the node’s status is offline, you can start upgrading the node. Copy the Bitbucket files you downloaded (from Downloading upgrade files section) to the node’s local file system:

To upgrade the first node:
  1. Extract the files to a directory. This will be your new installation directory and it must be different from your existing installation directory.

  2. Update the value of BITBUCKET_HOME in the <Installation-directory>/bin/set-bitbucket-home.sh file so the new Bitbucket installation points to your existing Bitbucket home directory.

    If you’re using a BITBUCKET_HOME environment variable to specify the home directory location, no change is required.

  3. Copy any other immediately required customizations from the old version to the new one. For example, if you aren’t running Bitbucket on the default ports or if you’re managing users externally, you'll need to update or copy the relevant files.

    If you’ve configured Bitbucket to run as a Linux service, don't forget to update its service configuration as well. Learn more about running Bitbucket as a Linux service

  4. Start Bitbucket on the node. Learn more about starting Bitbucket

Why doesn’t Bitbucket search work after the upgrade?

When you’re starting up your Bitbucket Data Center instance after the upgrade, the bundled search server won’t run. Instead, the remote search will be started.

This happens because Data Center instances use the remote search, not the bundled search.

During the upgrade, the --no-search flag is inserted in the /etc/init.d/atlbitbucket file. This flag allows running the remote search but blocks the bundled search.

Reconnecting the node to the load balancer

After Bitbucket starts successfully on the node, reconnect it to the load balancer. This will allow the node to rejoin the cluster. As soon as the first upgraded node joins the cluster, your cluster status will transition to Mixed. This means that you won’t be able to disable Upgrade mode until all nodes are running the same version.

Finalizing the rolling upgrade


Once all nodes are upgraded, finalize the rolling upgrade. To do this, use:

https://<host>:<port>/rest/zdu/approve

This call will automatically disable upgrade mode.


After completing the rolling upgrade, you should:

Node statuses

To get the status of a node, use:

https://<host>:<port>/rest/zdu/nodes/<nodeID>

ACTIVEBitbucket is connected to the cluster and running with no errors. 
STARTINGBitbucket is still loading, and should transition to Active once finished.
TERMINATINGBitbucket was gracefully shut down, and should transition to Offline once finished.
OFFLINEBitbucket is not responding on the node. This node will be removed from the cluster completely if it is still offline after Upgrade mode is disabled.
ERRORSomething went wrong with Bitbucket on the node.

Cluster statuses

To get the status of the cluster, use:

https://<host>:<port>/rest/zdu/state

STABLEYou can turn on Upgrade mode now. 
READY_TO_UPGRADE

Upgrade mode is enabled, but no nodes have been upgraded yet. You can start upgrading your first node now. 

MIXEDAt least one node is upgraded, but you haven't finished upgrading all nodes yet. Your cluster has nodes running different Bitbucket versions. You need to upgrade all nodes to the same bug fix version to transition to the next status (READY_TO_RUN_UPGRADE_TASKS)
READY_TO_RUN_UPGRADE_TASKS

All nodes have node been upgraded. You can now finalize the rolling upgrade:

https://<host>:<port>/rest/zdu/zdu/approve

Troubleshooting

Node errors during rolling upgrade

If a node’s status transitions to Error, it means something went wrong during the upgrade. You can’t finish the rolling upgrade if any node has an Error status. However, you can still disable Upgrade mode as long as the cluster status is still Ready to upgrade.

There are several ways to address this:

  • Shut down Bitbucket gracefully on the node. This should disconnect the node from the cluster, allowing the node to transition to an Offline status.

  • If you can’t shut down Bitbucket gracefully, shut down the node altogether.

Once all active nodes are upgraded with no nodes in Error, you can finalize the rolling upgrade. You can investigate any problems with the problematic node afterwards and re-connect it to the cluster once you address the error.

Disabling upgrade mode

You can disable Upgrade mode as long as all nodes in the cluster:
  • haven't been upgraded yet
  • aren't in an Error state

The cluster's status will transition to Mixed if an upgraded node joins the cluster or a node enters an error state. 

Mixed status with Upgrade mode disabled

If a node is in an Error state with Upgrade mode disabled, you can't enable Upgrade mode. Fix the problem or remove the node from the cluster to enable Upgrade mode. 

Last modified on Oct 10, 2023

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.