Upgrade a Bitbucket cluster through the API without downtime
The entire rolling upgrade process is governed by the following API:
This API has the following calls:
|Get an overview of the cluster's status.|
|Enable upgrade mode.|
|Get the status of the cluster.|
|Get an overview of a node's status.|
|Disable upgrade mode. You can only use this call if the upgrade progress is not MIXED.|
Once all nodes are upgraded, finalize the rolling upgrade. This will automatically disable upgrade mode.
For detailed information about each API call, see Bitbucket REST API Documentation.
Authenticating REST API calls
In a secure environment, you'll need to authenticate your REST API calls. See Personal access tokens and Bitbucket Server REST API Example - Basic Authentication for more information.
Downloading upgrade files
Alternatively, you can also use the Pre-upgrade planning tool to help you download a compatible bug fix version. Choose > Administration > Plan your upgrade to open the tool.
Initiating a rolling upgrade
To initiate a rolling upgrade, enable rolling upgrade first. To do this, use:
Enabling upgrade mode allows your cluster to accept nodes running a later bug fix version. This lets you upgrade one node and let it rejoin the cluster (along with the other non-upgraded nodes). Both upgraded and non-upgraded active nodes work together in keeping Bitbucket available to all users.
You can disable upgrade mode as long as you haven’t upgraded any nodes yet.
Upgrading each node individually
In general, upgrading a node during a rolling upgrade consists of four phases:
Start with the least busy node
We recommend that you start upgrading the node with the least number of running tasks and active users. This will typically be the node with the lowest amount of CPU usage.
When you disconnect a node from the load balancer, user requests will no longer be routed to the node. The following table provides guidance how to do so for popular load balancers:
NGINX defines groups of cluster nodes through the
With HAProxy, you can disable all traffic to the node by putting it in a
You can disable a node (or "worker") by setting its
|Azure Application Gateway|
We provide a deployment template for Bitbucket Data Center on Azure; this template uses the Azure Application Gateway as its load balancer. The Azure Application Gateway defines each node as a target within a backend pool. Use the Edit backend pool interface to remove your node's corresponding entry. Learn more about adding (and removing) targets from a backend pool
With upgrade mode enabled, you can now upgrade your first node. Start by shutting down Bitbucket gracefully on the node:
Access the node through a command line or SSH.
Shut down Bitbucket gracefully on the node. This will provide Bitbucket with some time to finish all of its tasks first before going offline. If you installed Bitbucket manually, run the
bin/stop-bitbucket.shscript to gracefully shut down Bitbucket. Learn more about gracefully shutting down Bitbucket
Wait for the node to go offline. You can monitor its status on the Node status column of the Rolling upgrade page’s Cluster overview section.
Once the node’s status is offline, you can start upgrading the node. Copy the Bitbucket files you downloaded (from Downloading upgrade files section) to the node’s local file system:
To upgrade the first node:
Extract (unzip) the files to a directory (this will be your new installation directory, and must be different to your existing installation directory)
Update the value of BITBUCKET_HOME in the
<Installation-directory>/bin/set-bitbucket-home.shfile so the new Bitbucket installation points to your existing Bitbucket home directory.
If you use a BITBUCKET_HOME environment variable to specify the home directory location, no change is required
Copy any other immediately required customizations from the old version to the new one (for example if you are not running Bitbucket on the default ports or if you manage users externally, you'll need to update / copy the relevant files)
If you configured Bitbucket to run as a Linux service, don't forget to update its service configuration as well. Learn more about running Bitbucket as a Linux service.
Start Bitbucket on the node. Learn more about starting Bitbucket
After Bitbucket starts successfully on the node, reconnect it to the load balancer. This will allow the node to rejoin the cluster. As soon as the first upgraded node joins the cluster, your cluster status will transition to Mixed. This means that you won’t be able to disable Upgrade mode until all nodes are running the same version.
Finalizing the rolling upgrade
Once all nodes are upgraded, finalize the rolling upgrade. To do this, use:
This call will automatically disable upgrade mode.
After completing the rolling upgrade, you should:
- Update your apps accordingly
- Perform UAT and other tests as needed
To get the status of a node, use:
|ACTIVE||Bitbucket is connected to the cluster and running with no errors.|
|STARTING||Bitbucket is still loading, and should transition to Active once finished.|
|TERMINATING||Bitbucket was gracefully shut down, and should transition to Offline once finished.|
|OFFLINE||Bitbucket is not responding on the node. This node will be removed from the cluster completely if it is still offline after Upgrade mode is disabled.|
|ERROR||Something went wrong with Bitbucket on the node.|
To get the status of the cluster, use:
|STABLE||You can turn on Upgrade mode now.|
Upgrade mode is enabled, but no nodes have been upgraded yet. You can start upgrading your first node now.
|MIXED||At least one node is upgraded, but you haven't finished upgrading all nodes yet. Your cluster has nodes running different Bitbucket versions. You need to upgrade all nodes to the same bug fix version to transition to the next status (READY_TO_RUN_UPGRADE_TASKS)|
All nodes have node been upgraded. You can now finalize the rolling upgrade:
Node errors during rolling upgrade
There are several ways to address this:
Shut down Bitbucket gracefully on the node. This should disconnect the node from the cluster, allowing the node to transition to an Offline status.
If you can’t shut down Bitbucket gracefully, shut down the node altogether.
Once all active nodes are upgraded with no nodes in Error, you can finalize the rolling upgrade. You can investigate any problems with the problematic node afterwards and re-connect it to the cluster once you address the error.
Disabling upgrade mode
- haven't been upgraded yet
- aren't in an Error state
The cluster's status will transition to Mixed if an upgraded node joins the cluster or a node enters an error state.
Mixed status with Upgrade mode disabled
If a node is in an Error state with Upgrade mode disabled, you can't enable Upgrade mode. Fix the problem or remove the node from the cluster to enable Upgrade mode.