How to upgrade Mesh H2 database inside Bitbucket Kubernetes pod

Related content

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.

Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

There are manual tasks needed to upgrade Bitbucket Mesh due to the H2 database usage. On bare-metal or physical servers, you can follow the steps in the Migrate H2 database guide. However, with Bitbucket Remote Mesh installations using Kubernetes, we will need to orchestrate the steps as explained in this article.

  • The H2 database is only used for Bitbucket Mirror and Bitbucket Mesh.
  • This is not an as-is steps article - you might not be able to run the same exact commands and achieve the same result.
  • The content in this article is to showcase a concept and should be adapted into the upgrade process for Bitbucket Mesh.

(warning) If the Mesh H2 database manual migration isn't completed and an attempt is made to upgrade the Mesh nodes to a newer version without performing the manual upgrade, you will encounter the below errors in the logs after the upgrade. Additionally, the Mesh pods will enter a CrashLoopBackOff status.

2024-11-29 10:07:42,607 WARN  [main] - o.s.c.a.AnnotationConfigApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'databaseValidator' defined in com.atlassian.bitbucket.mesh.MeshDatabaseWiring: Invocation of init method failed; nested exception is com.atlassian.bitbucket.mesh.UnsupportedH2StoreException: The storage format for the internal database that you are using has changed and requires manual migration.
2024-11-29 10:07:42,608 INFO  [main] - c.a.b.mesh.boot.StandaloneRunner Shutting down
2024-11-29 10:07:42,681 ERROR [main] - o.s.b.d.LoggingFailureAnalysisReporter

***************************
APPLICATION FAILED TO START
***************************

Description:

The storage format for the internal database that you are using has changed and requires manual migration.

Action:

Please perform the manual migration of the database. See https://confluence.atlassian.com/bitbucketserver088/migrate-h2-database-1216583295.html.

The purpose of this KB is to resolve the above errors.

Environment

  • Please refer to the Migrate H2 database guide for the exact steps to be orchestrated.
  • For this article, we are performing Bitbucket Mesh upgrade from 2.0.1 to 2.5.7.

Kubernetes Environment details

We're using the below values through this document, you will need to tweak the commands in this document to match your environment.

  • Kubernetes namespace: bitbucket-mesh 
  • Helm chart name: bitbucket-mesh 
  • Helm values file: values-bitbucket-mesh.yaml 
  • Mesh Home is stored in a Kubernetes PVC (persistent volume claim)

Solution

  • Make sure you have all the required backups before performing any changes.
  • Test the steps in a non-production environment.

Step 1: Bring down Bitbucket Mesh pods

  1. Update the values.yaml and make the replicaCount of the Mesh Nodes to 0 so the Mesh Nodes are not being attempted to be started

      # Mesh configuration
  mesh:

    # -- Enable Bitbucket Mesh. See: https://confluence.atlassian.com/bitbucketserver/bitbucket-mesh-1128304351.html
    #
    enabled: true
    nodeAutoRegistration: false
    setByDefault: false
    image:
      repository: atlassian/bitbucket-mesh
      pullPolicy: IfNotPresent
      tag: "2.0.1"

    # -- Number of Bitbucket Mesh nodes. Do not change it. Currently, only the quorum of 3 mesh nodes is supported.
    # Reducing the number of replicas will result in mesh degradation while increasing the number of Mesh nodes
    # will result in new nodes being unused by the Bitbucket server.
    #
    replicaCount: 3

    Edit the replicaCount: 3 to replicaCount: 0.

  2. Bring down the Mesh pods: 

    helm upgrade --install bitbucket-mesh atlassian-data-center/bitbucket --namespace bitbucket-mesh --values values-bitbucket-mesh.yaml

Step 2: Deploy the Mesh H2 upgrade pods

Deploy an H2 upgrade pod for each Mesh node. Since there are three Mesh nodes, three new pods need to be created, with each pod attached to the PVC of its corresponding Mesh node.

We are spinning up three new Mesh pods and mounting the local PVC of each production Mesh pod to one of the new pods. This configuration ensures that the upgrade is performed on each pod individually. Each new pod must have the PVC of its respective production Mesh pod attached, as illustrated in the examples provided.

  1. Locate the Mesh PVCs (already created when 3 Mesh Pods were created)in the bitbucket-mesh namespace using the below command:

    kubectl get pvc --namespace bitbucket-mesh

    Example result: 

    NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mesh-home-bitbucket-mesh-0   Bound    pvc-056074c8-d704-411e-a4a6-4c46264864fb   1Gi        RWO            nfs-client     20h
mesh-home-bitbucket-mesh-1   Bound    pvc-ee0433c0-4458-4fd0-b83e-426776f5631a   1Gi        RWO            nfs-client     20h
mesh-home-bitbucket-mesh-2   Bound    pvc-6e981422-eefb-4926-bfca-a8178086688e   1Gi        RWO            nfs-client     20h

  2. Set up a new YAML file called values-bitbucket-mesh-h2-upgrade.yaml : 

    apiVersion: v1
kind: Pod
 
metadata:
  namespace: bitbucket
  name: bitbucket-mesh-h2-upgrade0
 
spec:
  restartPolicy: Never
  containers:
  - image: atlassian/bitbucket-mesh:2.5.7
    name: bitbucket-mesh-h2-upgrade0
    command: ["tail"]
    args: ["-f", "/dev/null"]
    volumeMounts:
    - mountPath: /var/atlassian/application-data/mesh
      name: pvc-mount
 
  volumes:
  - name: pvc-mount
    persistentVolumeClaim:
      claimName: mesh-home-bitbucket-mesh-0
 
---
       
apiVersion: v1
kind: Pod
 
metadata:
  namespace: bitbucket
  name: bitbucket-mesh-h2-upgrade1
 
spec:
  restartPolicy: Never
  containers:
  - image: atlassian/bitbucket-mesh:2.5.7
    name: bitbucket-mesh-h2-upgrade1
    command: ["tail"]
    args: ["-f", "/dev/null"]
    volumeMounts:
    - mountPath: /var/atlassian/application-data/mesh
      name: pvc-mount
 
  volumes:
  - name: pvc-mount
    persistentVolumeClaim:
      claimName: mesh-home-bitbucket-mesh-1
 
---
       
apiVersion: v1
kind: Pod
 
metadata:
  namespace: bitbucket
  name: bitbucket-mesh-h2-upgrade2
 
spec:
  restartPolicy: Never
  containers:
  - image: atlassian/bitbucket-mesh:2.5.7
    name: bitbucket-mesh-h2-upgrade2
    command: ["tail"]
    args: ["-f", "/dev/null"]
    volumeMounts:
    - mountPath: /var/atlassian/application-data/mesh
      name: pvc-mount
 
  volumes:
  - name: pvc-mount
    persistentVolumeClaim:
      claimName: mesh-home-bitbucket-mesh-2

  3. Deploy the Mesh H2 Upgrade Pods

    kubectl create -f values-bitbucket-mesh-h2-upgrade.yaml

    You should see 3 new Mesh H2 upgrade pods:

    kubectl get pods -n bitbucket-mesh                                                                                                                                     
NAME                         READY   STATUS    RESTARTS      AGE
bitbucket-mesh-h2-upgrade0   1/1     Running   0             19s
bitbucket-mesh-h2-upgrade1   1/1     Running   0             19s
bitbucket-mesh-h2-upgrade2   1/1     Running   0             19s

Step 3: Perform the H2 upgrade

  1. Log in to the Mesh H2 upgrade pod bitbucket-mesh-h2-upgrade0.

    kubectl exec -it bitbucket-mesh-h2-upgrade0 -n bitbucket-mesh bash

    Info

    The same steps must be repeated for the other two Mesh upgrade pods as well.

  2. Obtain the H2 JAR file to do the migration inside the pod by running the below commands:

    cd /var/atlassian/application-data/mesh

#####Source H2 Jar#####
wget https://repo1.maven.org/maven2/com/h2database/h2/2.1.214/h2-2.1.214.jar
#####Target H2 Jar#####
wget https://repo1.maven.org/maven2/com/h2database/h2/2.2.220/h2-2.2.220.jar

    (info)  To identify the exact source and target H2 version JAR, refer to the document How do I check the H2 database version used by Mesh?

  3. Create a new file named h2-migrate-db-file.sh inside the Pod on /var/atlassian/application-data/mesh directory and copy paste the below code:

    #!/usr/bin/env bash
 
if [ -z "$SOURCE_H2_JAR_PATH" ]; then
  echo "Please provide the location of the <source H2> JAR in the 'SOURCE_H2_JAR_PATH' environment variable"
  exit 1
fi
 
# BIN_DIR & INST_DIR will be absolute paths, not relative
pushd $(dirname $0) > /dev/null
BIN_DIR=$(pwd)
popd > /dev/null
INST_DIR=$(dirname "$BIN_DIR")
 
source "$BIN_DIR"/set-jre-home.sh &&
    source "$BIN_DIR"/set-mesh-home.sh &&
    source "$BIN_DIR"/set-mesh-user.sh
 
DB_FILE_WITHOUT_EXT="$MESH_HOME/mesh"
DB_FILE="$DB_FILE_WITHOUT_EXT.mv.db"
DB_LOCK_FILE="$DB_FILE_WITHOUT_EXT.lock.db"
 
if [ -f "$DB_LOCK_FILE" ]; then
  echo "The file $DB_LOCK_FILE is present which means the database may be in use. Please stop Mesh or any other clients using the $DB_FILE file and try again. If the database is not in use, manually delete the file $DB_LOCK_FILE and try again."
  exit 1
fi
 
if [ ! -f "$DB_FILE" ]; then
  echo "Cannot run migration; $DB_FILE doesn't exist"
  exit 1
fi
 
if [ -z "$H2_JAR_PATH" ]; then
  H2_JAR_PATH="$BIN_DIR/h2.jar"
fi
 
JDBC_URL="jdbc:h2:$DB_FILE_WITHOUT_EXT"
SCRIPT_FILE="h2_script.sql"
SCRIPT_JAVA_OPTS="-cp $SOURCE_H2_JAR_PATH org.h2.tools.Script -user sa -url $JDBC_URL -script $SCRIPT_FILE"
RUN_SCRIPT_JAVA_OPTS="-cp $H2_JAR_PATH org.h2.tools.RunScript -user sa -url $JDBC_URL -script $SCRIPT_FILE -showResults -options FROM_1X"
 
BACKUP_FILE="$MESH_HOME/backup_mesh.mv.db"
 
echo "Creating script file..."
$JAVA_BINARY $SCRIPT_JAVA_OPTS
if [ $? -eq 0 ]; then
  echo "Script file created successfully"
  mv "$DB_FILE" "${BACKUP_FILE}"
else
  echo "Script generation failed"
  exit 1
fi
 
echo "Running script..."
$JAVA_BINARY $RUN_SCRIPT_JAVA_OPTS
if [[ $? -eq 0 && -f "$DB_FILE" ]]; then
  echo "Run script finished, DB file generated successfully at $DB_FILE"
  rm -rf "$SCRIPT_FILE"
  chmod u=rw,go=r $DB_FILE
else
  echo "Run script failed"
  rm -rf ${DB_FILE_WITHOUT_EXT}.*.db
  mv "${BACKUP_FILE}" "$DB_FILE"
  exit 1
fi

  4. Grant execute permissions to the script.

    chmod +x h2-migrate-db-file.sh

  5. Run the Mesh H2 Upgrade Script

    MESH_HOME=/var/atlassian/application-data/mesh SOURCE_H2_JAR_PATH=/var/atlassian/application-data/mesh/h2-2.1.214.jar H2_JAR_PATH=/var/atlassian/application-data/mesh/h2-2.2.220.jar ./h2-migrate-db-file.sh

  6. When the H2 DB Migration script is successfully executed, it will generate output that looks similar to the following:

    Creating script file...
Script file created successfully
Running script...
Run script finished, DB file generated successfully at /var/opt/atlassian-bitbucket/home/mesh.mv.db

  7. Fix the permissions for the file mesh.mv.db

    chown bitbucket:bitbucket /var/atlassian/application-data/mesh/mesh.mv.db

(info) Repeat the same steps for the bitbucket-mesh-h2-upgrade1 and bitbucket-mesh-h2-upgrade2 pods following the above steps

Step 4: Delete the H2 upgrade pod

  1. Once we have a successful H2 upgrade, we can delete the Mesh H2 upgrade pods:

    kubectl delete pod bitbucket-mesh-h2-upgrade0 bitbucket-mesh-h2-upgrade1 bitbucket-mesh-h2-upgrade2 -n bitbucket-mesh

Step 5: Upgrade and Start the Bitbucket Mesh Nodes

  1. Update the Bitbucket Mesh YAML file: 

    mesh:
	enabled: true
	replicaCount: 3

image:
	tag: "2.5.7"

  2. Bring up the Bitbucket Mesh Nodes and perform the upgrade: 

    helm upgrade --install bitbucket-mesh atlassian-data-center/bitbucket --namespace bitbucket-mesh  --values values-bitbucket-mesh.yaml




Last modified on Nov 29, 2024

Was this helpful?

Yes
No
Provide feedback about this article

Related content

Powered by Confluence and Scroll Viewport.