Maintaining a Crowd cluster on Azure

Once you've deployed Crowd Data Center to Azure using the reference deployment template, administering the application is similar to managing an application on your own hardware, with the exception that you'll need to go via the jumpbox to access your nodes and shared home directory. 

Connecting to your Azure jumpbox and nodes over SSH

Your cluster nodes have been deployed into a private subnet, which means you can’t access them directly. To work around that, we’ve also deployed a small Azure VM (jumpbox, or bastion host) into a public subnet, so you can use it to access your nodes over SSH.

Before you begin:

• Get the SSH credentials you provided during the deployment.
• Get the private key file that you created for SSH during the deployment.
• Get the IP address of your jumpbox. In Azure Portal, open your resource group, and then go to Deployments > atlassian.Crowd-data-center-<id> > Outputs.
• Get the private IP address of the instance you want to access. In your resource group, open the CROWDcluster resource, and go to Instances.

Once you're ready to connect:

1. Access the jumpbox through a command line: 

   ssh -i privatekeyfile ssh_username@dns_name_or_ip_address

   For example: 

   ssh -i privatekey Crowdadmin@Crowd-jumpbox-ip-73kaq.eastus.cloudapp.azure.com

2. Once you've accessed the jumpbox, you can connect to any of your nodes in the cluster: 

   ssh ssh_username@node_ip_address

   For example: 

   ssh Crowdadmin@10.0.2.4

Accessing your crowd.cfg.xml or crowd.version files

Once you've accessed your nodes over SSH, navigate to the /media/atl/CROWD/shared/ directory to find your crowd.cfg.xml and crowd.version files. In Azure, these files are copied from the shared home to the local home when a new node joins the cluster. 

When making changes to these files on existing nodes, it's important to also update the files in the shared home, otherwise a new node joining the cluster will be set up with outdated configuration.

So, when making changes in crowd.cfg.xml or crowd.version on existing nodes, it's important to also update the files in the shared home, otherwise a new node joining the cluster will be setup with outdated configuration.

These files are only accessible from the existing nodes. The shared home is mounted (think of it as a network hard disk) on each node under /media/atl/CROWD/shared/. 

Backing up and recovering from failures

We recommend you use the Azure native backup facilities where possible to make sure your data is backed up, and you can easily recover in the case of a failure. 

Database

We use Azure managed database instances with high availability. Azure provides several excellent options for backing up your database, so you should take some time to work out which will be the best, and most cost effective option for your needs. See the following Azure documentation for your chosen database:

• SQL Database: Automated backups
• SQL Database: Backup retention 
• PostgreSQL: Backup concepts

Shared home

The shared home is where your attachments and export files are stored. We create a general purpose Azure storage account, configured with local redundant storage, which means there are multiple copies of the data at any one time. 

Because of the redundancy strategy, it shouldn't be necessary to take regular backups, however you can take point in time backups using snapshots if necessary. 

Application nodes

The application nodes are VMs in an Azure Virtual Machine Scale Set. Each application node has a Crowd installation directory and a local home directory containing things like logs and search indexes. 

Like the shared home, application nodes are configured with local redundant storage, which means there are multiple copies of the data at any one time. 

If you've manually customised any configuration files in the installation directory (for example velocity templates), you may also want to manually back these up as a reference. 

Bastion host

As this VM acts as a jumpbox, and doesn't store any data it doesn't need to be backed up. If the VM becomes unresponsive it can be restarted from the Azure Portal. 

Application gateway

The application gateway is highly available. We deploy 2 instances by default. As with the bastion host, it doesn't need to be backed up. 

Upgrading

To upgrade Crowd Data Center to a later version:

Upgrading the node's operating system

The easiest way to upgrade the node's operating system is to reimage the node. It will be terminated, and will come back running the latest OS.

1. In Azure Portal, access the CROWDcluster Virtual Machine Scale Set (VMSS).
2. Click Instances.
3. Select a node, and click Reimage.

Upgrading Crowd in Azure

The process of upgrading Crowd is the same as if you were running the cluster on your own hardware. Stop Crowd on all nodes, upgrade one node, stop that node then copy the installation directory across to each remaining node in the cluster, before restarting each node, one at a time.    

Crowd offers two methods for upgrading each node. Refer to the following pages for detailed instructions on each one.

• Method 1: Upgrading Crowd via Automatic Database Upgrade
• Method 2: Upgrading Crowd via XML Data Transfer

Backing up and recovering from failures

We recommend you use the Azure native backup facilities where possible to make sure your data is backed up, and you can easily recover in the case of a failure. 

Database backups

We use Azure-managed database instances with high availability.  Azure provides several excellent options for backing up your database, so you should take some time to work out which will be the best, and most cost effective option for your needs. See the following Azure documentation for your chosen database:

• SQL Database: Automated backups
• SQL Database: Backup retention 
• PostGreSQL: Backup concepts

Shared home backups

The shared home stores your your attachments, profile pictures, and export files. We create a general purpose Azure storage account, configured with local redundant storage (LRS), which means there are multiple copies of the data at any one time. 

LRS provides a basic redundancy strategy for your shared home. As such, it shouldn't be necessary to take regular backups yourself. If you need to take point-in-time backups, use snapshots.

Application nodes

The application nodes are VMs in an Azure Virtual Machine Scale Set. Each application node has a Crowd installation directory and a local home directory containing things like logs and search indexes. 

Like the shared home, application nodes are configured with local redundant storage. This means there are multiple copies of the data at any one time. 

If you've manually customised any configuration files in the installation directory (for example velocity templates), you may also want to manually back these up as a reference. 

Bastion host

As this VM acts as a jumpbox, and doesn't store any data it doesn't need to be backed up. If the VM  becomes unresponsive it can be restarted from the Azure Portal. 

Application gateway

The application gateway is highly available. We deploy 2 instances by default. As with the bastion host, it doesn't need to be backed up. 

Disaster recovery

See information in the Azure documentation about recovering from a region-wide failure Azure resiliency technical guidance: recovery from a region-wide service disruption.