Purpose

1. Architecture Strategy

Often systems administration teams will have an established architecture for enterprise applications, including staging environments and failover setups. We offer these recommendations in this section not to supplant or change those company-wide strategies, but rather to help illustrate what some of the considerations will be with Atlassian products in staging environments. 

Definitions

For the purpose of this document, we'll assume the following definitions:

• Production: your live instance, expecting minimal downtime and well tested changes.
• Staging: a pre-production environment, where the systems administration team can establish exact procedures prior to rollout.
• Development: a free-for-all environment where users can play with cutting-edge or risky changes. 

Recommendation

If Atlassian products are critical systems, we recommend this 3-tier strategy for development, staging, and production.

• The staging environment is primarily for system administrators to test changes and upgrades before going into production.
• The development environment is for different business units to test changes on their own, before requesting a production rollout.

2. Governance Strategy

In addition to an architecture, we also recommend establishing a governance strategy for changes. This could include:

• Create a strategy for deploying and testing plugin installation requests. Note that some plugins that are extremely useful in some environments are not appropriate for high-volume critical systems.

• Publish a timeline for refreshing the development environment.

3. How to Refresh a Staging Server

We're assuming that you have an existing staging installation. If not, you can use these instructions to set up your staging environment now. 

3.1 Create a complete production backup using one of the alternatives below

1. Bitbucket Server Backup Client (not supported by Bitbucket Data Center)
2. Bitbucket DIY Backup

3.2 Copy your complete production backup to a staging environment 

1. Shut down your staging server.
2. Restore the backup onto your staging environment using one of the methods you had it backed up 
   1. Restore from Backup Client to use a newly created DB
   2. Restoring a DIY Backup

3. Verify your system environment variable BITBUCKET_HOME is set correctly. See Setting Bitbucket Server Home Directory for details.
   
   

3.3 Modify your staging environment for the unique configurations

<Connector port="7990"
     protocol="HTTP/1.1"
     connectionTimeout="20000"
     useBodyEncodingForURI="true"
     redirectPort="443"
     compression="on"
     compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript"
     secure="true"
     scheme="https"
     proxyName="mycompany.com" 
     proxyPort="443" />

3.4 Isolate the Staging Environment

Now that the data has been copied and the configuration of Bitbucket Server has been updated, it is advisable to fully isolate the staging environment from the production environment (and any others). This should be done to prevent the staging instance from contacting and interacting with any integrated systems, such as other applications connected via Application Links or Bitbucket Server Smart Mirrors. 

The easiest way to do this is by modifying the /etc/hosts (on Linux) or C:\Windows\System32\drivers\etc\hosts (on Windows) file and pointing all relevant hostnames to a non-routable IP address, such as 0.0.0.0, e.g.

0.0.0.0 jira.company.com crowd.company.com mirror.us.company.com

Instead of modifying the hosts file, you can also block traffic via firewall rules if you prefer.

3.5 Remove Smart Mirror Settings

If you are using Smart Mirrors or Mirror Farms with your production instance, you need to remove their configuration from the staging instance before starting it, because otherwise your staging environment may attempt to synchronize with the mirrors.

Starting with Bitbucket 8.0, this can be accomplished by adding the following property to bitbucket.properties for the first startup only. After startup, the property should be removed or commented out. 

# Deletes mirror configuration on startup - Should only be run once on staging only.
plugin.mirroring.delete.on.startup=true

For versions 7.21 and below, you can use the queries below to remove the configuration from the database.

DELETE FROM plugin_setting WHERE key_name LIKE 'acnct.bitbucket.mirror.%';
DELETE FROM plugin_setting WHERE key_name = 'ac.addon.list' AND key_value LIKE '%mirror%';
DELETE FROM "AO_8E6075_MIRRORING_REQUEST";
DELETE FROM "AO_8E6075_MIRROR_SERVER";

3.6 Restart your Staging Server

You are now ready to restart your server. Once you've restarted, perform the following checks to verify you've done the above steps safely:

1. Ensure the database is not pointing to production. To check this, see Viewing your System Information by going to Administration > Support Tools > Database Information. Check the 'Connection URL' to ensure it's pointing to the right place.
2. Ensure emails are disabled or configured for dev server. (Administration > Mail server)

3.7 Post-Startup Modifications

1. If you can't login. It could be that the External Directory that was configured in production is not available from the restored instance for some reason. You can get around this by using the Lockout recovery process procedure, removing the external directory, creating an admin account, logging out and start using the new admin account to perform your instance configuration.
2. Modify the Site Base URL. See Specify the Bitbucket base URL and change the Site URL to the staging URL.
   
3. Apply a Development License.  See our licensing FAQ to generate a license for the staging server. Refer to Update your license key to apply it.

4. Reconfigure applinks. If you are connecting to other servers via applinks, you'll need to change the server ID for those instances. 

   If you leave applinks in place, it's possible to have your production instance point back to the staging server, if a link is generated.

5. Disable HipChat integration. If you have integrations which notify HipChat in any of your Bitbucket Server workflows, then you will need to remove the token from the HipChat integration configuration in Administration > Add Ons > HipChat Integration, to prevent HipChat notifications being sent from the staging environment.
6. Ensure that no hooks are configured that will trigger builds if you are using CI tools. If you are using bamboo this will be taken care of when you disable application links. If you use something like Jenkins, you will need to manually disable or delete these hooks. 
7. Plugins that impact repositories or build systems would need to be reviewed to make sure they are not impacting production systems. For example, if you are using the Mirror plugin, the connections would need to be edited so that pushes to the staging environment are not mirrored out to some external production system.