Purpose

1. Architecture strategy

Often systems administration teams will have an established architecture for enterprise applications, including staging environments and failover setups. We offer these recommendations in this section not to supplant or change those company-wide strategies, but rather to help illustrate what some of the considerations will be with Atlassian products in staging environments. 

Definitions

For the purpose of this document, we'll assume the following definitions:

• Production: your live instance, expecting minimal downtime and well-tested changes.
• Staging: a pre-production environment, where the systems administration team can establish exact procedures prior to rollout.
• Development: a free-for-all environment where users can play with cutting-edge or risky changes. 

Recommendation

If Atlassian products are critical systems, we recommend this 3-tier strategy for development, staging, and production.

• The staging environment is primarily for system administrators to test changes and upgrades before going into production.
• The development environment is for different business units to test changes on their own, before requesting a production rollout.

2. Governance strategy

In addition to architecture, we also recommend establishing a governance strategy for changes. This could include:

• Create a strategy for deploying and testing plugin installation requests. Note that some plugins that are extremely useful in some environments are not appropriate for high-volume critical systems.

• Publish a timeline for refreshing the development environment.

3. How to refresh a Staging Server

We're assuming that you have an existing staging installation. If not, you can use these instructions to set up your staging environment now. 

3.1 Create a complete production backup using one of the alternatives below

1. 2023-10-24_01-30-41_Bitbucket Server Backup Client (not supported by Bitbucket Data Center)
2. Bitbucket DIY Backup

3.2 Copy your complete production backup to a staging environment 

1. Shut down your staging server.
2. Restore the backup onto your staging environment using one of the methods you had it backed up 
   1. Restore from Backup Client to use a newly created DB
   2. Restoring a DIY Backup

3. Verify your system environment variable BITBUCKET_HOME is set correctly. See Setting Bitbucket Server Home Directory for details.
   
   

3.3 Modify your staging environment for the unique configurations

<Connector port="7990"
     protocol="HTTP/1.1"
     connectionTimeout="20000"
     useBodyEncodingForURI="true"
     redirectPort="443"
     compression="on"
     compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript"
     secure="true"
     scheme="https"
     proxyName="mycompany.com" 
     proxyPort="443" />

3.4 Isolate the Staging Environment

Now that the data has been copied and the configuration of Bitbucket Server has been updated, it is advisable to fully isolate the staging environment from the production environment (and any others). This should be done to prevent the staging instance from contacting and interacting with any integrated systems, such as other applications connected via Application Links or Bitbucket Server Smart Mirrors. 

The easiest way to do this is by modifying the /etc/hosts (on Linux) or C:\Windows\System32\drivers\etc\hosts (on Windows) file and pointing all relevant hostnames to a non-routable IP address, such as 0.0.0.0, e.g.

0.0.0.0 jira.company.com crowd.company.com mirror.us.company.com

Instead of modifying the host's file, you can also block traffic via firewall rules if you prefer.

3.5 Remove Smart Mirror settings

If you are using Smart Mirrors or Mirror Farms with your production instance, you need to remove their configuration from the staging instance before starting it, because otherwise, your staging environment may attempt to synchronize with the mirrors.

Starting with Bitbucket 8.0, this can be accomplished by adding the following property to bitbucket.properties for the first startup only. After startup, the property should be removed or commented out. 

# Deletes mirror configuration on startup - Should only be run once on staging only.
plugin.mirroring.delete.on.startup=true

For versions 7.21 and below, you can use the queries below to remove the configuration from the database.

DELETE FROM plugin_setting WHERE key_name LIKE 'acnct.bitbucket.mirror.%';
DELETE FROM plugin_setting WHERE key_name = 'ac.addon.list' AND key_value LIKE '%mirror%';
DELETE FROM "AO_8E6075_MIRRORING_REQUEST";
DELETE FROM "AO_8E6075_MIRROR_SERVER";

3.6 Disable webhooks

The existing webhooks may try to contact production resources when triggered by Git activity. In order to avoid this from happening, you can disable the webhooks via the database:

1. View the webhooks:
   

   -- The select query below is an example for a specific webhook name. Remove the where clause to retrieve all webhooks.
   select a.*, r.name from "AO_A0B856_WEBHOOK" a JOIN repository r on cast(a."SCOPE_ID" as integer)=r.id where a."NAME" = "Test";

2. Run the update query to disable webhooks:

   These queries must be run on the staging copy of your Bitbucket Server database. Only execute if the instance is down. They must never be run on your production database unless explicitly instructed to do so by Atlassian Support.

   -- The update query below is an example for a specific webhook name. Remove the where clause to update all webhooks.
   update "AO_A0B856_WEBHOOK" where "NAME" = "Test"
   set "ACTIVE" = false;

3.7 Remove OAuth ConsumerService record from plugin_setting table

Remove com.atlassian.oauth.consumer.ConsumerService:host.__HOST_SERVICE__   record from plugin_setting table. This record contains oauth private key and Bitbucket Base URL which is referenced while creating application links with other Atlassian Products.
Having this record in the Bitbucket staging instance can lead to the display of Bitbucket production Base URL in integrated applications (such Jira, Jira Cloud, Bamboo etc.,) instead of Bitbucket staging URL.

delete from plugin_setting where key_name like '%com.atlassian.oauth.consumer.ConsumerService:host.__HOST_SERVICE__%';

Note that Bitbucket automatically creates the record (if not present) with the current instance setup during the process of creating application links.

3.8 Restart your Staging Server

You are now ready to restart your server. Once you've restarted, perform the following checks to verify you've done the above steps safely:

1. Ensure the database is not pointing to production. To check this, see Viewing your System Information by going to Administration > Support Tools > Database Information. Check the 'Connection URL' to ensure it's pointing to the right place.
2. Ensure emails are disabled or configured for the dev server. (Administration > Mail server)

3.9 Post-Startup Modifications

1. If you can't log in. It could be that the External Directory that was configured in production is not available from the restored instance for some reason. You can get around this by using the Lockout recovery process procedure, removing the external directory, creating an admin account, log out, and start using the new admin account to perform your instance configuration.
2. Modify the Site Base URL. See Specify the Bitbucket base URL and change the Site URL to the staging URL.
   
3. Apply for a Development License.  See our licensing FAQ to generate a license for the staging server. Refer to Update your license key to apply it.

4. Reconfigure app links. If you are connecting to other servers via app links, you'll need to change the server ID for those instances. 

   If you leave app links in place, it's possible to have your production instance point back to the staging server, if a link is generated.

5. Ensure that no hooks are configured that will trigger builds if you are using CI tools. If you are using bamboo this will be taken care of when you disable application links. If you use something like Jenkins, you will need to manually disable or delete these hooks. 
6. Plugins that impact repositories or build systems would need to be reviewed to make sure they are not impacting production systems. For example, if you are using the Mirror plugin, the connections would need to be edited so that pushes to the staging environment are not mirrored out to some external production system.