How to establish staging server environments for Bitbucket Server
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.
Purpose
This document describes best practices for an enterprise environment setup for Bitbucket Server:
- Best-practice recommendations for procedural governance around rolling out changes
- Recommendations for development / staging / production architecture
- Technical steps for how to deploy non-production servers
Assumptions:
- For this document we are assuming that as an administrator, you would rather script changes. Therefore we have omitted UI-based changes or separate tools such as the database configuration tool in favor of specifying file system locations.
On this page:
Please Note:
- The procedures described in this document will work with Bitbucket Server version 4.0 and later.
- Please read the entire document before bringing a staging server live. There are risks associated with connecting to production instances that require attention, which are called out in the document.
1. Architecture Strategy
Often systems administration teams will have an established architecture for enterprise applications, including staging environments and failover setups. We offer these recommendations in this section not to supplant or change those company-wide strategies, but rather to help illustrate what some of the considerations will be with Atlassian products in staging environments.
Definitions
For the purpose of this document, we'll assume the following definitions:
- Production: your live instance, expecting minimal downtime and well tested changes.
- Staging: a pre-production environment, where the systems administration team can establish exact procedures prior to rollout.
- Development: a free-for-all environment where users can play with cutting-edge or risky changes.
Recommendation
If Atlassian products are critical systems, we recommend this 3-tier strategy for development, staging, and production.
- The staging environment is primarily for system administrators to test changes and upgrades before going into production.
- The development environment is for different business units to test changes on their own, before requesting a production rollout.
2. Governance Strategy
In addition to an architecture, we also recommend establishing a governance strategy for changes. This could include:
- Create a strategy for deploying and testing plugin installation requests. Note that some plugins that are extremely useful in some environments are not appropriate for high-volume critical systems.
Publish a timeline for refreshing the development environment.
3. How to Refresh a Staging Server
We're assuming that you have an existing staging installation. If not, you can use these instructions to set up your staging environment now.
Take care to make sure your staging server setup does not interfere with your production environment. Read the tips below before launching your staging (or development) server.
3.1 Create a complete production backup using one of the alternatives below
- Bitbucket Server Backup Client (not supported by Bitbucket Data Center)
- Bitbucket DIY Backup
3.2 Copy your complete production backup to a staging environment
- Shut down your staging server.
- Restore the backup onto your staging environment using one of the methods you had it backed up
Verify your system environment variable
BITBUCKET_HOME is set correctly
. See Setting Bitbucket Server Home Directory for details.
3.3 Modify your staging environment for the unique configurations
This is extremely important! Make sure your staging environment is not pointing to your production database.
3.4 Isolate the Staging Environment
This step only needs to be performed the first time you set up a staging environment, or whenever the production environment was integrated with another server/application since the last refresh of the staging environment.
Now that the data has been copied and the configuration of Bitbucket Server has been updated, it is advisable to fully isolate the staging environment from the production environment (and any others). This should be done to prevent the staging instance from contacting and interacting with any integrated systems, such as other applications connected via Application Links or Bitbucket Server Smart Mirrors.
The easiest way to do this is by modifying the /etc/hosts
(on Linux) or C:\Windows\System32\drivers\etc\hosts
(on Windows) file and pointing all relevant hostnames to a non-routable IP address, such as 0.0.0.0, e.g.
0.0.0.0 jira.company.com crowd.company.com mirror.us.company.com
Instead of modifying the hosts file, you can also block traffic via firewall rules if you prefer.
3.5 Remove Smart Mirror Settings
If you are using Smart Mirrors or Mirror Farms with your production instance, you need to remove their configuration from the staging instance before starting it, because otherwise your staging environment may attempt to synchronize with the mirrors.
Starting with Bitbucket 8.0, this can be accomplished by adding the following property to bitbucket.properties
for the first startup only. After startup, the property should be removed or commented out.
# Deletes mirror configuration on startup - Should only be run once on staging only.
plugin.mirroring.delete.on.startup=true
For versions 7.21 and below, you can use the queries below to remove the configuration from the database.
These queries must be run on the staging copy of your Bitbucket Server database. They must never be run on your production database, unless explicitly instructed to do so by Atlassian Support.
DELETE FROM plugin_setting WHERE key_name LIKE 'acnct.bitbucket.mirror.%';
DELETE FROM plugin_setting WHERE key_name = 'ac.addon.list' AND key_value LIKE '%mirror%';
DELETE FROM "AO_8E6075_MIRRORING_REQUEST";
DELETE FROM "AO_8E6075_MIRROR_SERVER";
3.6 Restart your Staging Server
You are now ready to restart your server. Once you've restarted, perform the following checks to verify you've done the above steps safely:
- Ensure the database is not pointing to production. To check this, see Viewing your System Information by going to Administration > Support Tools > Database Information. Check the 'Connection URL' to ensure it's pointing to the right place.
- Ensure emails are disabled or configured for dev server. (Administration > Mail server)
3.7 Post-Startup Modifications
- If you can't login. It could be that the External Directory that was configured in production is not available from the restored instance for some reason. You can get around this by using the Lockout recovery process procedure, removing the external directory, creating an
admin
account, logging out and start using the new admin account to perform your instance configuration. - Modify the Site Base URL. See Specify the Bitbucket base URL and change the Site URL to the staging URL.
- Apply a Development License. See our licensing FAQ to generate a license for the staging server. Refer to Update your license key to apply it.
Reconfigure applinks. If you are connecting to other servers via applinks, you'll need to change the server ID for those instances.
If you leave applinks in place, it's possible to have your production instance point back to the staging server, if a link is generated.
- Disable HipChat integration. If you have integrations which notify HipChat in any of your Bitbucket Server workflows, then you will need to remove the token from the HipChat integration configuration in Administration > Add Ons > HipChat Integration, to prevent HipChat notifications being sent from the staging environment.
- Ensure that no hooks are configured that will trigger builds if you are using CI tools. If you are using bamboo this will be taken care of when you disable application links. If you use something like Jenkins, you will need to manually disable or delete these hooks.
- Plugins that impact repositories or build systems would need to be reviewed to make sure they are not impacting production systems. For example, if you are using the Mirror plugin, the connections would need to be edited so that pushes to the staging environment are not mirrored out to some external production system.