How to establish staging server environments for Bitbucket Server

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Purpose


This document describes best practices for an enterprise environment setup for Bitbucket Server:

  • Best-practice recommendations for procedural governance around rolling out changes
  • Recommendations for development/staging/production architecture
  • Technical steps for how to deploy non-production servers

Assumptions:

  • For this document, we are assuming that as an administrator, you would rather script changes. Therefore we have omitted UI-based changes or separate tools such as the database configuration tool in favor of specifying file system locations.

On this page:

tip/resting Created with Sketch.

Please Note:

  • The procedures described in this document will work with Bitbucket Server version 4.0 and later.
  • Please read the entire document before bringing a staging server live. There are risks associated with connecting to production instances that require attention, which is called out in the document.

1. Architecture strategy

Often systems administration teams will have an established architecture for enterprise applications, including staging environments and failover setups. We offer these recommendations in this section not to supplant or change those company-wide strategies, but rather to help illustrate what some of the considerations will be with Atlassian products in staging environments. 

Definitions

For the purpose of this document, we'll assume the following definitions:

  • Production: your live instance, expecting minimal downtime and well-tested changes.
  • Staging: a pre-production environment, where the systems administration team can establish exact procedures prior to rollout.
  • Development: a free-for-all environment where users can play with cutting-edge or risky changes. 
Recommendation

If Atlassian products are critical systems, we recommend this 3-tier strategy for development, staging, and production.

  • The staging environment is primarily for system administrators to test changes and upgrades before going into production.
  • The development environment is for different business units to test changes on their own, before requesting a production rollout.

2. Governance strategy

In addition to architecture, we also recommend establishing a governance strategy for changes. This could include:

  • Create a strategy for deploying and testing plugin installation requests. Note that some plugins that are extremely useful in some environments are not appropriate for high-volume critical systems.
  • Publish a timeline for refreshing the development environment.

3. How to refresh a Staging Server

We're assuming that you have an existing staging installation. If not, you can use these instructions to set up your staging environment now. 

Take care to make sure your staging server setup does not interfere with your production environment. Read the tips below before launching your staging (or development) server.

3.1 Create a complete production backup using one of the alternatives below

  1. Bitbucket Server Backup Client (not supported by Bitbucket Data Center)
  2. Bitbucket DIY Backup

3.2 Copy your complete production backup to a staging environment 

  1. Shut down your staging server.
  2. Restore the backup onto your staging environment using one of the methods you had it backed up 
    1. Restore from Backup Client to use a newly created DB
    2. Restoring a DIY Backup
  3. Verify your system environment variable BITBUCKET_HOME is set correctly. See Setting Bitbucket Server Home Directory for details.

3.3 Modify your staging environment for the unique configurations

This is extremely important! Make sure your staging environment is not pointing to your production database. 

If you used DIY...
  • Configure your database connection to point to your staging database.  Edit the bitbucket-config.properties in BITBUCKET_HOME/shared
  • Fix your <BBS_HOME>/shared/server.xml . In case you had a reverse proxy you need to remove the reference to the old name, otherwise, your new instance will redirect you to the production environment:
<Connector port="7990"
     protocol="HTTP/1.1"
     connectionTimeout="20000"
     useBodyEncodingForURI="true"
     redirectPort="443"
     compression="on"
     compressableMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/x-javascript"
     secure="true"
     scheme="https"
     proxyName="mycompany.com" 
     proxyPort="443" />

If you're running Bitbucket Server 5.0+, the proxy configuration is made in bitbucket.properties and server.xml is no longer used.

tip/resting Created with Sketch.

If you are using a database (called BitbucketServerDB for example) with your existing Bitbucket Server installation and the database for your new Bitbucket Server installation is running on the same machine or database server, create your new database with a different name (e.g. something intuitive like BitbucketServerDB_402 for Bitbucket Server 4.0.2). Oracle does not support schema names with periods or underscores. Ensure the new database has identical access permissions to the old Bitbucket Server database.

If you used the Restore client...
  • The client will write the specified parameters to the  bitbucket.properties  file in the newly restored Bitbucket Server home directory bitbucket.properties in BITBUCKET_HOME/shared. This will all have been done as part of the Restore from Backup Client to use a newly created DB executed on step 3.2.
  • Fix your <BBS_HOME>/shared/server.xml . In case you had a reverse proxy you need to remove the reference to the old name, otherwise, your new instance will redirect you to the production environment:

If you're running Bitbucket Server 5.0+, the proxy configuration is made in bitbucket.properties and server.xml is no longer used.

3.4 Isolate the Staging Environment

This step only needs to be performed the first time you set up a staging environment, or whenever the production environment was integrated with another server/application since the last refresh of the staging environment.

Now that the data has been copied and the configuration of Bitbucket Server has been updated, it is advisable to fully isolate the staging environment from the production environment (and any others). This should be done to prevent the staging instance from contacting and interacting with any integrated systems, such as other applications connected via Application Links or Bitbucket Server Smart Mirrors. 

The easiest way to do this is by modifying the /etc/hosts (on Linux) or C:\Windows\System32\drivers\etc\hosts (on Windows) file and pointing all relevant hostnames to a non-routable IP address, such as 0.0.0.0, e.g.

0.0.0.0 jira.company.com crowd.company.com mirror.us.company.com

Instead of modifying the host's file, you can also block traffic via firewall rules if you prefer.

3.5 Remove Smart Mirror settings

If you are using Smart Mirrors or Mirror Farms with your production instance, you need to remove their configuration from the staging instance before starting it, because otherwise, your staging environment may attempt to synchronize with the mirrors.

Starting with Bitbucket 8.0, this can be accomplished by adding the following property to bitbucket.properties for the first startup only. After startup, the property should be removed or commented out. 

# Deletes mirror configuration on startup - Should only be run once on staging only.
plugin.mirroring.delete.on.startup=true

For versions 7.21 and below, you can use the queries below to remove the configuration from the database.

These queries must be run on the staging copy of your Bitbucket Server database. Only execute if the instance is down. They must never be run on your production database unless explicitly instructed to do so by Atlassian Support.

DELETE FROM plugin_setting WHERE key_name LIKE 'acnct.bitbucket.mirror.%';
DELETE FROM plugin_setting WHERE key_name = 'ac.addon.list' AND key_value LIKE '%mirror%';
DELETE FROM "AO_8E6075_MIRRORING_REQUEST";
DELETE FROM "AO_8E6075_MIRROR_SERVER";

3.6 Disable webhooks

The existing webhooks may try to contact production resources when triggered by Git activity. In order to avoid this from happening, you can disable the webhooks via the database:

  1. View the webhooks:

    -- The select query below is an example for a specific webhook name. Remove the where clause to retrieve all webhooks.
    select a.*, r.name from "AO_A0B856_WEBHOOK" a JOIN repository r on cast(a."SCOPE_ID" as integer)=r.id where a."NAME" = "Test";
  2. Run the update query to disable webhooks:

    These queries must be run on the staging copy of your Bitbucket Server database. Only execute if the instance is down. They must never be run on your production database unless explicitly instructed to do so by Atlassian Support.

    -- The update query below is an example for a specific webhook name. Remove the where clause to update all webhooks.
    update "AO_A0B856_WEBHOOK" where "NAME" = "Test"
    set "ACTIVE" = false;

3.7 Remove OAuth ConsumerService record from plugin_setting table

Remove com.atlassian.oauth.consumer.ConsumerService:host.__HOST_SERVICE__   record from plugin_setting table. This record contains oauth private key and Bitbucket Base URL which is referenced while creating application links with other Atlassian Products.
(warning) Having this record in the Bitbucket staging instance can lead to the display of Bitbucket production Base URL in integrated applications (such Jira, Jira Cloud, Bamboo etc.,) instead of Bitbucket staging URL.

delete from plugin_setting where key_name like '%com.atlassian.oauth.consumer.ConsumerService:host.__HOST_SERVICE__%';

(info) Note that Bitbucket automatically creates the record (if not present) with the current instance setup during the process of creating application links.

3.8 Restart your Staging Server

You are now ready to restart your server. Once you've restarted, perform the following checks to verify you've done the above steps safely:

  1. Ensure the database is not pointing to production. To check this, see Viewing your System Information by going to Administration > Support Tools > Database Information. Check the 'Connection URL' to ensure it's pointing to the right place.
  2. Ensure emails are disabled or configured for the dev server. (Administration > Mail server)

3.9 Post-Startup Modifications

  1. If you can't log in. It could be that the External Directory that was configured in production is not available from the restored instance for some reason. You can get around this by using the Lockout recovery process procedure, removing the external directory, creating an admin account, log out, and start using the new admin account to perform your instance configuration.
  2. Modify the Site Base URL. See Specify the Bitbucket base URL and change the Site URL to the staging URL.
  3. Apply for a Development License.  See our licensing FAQ to generate a license for the staging server. Refer to Update your license key to apply it.
  4. Reconfigure app links. If you are connecting to other servers via app links, you'll need to change the server ID for those instances. 

    tip/resting Created with Sketch.

    If you leave app links in place, it's possible to have your production instance point back to the staging server, if a link is generated.

  5. Ensure that no hooks are configured that will trigger builds if you are using CI tools. If you are using bamboo this will be taken care of when you disable application links. If you use something like Jenkins, you will need to manually disable or delete these hooks. 
  6. Plugins that impact repositories or build systems would need to be reviewed to make sure they are not impacting production systems. For example, if you are using the Mirror plugin, the connections would need to be edited so that pushes to the staging environment are not mirrored out to some external production system.


Last modified on Apr 29, 2024

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.