Bitbucket Data Center FAQ

This page covers some of the most frequently asked questions we hear at Atlassian about Bitbucket Data Center. If you have questions that aren't answered by this page, please contact your Atlassian Enterprise Advocate, Atlassian's Enterprise team, or your Atlassian Expert.

What is Bitbucket Data Center?

Does Bitbucket Data Center provide disaster recovery at an off-site location?

Yes. Bitbucket Data Center can be configured to implement and manage a disaster recovery strategy. Atlassian provides guidance for setting up and deploying a cold standby instance that can be configured to continuously backup your data. See the Disaster recovery guide for Bitbucket Data Center for guidance on including Bitbucket Data Center as part of your disaster recovery plan.

Does Bitbucket Data Center support Windows?

No, Windows is not supported. See Supported platforms for related information.

Can I deploy Bitbucket Data Center on Microsoft Azure?

Yes, Bitbucket Data Center supports Microsoft Azure. For more information, see Deploy Bitbucket Data Center in Azure. 

Where can I download the Bitbucket Data Center distribution?

From https://www.atlassian.com/software/bitbucket/download, the same location as Bitbucket Server. Bitbucket Data Center and Bitbucket Server are in fact the same build of Bitbucket, the Clustering and Smart Mirroring functionality just needs a Data Center license key to enable.

Clustering

What are the benefits of a clustered Bitbucket Data Center instance?

Bitbucket Data Center appears to users as a single instance of Bitbucket, but "under the hood" it is actually a cluster of multiple machines ("cluster nodes") each running the Bitbucket web application, behind a load balancer. This provides important benefits over a single node Bitbucket Data Center:

There are lots of different kinds of "clustering" out there. Which one is Bitbucket Data Center, exactly?

Bitbucket Data Center is a full ("active-active") cluster configuration. All cluster nodes participate equally in servicing user requests and other processing. Load is balanced fairly between the nodes. If one node goes down, failover to the other nodes happens automatically without manual intervention, typically within seconds. When a node joins (or re-joins) the cluster, it begins servicing user requests and other processing automatically, as soon as it comes online. 

"Active-active" clustering is the key to the performance at scale and high availability advantages of Bitbucket Data Center. This is not the same as "active-passive" and other forms of clustering where data must be replicated to "cold standby" servers. 

Can a Bitbucket Data Center instance have cluster nodes in more than one geographic location, such as the Americas, Europe, and the Asia-Pacific region?

No. A Bitbucket Data Center cluster is designed to support cluster nodes in a single geographic location.

You can, though, use Mirrors to provide instances in different geographic locations that accelerate Git clones and fetches (typically the longest running operations) for your remote teams.

Which apps or plugins are compliant with Bitbucket Data Center?

The best and most up to date way to find out whether a plugin is supported in Bitbucket Data Center is to check for the "Data Center" badge on the corresponding Atlassian Marketplace listing.

I've developed a Bitbucket app, how do I make it compatible with Bitbucket Data Center?

Some apps may not need any development work at all, and will largely "just work" in Bitbucket Data Center without modification. But any app that assumes (implicitly or explicitly) that it is running in a single JVM may need some development work to be made compatible. The Bitbucket API for plugin developers includes a number of new services that make it easy to write cluster-safe apps. These services generally provide similar API's to standard Java API's such as ConcurrentMap, Cache, and  Lock, so in many cases an app can be made cluster safe by just changing a few classes.  See Making cluster-safe plugins for more information.

Do integrations with other Atlassian products, such as Jira and Bamboo, need special configuration in Bitbucket Data Center?

No. All integrations with other Atlassian products, including Jira and Bamboo, work just as they do with Bitbucket Server.

Clustering performance at scale

How many cluster nodes should I deploy for my organization?

You can review our Bitbucket Data Center Performance page for some guidelines, and you can contact an Enterprise Expert Partner for a more specific recommendation for your environment. The most thorough way to determine this, however, is with your own test environment: start with two cluster nodes and upgrade to more as necessary.

Won't NFS be the bottleneck in my Bitbucket Data Center instance?

It can be, especially if your file server is under-resourced or misconfigured. But in Atlassian's own Bitbucket Data Center Performance  testing, we have found that a properly resourced and configured NFS server can perform well even under very heavy load. 

Clustering high availability

What's the minimum number of cluster nodes I should use?

In order for hot failover to be possible, we recommend that you have a minimum of two nodes and a maximum of 8 nodes.

How quickly does failover work in Bitbucket Data Center?

Because Bitbucket Data Center provides full active-active clustering, failover if a node goes down can happen in seconds. You do not have to wait for a "cold standby" to start up, for all the data to replicate consistently, or for traffic redirection to propagate in the DNS. All this is handled automatically and seamlessly by Bitbucket Data Center, without intervention by the system administrator. For more detailed information, see Failover for Bitbucket Data Center.

If a cluster node goes down, what happens to users who are logged in to that node?

If a cluster node goes down, most load balancers are able to detect the failure and direct traffic to the other nodes within seconds. Users who were actively using that cluster node when it went down, however, may notice the failure in a number of ways:

• Any open Git clone, fetch, or push connections being handled by the failing node may be broken (but will succeed if retried). 

• Any users whose session was directed to the failing node will be logged out (by default), and will have to log in to Bitbucket again.  This extra login, though, can be avoided in a few ways:
  • Users who check "Keep me logged in" when logging in should not have to log in again. 
  • If you have enabled Single sign-on (SSO) with Crowd, then users whose Crowd directory has been configured for SSO should not have to log in again. See Connecting Bitbucket to Crowd for more information. 
  • If you have your own custom (in house or third party) authentication plug-in, then you can avoid the extra login provided your plug-in is written to be cluster aware. 

• Any other data stored in the session (such as fields entered in one page of a multi-page form) on the failing node may be lost and have to be re-entered. Bitbucket and most Atlassian-supplied apps do not store any such state in the session, but third party apps that were not designed for Bitbucket Data Center may. 

• Any background work running on the failing node, such as pull request rescopes, comment drift calculations, or user directory synchronization, may take a bit longer to complete, as this work is taken over by another cluster node. 

For more detailed information, see Failover for Bitbucket Data Center.

Can Bitbucket Data Center be upgraded without downtime (i.e., a rolling upgrade)?

Bitbucket Data Center allows rolling upgrades without downtime only for bug fix upgrades (for example, from Bitbucket 7.9.0 to 7.9.4). For platform upgrades (such as Bitbucket 7.0 to 8.0) or feature upgrades (such as 7.4 to 7.8), rolling upgrades are not available at the moment. Learn more about how to upgrade Bitbucket without downtime and see Bitbucket Data Center upgrade guide.

Clustering technical requirements

How do I move from my Bitbucket Server instance to Bitbucket Data Center?

Simply follow the steps in Upgrade from Bitbucket Server to Bitbucket Data Center. Don't forget to update your license key to a Bitbucket Data Center key.

Does Bitbucket Data Center clustering support non-multicast-enabled networks?

Yes. Multicast discovery is one common method for cluster nodes to find each other, but if your network does not support multicast, then you can set  hazelcast.network.tcpip=true  in your bitbucket.properties file, and  hazelcast.network.tcpip.members  to a comma-separated list of cluster nodes instead. See Installing Bitbucket Data Center.

Does Bitbucket Data Center clustering support CIFS/SMB, iSCSI, GlusterFS, or other file system mount types?

Not at this time. Please contact us. 

Does Bitbucket Data Center clustering support Amazon Web Services (AWS)?

Yes! Atlassian and Amazon even publish a standard, automated reference deployment for Bitbucket Data Center. This includes a Quick Start guide and AWS CloudFormation template that launches, configures, and runs the AWS compute, network, storage, and other services required for a complete production deployment of Bitbucket Data Center in AWS, using AWS best practices for security and availability. See Amazon Quick Starts and Bitbucket Data Center in AWS.

Does Bitbucket Data Center clustering support Amazon EFS?

Yes, Bitbucket Data Center supports Amazon EFS. You can use it if you’ve migrated all Git repositories from the Bitbucket shared home directory to Bitbucket Mesh. Learn more about the support for cloud platforms.

Which versions of NFS are supported by Bitbucket Data Center?

NFSv3 is supported. If you’re using Bitbucket Mesh and have migrated all Git repositories to it, you can use NFSv4 for the shared home directory.

Which load balancers are supported by Bitbucket Data Center?

There are too many vendors providing load balancers (either hardware or software) for Atlassian to provide specific guidance on all of them. You can choose any load balancer you like, provided it supports:

• both HTTP mode (for web traffic) and TCP mode (for ssh traffic),
• SSL termination to offload HTTPS encryption and decryption from Bitbucket, and
• cookie-based session affinity ("sticky sessions").

If you have no specific preference for your load balancer, Installing Bitbucket Data Center provides instructions for haproxy, a popular Open Source software load balancer. 

Which databases are supported by Bitbucket Data Center?

All the usual databases supported by Bitbucket Data Center (see Supported platforms) with one exception: MySQL (all versions) is not supported for Bitbucket Data Center at this time due to inherent deadlocks that can occur in this database engine at high load.  If you currently use MySQL, you should first migrate your data to another supported database such as PostgreSQL. See Connect Bitbucket to an external database for more information. 

Can I migrate databases for a Bitbucket Data Center instance while clustered?

No. The Database Migration Wizard is not supported in Bitbucket Data Center instances while more than one cluster node is running. To migrate databases for a Bitbucket Data Center instance, you should perform the migration first, before starting multiple cluster nodes.

What methods are supported to backup a Bitbucket Data Center clustered instance?

Zero downtime backup is a technique introduced in Bitbucket 4.8 which backs up a Bitbucket instance without requiring it be locked for maintenance. It requires your shared home directory to be on a file system volume capable of atomic (block level) snapshots, and your database to be capable of either atomic backup or point in time recovery. Use of these technologies allows you to take backups as often as you need (e.g., hourly), without inconveniencing your users and build agents with frequent downtimes.

DIY Backup is fully supported in all versions of Bitbucket Data Center. It doesn't matter whether you point your DIY Backup script to the URL of the Bitbucket instance (i.e., the load balancer) or a specific cluster node, provided the machine where your DIY Backup runs has connectivity to that endpoint. All cluster nodes will drain their running database and filesystem operations, and display the "Performing Backup" page for as long as your DIY Backup script runs with Bitbucket under lock.  Most DIY Backup scripts that work with Bitbucket Server should work with little or no modification with Bitbucket Data Center. 

The Bitbucket Backup Client is not compatible with clustered Bitbucket Data Center instances, even if you switch back to a single node. There are currently no plans to allow the Bitbucket Backup Client to work on clustered instances of Bitbucket.

Be sure to read the article Data recovery and backups for details about the different methods of backing up Bitbucket Data Center.

Do I have to back up Elasticsearch data as part of my backup strategy?

No. If you have a remote Elasticsearch instance then search indexes are maintained in Elasticsearch's data directory, but you don't have to include this in your backup as it can be completely rebuilt if necessary from the data in your home directory and database. Note that for large instances, rebuilding the Elasticsearch index can potentially take several hours. Search functionality could be degraded while an Elasticsearch index is rebuilt. 

Bitbucket Mesh

What are the benefits of Bitbucket Mesh?

Bitbucket Mesh is a distributed, replicated, horizontally scalable Git repository storage for improved performance and high availability. Mesh spreads Git hosting and data processing across multiple independent nodes, which significantly decreases I/O latency and offers confidence to scale without worrying about performance degradation. Learn more about Bitbucket Mesh

Is there a minimum number of Mesh nodes?

In order for high-availability to be possible, we recommend having a minimum of three Mesh nodes. There is no maximum on the number of nodes.

Can I continue to run the Bitbucket Data Center with NFS for Git storage, or do I have to deploy Mesh nodes after upgrading to 8.x?

You can indeed upgrade to 8.x and continue to use a shared file system. Mesh is something you can choose to use, and without opting in, it all should just work like it did pre-8.0. Under the hood, there are some changes to the way we access those repositories. When Bitbucket starts, it will now also start something we call the sidecar; this is a second Java process that is now responsible for all interactions with the repositories on disk.

Does Bitbucket Data Center still use Network File System (NFS)?

Yes, it depends on NFS to store latency-tolerant data (Git LFS, plugins, attachment, avatars, etc). See supported platforms.

Does Mesh support LFS objects?

A shared file system is still required to store non-repository data. LFS objects in your Git repositories are also still stored on the shared file system, and we’re tracking the ability to migrate that data in BSERV-13269 - Getting issue details... STATUS

Can I deploy Mesh nodes in different locations or availability zones?

Currently, we will not be supporting deploying Mesh nodes into multiple availability zones. Just like the shared file system based deployments, the Mesh nodes (that is, the repository storage) and the application nodes must be co-located. We are exploring the ability to support the deployment of Mesh nodes to multiple Availability Zones in BSERV-13269 - Getting issue details... STATUS

How should Mesh nodes be upgraded?

Bitbucket Data Center nodes and Bitbucket Mesh nodes are separate applications and have cross-version compatibility. It is possible to upgrade them independently.

Can 3rd-party apps have direct access to Mesh nodes?

No. Our Java and REST APIs will remain the only publicly available mechanisms for interacting with the Bitbucket Mesh nodes.

Can I deploy Bitbucket Mesh in AWS?

Yes. Bitbucket Mesh supports AWS. 

Can I deploy Bitbucket Mesh on Azure?

Yes, Bitbucket Mesh supports Azure.

Does Bitbucket Mesh support Windows?

No, Windows is not supported. See Supported platforms.

Smart Mirroring

What are the benefits of Smart Mirroring?

Smart Mirroring is a term we use to cover standalone mirrors as well as mirror farms. It can greatly improve Git clone speeds for distributed teams working with large repositories. Large repositories that take hours to clone from a Bitbucket instance over the Internet from the other side of the world, can take minutes when cloned from a local mirror on a fast network. Mirror farms increase your CI/CD capacity and the clustering of mirrors provides higher availability.

See Mirrors for more information. 

Do my mirrors need to be on the same version as my primary Bitbucket instance?

No, they don't. Your primary Bitbucket instance and all existing mirrors can run any version of Bitbucket, provided it's 4.2.0 or higher. For mirrors that are newer than 6.7, they can't point to a primary older than Bitbucket version 6.7. Additionally, for mirrors that are newer than 8.13, they can't point to a primary older than Bitbucket version 8.13.

Do my mirrors need to share a file system or database with my primary Bitbucket instance?

No. Mirrors and mirror farms do not share a file system or database with the primary Bitbucket instance.

What if data is stored on ephemeral storage?

If all data that's stored locally is lost, the mirror or mirror farm will lose authentication with the primary Bitbucket instance and will have to be reinstalled, so this needs to be kept in mind if data is stored on ephemeral storage. 

Does my primary Bitbucket instance need to be a cluster to use Smart Mirroring?

No. You must have a Data Center license to use Smart Mirroring, but you don't actually have to have more than one cluster node in your primary Bitbucket instance.

Can I mirror repositories from non-Bitbucket servers using Smart Mirroring?

No. Try one of the several mirroring apps for Bitbucket listed on Atlassian Marketplace.

Once I've cloned from a standalone mirror, am I supposed to push to the mirror as well?

Yes, you can push to the repository using the same URL and Bitbucket relays the push to the primary repository.

Does Smart Mirroring provide higher availability if my primary Bitbucket instance is down or unreachable?

Yes. Auth caching was added in 5.1, so if the primary instance is down, authentication can still be done for as long as the cache lives through its configuration.

In the case where credentials are no longer cached, the mirror won’t be able to serve requests until the primary instance is available again.

Can I point my continuous integration (CI) builds to mirror farms?

Yes you can. This will allow you to scale your teams' CI/CD capacity simply by adding additional nodes to your mirror. This frees up the load on the primary and mirror farms can scale the needs of your CI system. In addition using mirror farms can help protect your upstream against miss behaving CI systems.

For information on using Smart mirror farms with bamboo see Smart Mirroring for more information.

Can I install apps on a mirror?

No.

Does Git Large File Support (LFS) work with Smart Mirroring?

Yes. When a file managed by LFS is first requested from a mirror, it will be streamed to the user and cached locally for future requests. This strikes a balance of performance, where commonly requested files are cached for fast access, and storage efficiency, where historical files managed by LFS that are not of interest to mirror users are not actively synced.

How secure is Smart Mirroring?

The short answer to this is: No less secure than your primary Bitbucket instance. 

The slightly longer answer is: Smart Mirroring has been designed to require secure communication throughout and restrict all functionality to the appropriate privilege level (e.g., system administrator) to ensure the security of all your sensitive information. But mirrors must necessarily store copies of your repository data, so you must ensure that not only your primary Bitbucket instance but all your mirrors have been set up with the best security practices to prevent unauthorized access via some other route. Bitbucket prevents or warns about some common misconfigurations that are considered insecure. For example, if you attempt to set up a new mirror without SSL Bitbucket will not let you proceed, or if you configure Bitbucket's user account to allow all users read permission to repository data in the ${BITBUCKET_HOME} directory Bitbucket will give you a warning. But it does not catch every possible misconfiguration, and it is your responsibility to set up mirrors with the same stringent security practices as your primary Bitbucket instance.

Among the best practices recommended for both mirrors and standard Bitbucket installations are:

• Ensure the primary instance and all mirrors have valid SSL certificates and do not allow requests over plain HTTP.
• Lock down the mirror environments against unauthorized login or other access by anyone except a system administrator.
• Never run the Bitbucket application as root, either on standard instances or mirrors.
• Configure the Bitbucket user account (typically atlbitbucket) securely, for example by allowing login only with SSH keys instead of passwords and configuring a umask that doesn't allow other users to read and write files by default.
• Make sure your operating system and all packages/components (e.g., Java, Git, proxies) are fully patched with the latest updates.
• Avoid running other services that may be vulnerable on the same environment as your Bitbucket primary instance or mirrors.

Watch the Supported platforms page for minimum versions of third party software components and known versions to avoid. See the documentation for your vendor's operating system for more information about securing services in that environment. 

Smart Mirroring performance

How much faster is a mirror than my primary Bitbucket instance?

It depends. The speedup will vary with repository size, network bandwidth and latency, how well provisioned and heavily loaded your mirror is, and how fast the CPU is on the user's machine running git. For small repositories and fetches there may not be much in it. For moderately large repositories with good network connections the speedup will be a few times (or in absolute terms, measured in minutes).  For enormous (multi-GByte) repositories with not-so-good network connections the speedup will be considerably more, and in absolute terms could easily be measured in hours.

No, seriously, how much faster is a mirror?

We use local mirrors all the time, and we have some fairly small repositories (100's of MBytes) that normally take 2 minutes to clone from the other side of the world take about 40 seconds from a local mirror (3x speedup). Larger repositories (> 4 GBytes) that take over an hour to clone from the other side of the world take 2 – 3 minutes from a local mirror (anywhere from 25x to 50x speedup).

What about mirror farms?

In our performance testing, we used environments that represent large enterprises. The results for a sample repository were a linear increase in concurrent clone operations. This is due to the architectural design, in which each node serves data from local storage instead of using one shared NFS storage in a mirror farm.

How much CPU, memory, I/O, etc. does a mirror need?

Just as with your primary Bitbucket Data Center instance, your mirrors need to be provisioned with enough CPU, memory, and I/O resources to handle their peak workload. See Scaling Bitbucket Data Center for more information.

A mirror of course needs as much disk space for repository data as the space those repositories take up on primary Bitbucket instance. (Initially the mirror may use a little less space for repository data than the identical repositories on the primary instance, due to Git's garbage collection and repacking, but this difference will tend to stabilize with time.) As with standard Bitbucket instances, provisioning more disk space is better for performance, as Bitbucket uses some free disk space for its SCM cache. 

Will mirrors increase or decrease the load on my primary Bitbucket instance?

It depends. Mirrors take some load (possibly a lot) off their primary instance as every clone or fetch from a mirror represents an operation that the primary instance doesn't have to handle. On the other hand mirrors also add a bit of load to their primary instance as they receive webhook notifications informing them of pushes and other events on all the repositories they are mirroring, and periodically perform fetches from the primary to keep those repositories in sync. This additional load is usually small, but if you have a lot of mirrors mirroring repositories that receive a lot of pushes and/or don't get cloned very much, it can add up. 

The time when mirrors generally add the most load to your primary instance is when they are first set up to start mirroring. This is because they start with no data and have to pull everything down from the primary instance from scratch. Bitbucket mirrors are designed to perform this initial sync of repository contents gradually so as not to place a lot of load on the primary instance all at once, but under some circumstances the added load can be significant – especially if your primary instance is already at or near capacity. To minimize the load and impact of mirrors on your primary instance, here's a few common sense guidelines:

• Know the load and capacity of your primary Bitbucket instance, and tune its SCM cache and scm-hosting ticket limits appropriately. See Scaling Bitbucket Data Center for Continuous Integration performance and Bitbucket Server is reaching resource limits for more information.
• Provision a bit of spare capacity on your primary Bitbucket instance. The number of concurrent Git hosting operations sustainable by your instance is directly related to the amount of available CPU, memory, and IOPS resources. More free disk space will also help, as this is used by SCM cache to reduce load on other resources.
• If possible, spin up new mirrors at periods of lighter than usual load (e.g., weekends or times when most users and build agents aren't active), not at periods of peak load. 
• Avoid spinning up multiple new mirrors to mirror everything all at the same time. Spin them up one at a time, spaced some time apart. 
• If some remote locations only need to work with some projects regularly, consider mirroring only the projects they need at that location, instead of all projects.
• Once your mirrors are available, encourage your users to use them wherever possible so as to take as much load as possible off of the primary instance. 

Do mirrors sync everything?

No. Mirrors only synchronize the repositories in the Projects you specify. An option to sync all projects is available. Note that not all refs are synced to Smart Mirrors in real time. Certain refs, like those found under refs/pull-requests/, are only synced when a new change is pushed to the repository. All public refs, like branches, found under refs/heads/, and tags, found under refs/tags/, are synced in real time.  

Smart Mirroring technical requirements

Can I use plain (unencrypted) HTTP for my mirror?

No. Both your primary Bitbucket instance and all your mirrors must use secure HTTP (HTTPS) and have valid SSL certificates issued by a Certificate Authority (CA). This is a strict requirement of Smart Mirroring on both the primary instance and all mirror(s), and cannot be bypassed. The mirror setup wizard will not let you proceed if either the mirror or the primary instance does not have a valid SSL certificate.  

Can I disable SSH when I use Smart Mirroring?

No. Mirrors use SSH to keep their repositories synchronized with the primary instance and cannot use HTTP or HTTPS for this. It is also not possible to disable SSH access to mirrors by users. See Enabling SSH access to Git repositories for instructions on enabling SSH access on your primary instance.

Can I clone or fetch from a mirror using my SSH key or my HTTPS credentials?

Either. Or both. Mirrors allow clones and fetches over both SSH and HTTPS, and provide identical authentication mechanisms (SSH key, project/repository access key, HTTP credentials) and authorization policies (global permissions, project permissions, repository permissions, and/or other customizations) as the primary instance.

Do mirrors support CAPTCHA?

Yes. Mirrors don't actually do the CAPTCHA themselves, though, they delegate all this up to the primary Bitbucket instance. So it doesn't matter whether the failed authentication attempts are on the primary instance or a mirror, if you perform too many of them it will trigger CAPTCHA of the account on the primary instance, and you'll need to pass a CAPTCHA challenge to unlock it. 

What external databases are supported by Smart Mirroring?

There is no need to set up an external database in the mirror itself. The internal H2 database is sufficient.

The primary Bitbucket instance, of course, should use one of the supported external databases listed in Supported platforms. 

Does Smart Mirroring support Amazon Web Services (AWS)?

Yes.

What license do I need for my mirrors?

You don't need any license in the mirror itself. Your primary Bitbucket instance, though, must have a valid Data Center license for mirrors to work.

How do I back up my mirrors?

Mirrors do not really store any persistent state that can't be reconstructed by just setting it up again with the same primary Bitbucket Data Center instance from scratch. There is no need for elaborate backup strategies for mirrors. You should, of course, back up its important configuration files such as SSL certificate, server.xml, config/ssh-server-keys.pem, bitbucket.properties file, and so on in a safe place. But these configuration files typically do not change while the mirror is running, so ongoing backup is not required.

How do I upgrade a mirror node?

The same way as a standard Bitbucket Server instance, see the Bitbucket Server upgrade guide.

What happens if we stop paying for the Bitbucket Data Center subscription service?

You can still access your information in Bitbucket. However, you aren't able to push or make any other modifications in your instance. 

Troubleshooting Bitbucket Data Center clustering

I get "Lock file .../shared/.lock cannot be obtained in home directory." What's going on?

A common reason for this error is that your NFS lock service is disabled or misconfigured. Make sure the lockd, portmap, and dbus services are all running on your NFS server, and restart Bitbucket. See  Installing Bitbucket Data Center for more information.

If you still have leftover .lock file(s) after shutting down all your instances of Bitbucket, it's safe to delete them and start over.

Does Bitbucket really lock the shared home directory?

Yes, Bitbucket creates .lock files in both ${BITBUCKET_HOME} and ${BITBUCKET_HOME}/shared.

Bitbucket Server acquires an exclusive lock on both directories, preventing any other instances of Bitbucket (Server or Data Center) from using them.

Bitbucket Data Center acquires an exclusive lock on ${BITBUCKET_HOME} (preventing any other Bitbucket instances from using it) and a shared lock on ${BITBUCKET_HOME}/shared (allowing it to be shared with other Bitbucket Data Center cluster nodes but not Bitbucket Server instances).

These checks might seem strict, but they are in place to make sure that data loss through accidentally misconfiguring Bitbucket instances to use the same home directory can never happen. 

My Bitbucket Data Center instance gets slow or unresponsive under heavy load, and in my logfiles I see a lot of "org.hibernate.exception.JDBCConnectionException: Could not open connection" or "java.sql.SQLException: Timed out waiting for a free available connection." What's going on?

These errors mean Bitbucket is running out of connections to the database, which could be due to a number of underlying causes:

1. Your database may be misconfigured not to allow enough concurrent connections. Bitbucket by default uses up to 80 connections  per cluster node , which can exceed the default connection limit of some databases. For example, in PostgreSQL the default limit is usually 100 connections. If you use PostgreSQL, you may need to edit your  postgresql.conf  file, increase the value of  max_connections, and restart Postgres. See Installing Bitbucket Data Center for more information. 
2. The system clocks on your cluster nodes may be drifting out of synchronization or may be being tampered with. This can sometimes be an issue in virtualized environments if time synchronization between the host OS and guest OS is misconfigured. Make sure all your cluster nodes are configured with the same timezone and are running the NTP service with identical configuration. See Installing Bitbucket Data Center for more information.  

If this problem persists, please contact Atlassian Support.

My Bitbucket Data Center instance gets slow or unresponsive for no apparent reason, and in my logfiles I see a lot of "com.hazelcast.core.OperationTimeoutException: No response for 60000 ms. Aborting invocation!" or "c.a.s.i.s.s.DefaultPublicKeyAuthenticator Timed out while authenticating SSH user at ...". What's going on?

These errors mean that one or more Bitbucket cluster nodes can't talk to each other over the network, which could be due to a number of underlying causes:

1. Network failures. Check that all your cluster nodes can ping each other without too much latency or packet loss, and that traffic to and from port 5701 is allowed by your router and firewall configuration. 
2. The Java virtual machines running Bitbucket may be running out of memory. Try following the guidelines in Scaling Bitbucket Data Center and How to debug Out of Memory Heap Space.
3. The system clocks on your cluster nodes may be drifting out of synchronization or may be being tampered with. This can sometimes be an issue in virtualized environments if time synchronization between the host OS and guest OS is misconfigured. Make sure all your cluster nodes are configured with the same timezone and are running the NTP service with identical configuration. See  Installing Bitbucket Data Center  for more information.   

If this problem persists, please contact Atlassian Support.