Summary

When there is a single large repository with multiple plans using it for the build activities, including the usage of multiple local agents, you could see the build plan queues for an extended duration. This could be due to the Change detection and Plan execution threads taking too long in the Bamboo server while the code checkout is happening.

Diagnosis

Sample Logs on change detection taking too long

2022-01-23 10:55:39,356 DEBUG [7-DelayedChangeDetectionThread:pool-13-thread-890] [DefaultChangeDetectionManager] Change detection for repository "MyRepo" in plan "POR1-PLA123" took 21.70 min
2022-01-21 10:58:00,730 ERROR [10-BAM::PlanExec:pool-16-thread-7] [DefaultChangeDetectionManager] Change detection for plan PRO1-PLAN2 took 23.41 min

Cause

In case the Linked repository is enabled to use the cache (by the "Enable repository caching on agents" checkbox), it will create a cache on the Bamboo server under $BAMBOO_HOME/xml-data/build-dir/_git-repositories-cache directory.

Bamboo DefaultChangeDetectionManager will detect the changes by connecting to the git cache on the local disk. If there are a number of plans using the same repository, the build execution threads will wait for the thread that is accessing the git cache. A change detection delay is caused by the high concurrent number of requests to the same Git repository coming from multiple local agents or by several Plan execution threads used internally by the Bamboo Server. Since the local agent and the Plan execution threads will use the same Git cache directory in Bamboo Server when performing a checkout, in parallel, that would lock the repository and Bamboo will have to wait for the cache lock to be free to proceed with the next build or code detection activities. 

Environment

• Multiple build plans using the same single large repository
• The linked repository is enabled to use caching on agents
• Use of local Agents that are performing source code checkouts on that same repository

Solution

Option 1 - Migrate to Remote Agent usage so the Git cache will be distributed

We recommend setting up remote agents in another server and disabling all local agents that are checking out code. As the remote agent has its own local Git cache, it will not be requesting it from the Bamboo server and will not be blocking the build queue.

Option 2 - Disable Git Cache on the linked Repository

Consider disabling the Git cache on the large repository for some time and see how the Bamboo server performs. Please keep in mind that by disabling caching on the repository, you may see an increase in the network traffic and load on the Bitbucket server; build times may increase as the local agents may not have a local copy of the repository and will need to fetch it from Bitbucket on their first run (or every run if the Plans required a clean build environment). The git fetch command will appear more often as the local agents will pull from your Bitbucket repository more often.

Disabling the cache while having its downsides, can help to alleviate lock contention on the local Git cache in Bamboo and help it to dispatch the builds in a timely fashion. The local git cache is not optimised for large single-repository usage with local agents as it imposes a bottleneck on the threads and ends up slowing down things.

Option 3 - Configure multiple git URLs for the single large repository and have multiple linked repositories to use them across builds

Bamboo uses a combination of the git username and hostname to create a hash. That hash will be used to create a directory that will contain the cached Git content. By creating different hash based on the mentioned criteria, you can end up with multiple local caches that are less prone to bottlenecks.

• Using Forks:
  • Fork your large repository, giving the forks distinct names. Make sure to get them synced with their "master."
  • Create multiple linked repositories in Bamboo pointing to the forks
  • The linked repositories need to use "git" as a repository type. Do not use Bitbucket
  • Distribute the expensive build plans across the new, fork-based, linked repositories
  • This will create a separate local Git cache
• Using DNS:
  • Add multiple A or CNAME records to your git repository hostname
  • Create multiple linked repositories pointing to each individual A/CNAME record
  • The linked repositories need to use "git" as a repository type. Do not use Bitbucket
  • Distribute the expensive build plans across the new, DNS-based, linked repositories
  • This will create a separate local Git cache