Data Center 10.2
Versions

Bamboo guardrails

Administering Bamboo

On this page

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

The content of this page applies to Bamboo 10.2. If you're looking for information about a different version, select it from the menu in the top-right corner.

Background

We’re committed to supporting the needs of our largest customers, and this includes continually improving the performance and scalability of our products. The amount of data in your instance can be a factor in performance and stability problems. As your instance grows, so does your risk of performance degradation over time. Often this is a gradual degradation and can go unnoticed until you reach a point where it has a significant impact on your team.

In the table below, we’ve described the performance and stability impacts that we’ve observed and suggested some actions you can take to reduce your risk. The guardrails are based on real-world experiences with some of our largest customers, but won’t necessarily be representative of every organization’s experience.

Ways you can reduce the risk of experiencing serious performance and stability problems may include:

  • application changes, such as upgrading to a newer application version to get the benefit of performance improvements, or changing the way users are managed.

  • infrastructure changes, such as increasing memory, CPU, or running a cluster or mirrors.

  • data cleanup activities to reduce your footprint, such as archiving or breaking up monolith sites.

It’s important to note that these aren’t hard limits, and some of your product instances may already exceed these thresholds. There are a number of factors, including the interplay between different data types, and site load, which will influence whether you experience the potential impacts listed below, and to what degree. As with any type of risk, it’s essential to identify the risk and make a plan, so you can prioritize those actions that will help you reduce the probability of future performance problems.

Definition

Product Guardrails are data type recommendations designed to help you identify potential risks and aid you in making decisions about next steps in your instance optimization journey.

Bamboo guardrails

The following guardrails are provided to help you identify and mitigate scale risks, and make decisions about cleaning up your instance.

Number of environments

Content type

Number of environments

Guardrail

60,000 deploy environments

How to find this number

How to find the number of deployment projects and environments from Bamboo database

Risks

  • Slower performance of UI screens that depend on deployment information. For example, Agent executable environments.
  • Slower performance of REST API endpoints that depend on deployment information. For instance,  /rest/api/latest/agent/.

  • Stability and performance issues in Bamboo due to increased memory footprint. For example: Bamboo application crashes with OutOfMemory errors, utilizes excessive CPU when garbage collecting, and executes expensive database queries.

  • Longer-running background deployment expiry process that may degrade performance in other areas of the application.

  • Increased maintenance cost of the Bamboo application server due to an increase of heap memory to avoid running out of memory, increased database storage and compute to accommodate deployment environment data.

  • Database deadlocking during the expiry process that’s more prevalent on Microsoft SQL Server.

Mitigation options

  • Increase the Java heap size if experiencing crashes due to OutOfMemory errors or performance issues. For example, for 60,000+ deployment environments, we have observed an estimated required heap of 31G.

    • We recommend discussing this issue with Atlassian Support. We can provide insights based on your data, such as garbage collection logging.

  • Ensure you have Global Expiry configured for deployment results for better data hygiene.

    • Re-examine the required retention period for deployment data based on your organization's requirements.

    • Increasing the frequency of the expiry process can help Bamboo keep deployment results to a minimum, which would shorten the deployment expiration time and reduce performance impact on the rest of the application.

    • As the expiry process can be resource heavy and run times can take hours for large instances, it is advisable to plan when it's scheduled to run. For example, if you could schedule it during times of least application usage, you can avoid resource contention and observable performance issues in Bamboo.

  • You may consider whether you require multiple Bamboo instances for separate teams to accommodate excessive deployment environment usage.

Build log sizes / ActiveMQ bottlenecks

Content type

Build logs size (per job)

Guardrail

25Mb or 100000 lines

How to find this number

While viewing a build log, observe the values of the following sentence immediately above the preview panel:

The build generated xxxxxx lines of output.

It is also possible to verify the logs sizes by checking the stored files on the Bamboo server at <bamboo-shared-home>/builds/plan-<number>/download-data/build_logs.

The Disk space hotspots and cleanup best practices article contains a command that may be used to scan the Bamboo Shared folder for large log files, which can be later matched to their respective builds.

Risks

  • Significant delays or lost build triggers, either manually or automatically via commits.
  • Backend and UI instabilities due to excessive memory usage.
  • Unexpected ActiveMQ errors and system crashes.
  • Challenges with Agent communication, including the "connection refused" and "connection reset" errors found in the logs.

Mitigation options

Linked repositories count

Content type

Number of linked repositories

Guardrail

18000

How to find this number

How to obtain the total count of linked repositories present in Bamboo Data Center

Risks

  • Significant delay in converting plan-level repositories to linked repositories.
  • Performance issues when adding a repository at the plan level.
  • Errors when loading repository configurations.
  • Performance issues on the Repository settings page. To view the page, go to settings, then Build resources, and select Repository settings.
  • Long delays in loading the list of linked repositories.

Mitigation options

  • Regularly review and clean up unused or obsolete repositories.
  • Archive or delete repositories that are no longer needed to reduce the load on Bamboo.
  • To reduce the load on the server, perform operations on repositories in batches rather than all at once.

Excessive test cases in the build result

Content type

Excessive test cases in the build result

Guardrail

  • Small instance ( <5 GB Heap ) - 10,000 test cases per job

  • Large instance ( >5 GB Heap ) - 50,000 test cases per job

How to find this number

How to obtain the test case count

Risks

We've observed these problems when operating above this guardrail:

Memory and storage issues
  1. Frequent full Garbage Collection (GC) pauses.
  2. Out of Memory (OOM) error when opening the Tests tab for a build plan due to High heap consumption.
  3. Tables with test case-related data consume a significant amount of space out of the total available storage, leading to storage limit issues.
Performance issues

  1. Long time to process the build results after the build completion.

  2. The expiry process is taking a significant amount of time.

  3. The Build result summary page takes a lot of time to load.


Inconsistent state of buildresults due to large time taken to process them

  1. Build XXX had to be canceled. It was marked as queued but was not present in the queue for (at least) the past 720 seconds.

  2. Build XXX had to be canceled. It was marked as in progress in the database but had no assigned Agents.


Mitigation options

Mitigation options available when exceeding the limit of Bamboo test case count as defined under Bamboo Data Center Guard rails

Size of Git repository

Content type

Size of individual Git repositories

Guardrail

2.0 GB per Git repository

How to find this number

How to determine the size of a Git repository

Risks

When operating above this guardrail, we have observed several issues, including but not limited to:

  • Builds that come from plans linked to the large repository experience delays in getting queued on the Build activity page.

    • This delay has the potential to affect all builds, not just those connected to the large repository, especially for users running Bamboo versions before 9.3.0.

  • Builds consume a substantial amount of time during the Source Code Checkout task on the Agent side.

A less common but more severe side effect involves not just a delay but the disruption of all builds connected to the large repository. This situation arises when the Git operations, triggered by Bamboo to maintain the repository cache or build working directory, time out and are left incomplete. This creates a loop where Bamboo continuously tries to update or rebuild the repository cache (on the Bamboo server/agent) or the build working directory (on the Agent) without success. Learn more about Bamboo Agents and their configuration

Delays and interruptions frequently occur due to prolonged Git operations on large repositories. These happen while Bamboo is updating or rebuilding the repository cache (on the Bamboo server/agent) or the build working directory (on the Agent).

Mitigation options

Limit the repository size to 2.0 GB or less

Limit the repository size to 2.0 GB or less to optimize the speed of Git operations on both Bamboo's server and Agent sides. The following articles outline some steps that can assist you in accomplishing this task:

  1. Reduce the size of Bitbucket cloud repository

  2. Compute various size metrics for a Git repository

  3. Reduce the size of GitLab repository 

Store large files outside your repository

Opt for storing large files outside your repository, such as JFrog Artifactory, Sonatype Nexus, or AWS S3. If it's essential to include the files in the repository, consider using Git LFS.

Note that Git isn’t intended to be used as a backup tool. However, many solutions, such as Arq, Carbonite, and CrashPlan are specifically designed for performing backups.

Ensure that you're running Bamboo 9.3.0 or higher

Ensure that you are running Bamboo 9.3.0 or later, as these versions contain crucial improvements to the change detection process.

Upgrading to version 9.3.0 or later will not resolve the issue entirely. Even after the upgrade, you might encounter delays in builds linked to a large repository being queued. The changes were designed to avoid a situation where prolonged Git operations from a single large repository affect all builds in Bamboo. While builds connected to the large repository may still encounter delays, the positive aspect is that not all of your builds will be affected.

Avoid different URLs in multiple repository configurations pointing to the same repository

Avoid setting up multiple Git repository configurations in Bamboo that point to the same remote repository using different URLs.

This is especially important if the remote repository is larger than 2.0 GB. Bamboo considers these configurations as separate repositories for cache management due to the multiple URLs. If plans associated with these configurations are triggered at the same time, this may lead to delays in builds entering the queue, as Bamboo is still updating or rebuilding the cache for the large repository. This delay can impact all builds, not just those linked to the large repository.

The recommended approach is to have one repository cache folder for each remote repository. Therefore, avoid creating multiple repository configurations in Bamboo that point to a single large repository (> 2.0 GB) with different URLs.

Determine if there are multiple Git repository configurations in Bamboo

Move away from the repository polling trigger

Make sure that none of the plans that are connected to the large repository are actively polling for changes. Check the following articles for more information:

The repository should be set up to notify Bamboo of any changes, rather than Bamboo continuously checking for updates. You can find detailed instructions on how to configure this in the Repository triggers the build when changes are committed article. 

Enable repository caching on agents

Turn on the Enable repository caching on agents option in Bamboo within your large repository configuration. This feature guarantees the creation of a repository copy (cache) on the Agent side. Any modifications to the repository will trigger Bamboo to refresh the cache on the Agent side. This will direct the Source code checkout task to retrieve data from the cache instead of the remote repository.

Don't clean your build working directory

Avoid enabling the Force clean build checkbox for the source code checkout task.

Additionally, don’t enable the Clean working directory after each build option that’s available in the job configuration on the Other options. These settings will force the Agent to perform a fresh checkout of the repository whenever a new build is triggered. Although this may lead to higher disk space consumption (due to the build working directory not being cleaned), it can effectively decrease build times.

Configure Git to ignore LFS objects on the Bamboo server side

If you are using Git LFS on a large repository with Bamboo Specs enabled on Bamboo versions before 9.3.0, you can instruct Git to ignore any LFS objects on the Bamboo server side to optimize Git operations.

This issue has been resolved in Bamboo 9.3.0 and later versions. Bamboo no longer fetches LFS files during a Specs scan.

To exclude all LFS objects from fetch operations, you should add the following line to the ~/.gitconfig file located in the home directory of the user running the Bamboo process:

[lfs] fetchexclude = *

This change might result in:

  • build failures on local Agents that depend on Git LFS files. Builds relying on Git LFS running on remote Agents, elastic Agents, and others will not be affected. Note that local Agents have been removed starting from Bamboo 9.6. Check Bamboo 9.6 release notes

  • Bamboo Specs scanning failures if your Specs code is being tracked using Git LFS.


Adjust SSH proxy timeouts

If you experience timeouts while working with a large Git repository, reach out to Atlassian Support for help in identifying the issue and determining the cause of the timeouts.

Since Bamboo 8.0, you can use a new set of system properties (SSH_PROXY_*) that simplify the adjusting and configuration of timeouts on the SSH proxy used by Bamboo.




Last modified on Dec 19, 2024

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.