Scaling Bitbucket Server for Continuous Integration performance
If you've got CI or other automatic tooling set up to poll Bitbucket Server for changes, you can end up with high load on your Bitbucket Server instance. Consider for instance a CI server that has a number of builds set up for a given repository. Each of those builds polls Bitbucket Server for changes and when it detects a change, it starts a new build. If your CI server supports parallel and/or chained build steps, each of these builds typically results in multiple clone operations of the same repository. The result: lots of polling for changes, and bursts of clones of a repository.
Caching
CI results in highly repetitive calls to Bitbucket Server: polling for changes typically results in the same response 90% of the time and a build triggers a number of identical clone calls to Bitbucket Server. Both operations can benefit greatly from caching when you experience repetitive load from CI.
On this page:
Limitations
- Caching can be applied to 'ref advertisement' (polling for changes), clone and shallow clone requests only. Fetch/pull operations are not cached, but these operations will still benefit from the 'ref advertisement' cache.
As a precaution, ref advertisement caching is disabled by default. The openness of the plugin system means that plugins (or manually installed git hooks) could be updating refs in repository without the caching plugin detecting these changes. The result would be a stale refs cache and failing clone/fetch operations. However, if you know that there are no plugins or git hooks installed that make changes to the repository directly, you can enable the ref advertisement caching using the system properties listed in the Configuration section below. Note that as an additional precaution, the ref advertisement caches are configured to automatically expire after a three minutes.
- Git 2.18 introduced version 2 of the Git wire protocol, which is fully supported by Bitbucket Server's caching. Please note, that due to differences in the protocols, Git clients using version 1 of the wire protocol cannot use caches generated for clients using version 2 of the wire protocol, and vice versa. As a result, when there's a mix of clients using version 1 and 2 of the wire protocol, disk usage for caching can be higher and cache efficiency can be lower, compared to when all clients use the same version of the wire protocol.
Considerations
Cache data is stored on disk for both ref advertisements and for clone operations for a configurable period of time. Since large instances can potentially consume an entire disk the SCM Cache Plugin monitors remaining disk space and is automatically disabled when the configured minimum free disk space is reached.
See Bitbucket Server config properties for descriptions of the available system properties.
Configuration
The SCM Cache Plugin for Bitbucket Server can be configured using either the REST endpoint (some settings, not all) or the system properties in <BITBUCKET_HOME>/bitbucket.properties. Settings configured through REST are considered before the settings in bitbucket.properties.
REST API
| Method | Url | Description |
|---|---|---|
| GET | /rest/scm-cache/latest/config/protocols | Retrieve the protocols for which caching has been enabled. |
| PUT | /rest/scm-cache/latest/config/protocols/{protocol} | Enable caching of hosting requests for the protocol (HTTP or SSH). |
| DELETE | /rest/scm-cache/latest/config/protocols/{protocol} | Disable caching of hosting requests for the protocol (HTTP or SSH). |
| GET | /rest/scm-cache/latest/config/refs/enabled | Retrieve whether ref advertisement caching is enabled (true) or disabled (false). |
| PUT | /rest/scm-cache/latest/config/refs/enabled/{status} | Enable (status = true) or disable (status = false) ref advertisement caching. |
| GET | /rest/scm-cache/latest/config/refs/ttl | Retrieve the expiry in seconds for the ref advertisement caches. |
| PUT | /rest/scm-cache/latest/config/refs/ttl/{expiryInSec} | Set the expiry in seconds for the ref advertisement caches. |
| GET | /rest/scm-cache/latest/config/upload-pack/enabled | Retrieve whether clone caching is enabled (true) or disabled (false). |
| PUT | /rest/scm-cache/latest/config/upload-pack/enabled/{status} | Enable (status = true) or disable (status = false) clone caching. |
| GET | /rest/scm-cache/latest/config/upload-pack/ttl | Retrieve the expiry in seconds for the clone caches. |
| PUT | /rest/scm-cache/latest/config/upload-pack/ttl/{expiryInSec} | Set the expiry in seconds for the clone caches. |
| GET | /rest/scm-cache/latest/caches | Retrieve information about the current caches: size, number of cache hits and misses, etc. |
| DELETE | /rest/scm-cache/latest/caches | Clear all caches. |
| GET | /rest/scm-cache/latest/caches/{projectKey}/{repoSlug} | Retrieve information about the caches for the repository identified by projectKey and repoSlug: size, number of cache hits and misses, etc. |
| DELETE | /rest/scm-cache/latest/caches/{projectKey}/{repoSlug} | Clear the caches for the repository identified by projectKey and repoSlug. |