Scaling Stash for Continuous Integration performance

If you've got CI or other automatic tooling set up to poll Stash for changes, you can end up with high load on your Stash server. Consider for instance a CI server that has a number of builds set up for a given repository. Each of those builds polls Stash for changes and when it detects a change, it starts a new build. If your CI server supports parallel and/or chained build steps, each of these builds typically results in multiple clone operations of the same repository. The result: lots of polling for changes, and bursts of clones of a repository.

Caching

CI results in highly repetitive calls to Stash: polling for changes typically results in the same response 90% of the time and a build triggers a number of identical clone calls to Stash. Both operations can benefit greatly from caching when you experience repetitive load from CI.

Stash 2.5, and later versions, ship with the SCM Cache plugin already bundled. The plugin is enabled by default, but note that ref advertisement is disabled by default – see the 'Limitations' section below. The plugin is also available from the Atlassian Marketplace.

On this page:

Related pages:

Limitations

Caching is applied to the HTTP protocol only. If your CI builds access Stash over SSH, consider switching to HTTP/HTTPS to benefit from caching. Requests over SSH will not benefit from caching.
Caching can be applied to 'ref advertisement' (polling for changes), clone and shallow clone requests only. Fetch/pull operations are not cached, but these operations will still benefit from the 'ref advertisement' cache.
As a precaution, ref advertisement caching is disabled by default. The openness of the plugin system means that plugins (or manually installed git hooks) could be updating refs in repository without the caching plugin detecting these changes. The result would be a stale refs cache and failing clone/fetch operations. However, if you know that there are no plugins or git hooks installed that make changes to the repository directly, you can enable the ref advertisement caching using the system properties listed in the Configuration section below. Note that as an additional precaution, the ref advertisement caches are configured to automatically expire after a minute.

Enabling and disabling caching

Enable SCM cache from the admin area in Stash. Click Manage Add-ons (under 'Add-Ons') and filter for system add-ons. Click SCM Cache Plugin for Stash and then either Enable or Disable.

Configuration

The SCM Cache plugin for Stash can be configured using either the REST endpoint (some settings, not all) or the system properties in <STASH-HOME>/stash-config.properties. Settings configured through REST are considered before the settings in stash-config.properties.

System properties settings

Setting	Default value	Description
plugin.stash-scm-cache.expiry.check.interval	300 (5 min)	Controls how frequently expired caches are checked and deleted from disk. Time is in SECONDS.
plugin.stash-scm-cache.minimum.free.space	1073741824 (1 Gb)	Controls how much space needs to be available on disk (specifically under <STASH-HOME>/caches) for caching to be enabled. This setting ensures that the cache plugin does not fill up the disk. Value is in BYTES.
plugin.stash-scm-cache.refs.enabled	false	Controls whether ref advertisement operations are cached.
plugin.stash-scm-cache.refs.ttl	60 (1 min)	Controls how long the caches for ref advertisements are kept around when there no changes to the repository. Caches are automatically invalidated when someone pushes to a repository or when a pull request is merged. Time is in SECONDS.
plugin.stash-scm-cache.upload-pack.enabled	true	Controls whether clone operations are cached.
plugin.stash-scm-cache.upload-pack.ttl	3600 (1 hour)	Controls how long the caches for clone operations are kept around when there no changes to the repository. Caches are automatically invalidated when someone pushes to a repository or when a pull request is merged. Time is in SECONDS.

REST API

Method	Url	Description
GET	/rest/scm-cache/latest/config/refs/enabled	Retrieve whether ref advertisement caching is enabled (true) or disabled (false).
PUT	/rest/scm-cache/latest/config/refs/enabled/{status}	Enable (status = true) or disable (status = false) ref advertisement caching.
GET	/rest/scm-cache/latest/config/refs/ttl	Retrieve the expiry in seconds for the ref advertisement caches.
PUT	/rest/scm-cache/latest/config/refs/ttl/{expiryInSec}	Set the expiry in seconds for the ref advertisement caches.
GET	/rest/scm-cache/latest/config/upload-pack/enabled	Retrieve the whether clone caching is enabled (true) or disabled (false).
PUT	/rest/scm-cache/latest/config/upload-pack/enabled/{status}	Enable (status = true) or disable (status = false) clone caching.
GET	/rest/scm-cache/latest/config/upload-pack/ttl	Retrieve the expiry in seconds for the clone caches.
PUT	/rest/scm-cache/latest/config/upload-pack/ttl/{expiryInSec}	Set the expiry in seconds for the clone caches.
GET	/rest/scm-cache/latest/caches	Retrieve information about the current caches: size, number of cache hits and misses, etc.
DELETE	/rest/scm-cache/latest/caches	Clear all caches.
GET	/rest/scm-cache/latest/caches/{projectKey}/{repoSlug}	Retrieve information about the caches for the repository identified by projectKey and repoSlug: size, number of cache hits and misses, etc.
DELETE	/rest/scm-cache/latest/caches/{projectKey}/{repoSlug}	Clear the caches for the repository identified by projectKey and repoSlug.

Child pages