Caching dependencies

Bitbucket Pipelines is able to cache external build dependencies and directories, such as 3rd-party libraries, between builds.

As well as providing faster builds and developer feedback, caching can also reduce the number of build minutes consumed.


What is dependency caching?

Most builds start by running commands that download dependencies from the internet, which can take a lot of time for each build. As the majority of dependencies stay the same, rather than download them every time, we recommend downloading them once into a cache which you can reuse for later builds.

Set up caching

To enable caching, add a caches section to your step .

Here's an example of how to cache your node_modules directory for a Node.js project using a pre-defined cache.

bitbucket-pipelines.yml
pipelines:
  default:
    - step:
        caches:
          - node
        script:
          - npm install
          - npm test

The first time this pipeline runs it won't find the node cache and so the npm commands will download dependencies from the internet. For future builds, our service will have them cached, so loading them into your build will be quicker.

Pre-defined caches

Pipelines provides pre-defined cache locations for commonly used language tools:

Cache name In your step Directories
docker
caches:
- docker
n/a -  we create the cache from the layers generated by your build
composer
caches:
- composer
~/.composer/cache
dotnetcore
caches:
- dotnetcore
~/.nuget/packages
gradle
caches:
- gradle
~/.gradle/caches
ivy2
caches:
- ivy2
~/.ivy2/cache
maven
caches:
- maven
~/.m2/repository
node
caches:
- node
node_modules
pip
caches:
- pip
~/.cache/pip
sbt
caches:
- sbt
- ivy2

~/.sbt

~/.ivy2/cache

Note that when you use the sbt tool you need to enable both the sbt and ivy2 caches.


Custom caches for other build tools and directories

If your build tool isn't listed above, you can still define a custom cache for your repository in your bitbucket-pipelines.yml file. First, in the definitions section of the yml, define the cache name and the directory to be cached, then you can refer to that cache in any step:

bitbucket-pipelines.yml
image: ruby:2.4.0
pipelines:
  default:
    - step:
        caches:
          - bundler #cache is defined below in the definitions section
        script:
          - bundle install --path vendor/bundle

definitions:
  caches:
    bundler: vendor/bundle

The defined cache will be saved after the first successful build.

Cache directory paths can be absolute, or relative to the clone directory, for example:

  • $HOME/.npm
  • ~/.gradle/wrapper
  • /usr/local/lib/node_modules
  • vendor/bundle

Note that, for Ruby, you should install gems in a separate location from the system's gem repo, like this:


$ bundle install --path vendor/bundle

Caching multiple directories

Some builds might benefit from caching multiple directories. Simply reference multiple caches in your step like this:

bitbucket-pipelines.yml
image: openjdk:8
pipelines:
  default:
    - step:
        caches:
          - gradle               # pre-defined cache
          - gradlewrapper        # custom cache that must be defined below
        script:
          - ./gradlew build
definitions:
  caches:
    gradlewrapper: ~/.gradle/wrapper 

How does caching work?

When does a cache get saved?

Caches are saved on successful builds when the cache is empty. Only caches under 1GB once compressed are saved.

For the cache to compress to under 1GB, the size of the original images in the docker daemon must be < 2GB.
You can check the size by adding this command to the script in your bitbucket-pipelines.yml:

docker image inspect $(docker image ls -aq) --format {{.Size}} | awk '{totalSizeInBytes += $0} END {print totalSizeInBytes}'

When does a cache get restored?

At the beginning of each build, saved caches are downloaded if available and placed in their directories.

When does a cache get cleared?

Any cache which is older than 1 week will be cleared automatically and repopulated during the next build. Additionally, caches can be cleared manually in the Bitbucket UI. Just choose Caches at the top right of the Pipelines page, then trash a cache from the list:


What should I cache and not cache?

Dependency caching is all about reducing build times by avoiding downloading and building the same files repeatedly.

That means you should cache things like:

  • Language-specific dependencies.
  • Binaries of dependencies that take a while to compile.

You shouldn't cache:

  • Sensitive data such as passwords and credentials

Remember that caches are temporary, so your builds should be configured to work whether or not the cache is present.

Last modified on Jun 7, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.