Building multi-architecture docker images with Bitbucket Pipelines
Platform Notice: Cloud Only - This article only applies to Atlassian products on the cloud platform.
Summary
When using Bitbucket Pipelines to build docker images, some use cases might require the image to be built to run in multiple CPU architectures, such as ARM64, ARMv7, x86_64, etc. With the introduction of docker buildx command, it's now possible to build images for different platforms using the QEMU processor emulator, which is essentially a software-based emulation tool that eliminates the requirement of having hosts that run on different CPU architectures. QEMU will translate the instructions of a given architecture to the instruction set of the host machine.
This article provides instructions on how to configure your bitbucket-pipelines.yml file using QEMU to build multi-architecture images in a Linux self-hosted runner.
Environment
Building multi-architecture docker images is only currently available when using Linux docker self-hosted runners.
The usage of docker buildx command to build multi-architecture docker images requires the execution of the docker command with the --privileged flag. For security reasons, the --privileged flag and the buildx command are restricted to being used with cloud-hosted runners, meaning that for steps that run on Atlassian infrastructure, those flags/commands are not available. More details on the restricted command and flags are listed in Run Docker commands in Pipelines.
There's an open feature request to implement the ability to build multi-architecture images also in cloud runners: BCLOUD-15317.
Solution
Following are described the required steps to construct your bitbucket-pipelines.yml file to build multi-architecture images in a Linux docker self-hosted runners:
- Create and start a Linux docker self-hosted runner.
If you haven't already, create a Linux docker self-hosted runner under your repository or workspace, and start the runner with the provided pre-configured command. Define a custom docker daemon
The default pipelines docker daemon does not allow the usage of --privileged flag. When using self-hosted runners, it's possible to define a custom docker daemon that does not have this restriction, such as the docker:dind :definitions: services: docker: # Define a custom docker daemon - can only be used with a self-hosted runner image: docker:dind
Enable docker CLI experimental Feature
As buildx is considered an experimental feature in Docker v20(version used by Pipelines), you'll need to enable the experimental feature by exporting the environment variable DOCKER_CLI_EXPERIMENTAL :export DOCKER_CLI_EXPERIMENTAL=enabled
Download buildx plugin from the source and move it to the docker-cli plugins folder
Buildx is a docker plugin that does not come by default in most of the images. If your docker images don't have buildx already installed, you can download it directly from the official releases page :https://github.com/docker/buildx/releases
and move the file to the $HOME/.docker/cli-plugins/ folder as per the instruction of Buildx - Manual Download.
mv docker-buildx $HOME/.docker/cli-plugins/ && chmod +x ~/.docker/cli-plugins/docker-buildx
Run the multiarch/qemu-user-static to setup the QEMU emulation environment
The multiarch/qemu-user-static image already comes with the QEMU simulator for several architectures and with the scripts to setup those simulators into the host's kernel. This requires this image to be executed with the --privileged flag :docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Create a new buildx builder :
With the buildx plugin installed and QEMU emulation setup, it's now necessary to create a builder that buildx can use :docker buildx create --use
The --use flag will make buildx already switch to the newly created builder.
Build the multi-architecture image with buildx build
Finally, we are ready to execute the buildx build command specifying the --platform flag with the desired target architectures to build against :docker buildx build -t $DOCKER_HUB_USER/my_image_multi_arch:my_tag --push --platform linux/amd64,linux/arm64,linux/arm/v7 .
By default, buildx will only save the resulting image on docker’s internal build cache, so it will not show up when executing the docker images command. To store the image, you will have to push it to the Docker registry (e.g DockerHub). The command above already includes the --push flag and assumes you have previously logged in to Dockerhub with the user $DOCKER_HUB_USER.
Below is an example of the resulting bitbucket-pipelines.yml file following the instructions above:
pipelines:
default:
- step:
name: My Runner
image: atlassian/default-image:4
runs-on:
- linux
- self.hosted
services:
- docker
script:
- echo "Executing on self-hosted runner"
- export DOCKER_CLI_EXPERIMENTAL=enabled # Enable usage of buildx in Docker version < 23
- export BUILDX_VERSION=0.11.0 # define what BUILDX_VERSION to download and install
- curl -fsSLO https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64
- mkdir -p $HOME/.docker/cli-plugins/ && mv buildx-v${BUILDX_VERSION}.linux-amd64 $HOME/.docker/cli-plugins/docker-buildx && chmod +x ~/.docker/cli-plugins/docker-buildx # download buildx and move it to the docker plugin folder
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes; docker buildx create --use # setup the QEMU emulation environment
- echo $DOCKER_HUB_PASS | docker login --username $DOCKER_HUB_USER --password-stdin # login to Dockerhub
- docker buildx build -t $DOCKER_HUB_USER/my_image_multi_arch:my_tag --push --platform linux/amd64,linux/arm64,linux/arm/v7 . # Build multi-architecture image for the platforms provided and push to Dockerhub
definitions:
services:
docker: # Define a custom docker daemon - can only be used with a self-hosted runner
image: docker:dind
If you are still encountering issues after following this troubleshooting article, please raise a support ticket or raise a community support ticket for further assistance.