This is the documentation for Bamboo 5.5. View this page for the

Unknown macro: {spacejump}

of Bamboo, or visit the latest Bamboo documentation.

The Amazon Elastic Block Store (EBS) provides 'EBS volumes' which can attach to EC2 instances. EBS volumes (and the 'EBS snapshots' created from these volumes) provide persistent storage for your elastic instances.

If you have relatively static resources required for building your Bamboo jobs (such as, source code checkouts and Maven repository artifacts), you can add these to an EBS volume. From this volume, you can create an EBS snapshot, which effectively records the 'state' of an EBS volume at a given point in time.

After setting up an EBS snapshot, you can then associate it with an elastic image configuration. When this elastic image is started:

  • its elastic instance will be started, along with the EBS volume (derived from the EBS snapshot associated with the elastic image) and
  • this EBS volume will be attached to this elastic instance
    (info) any build resources (added to the EBS volume prior to creating its snapshot) will be available to this elastic instance.

Why should I use the EBS with Elastic Bamboo?
Because an elastic instance is stateless, so also is the elastic agent that runs on it. Hence, every time an elastic instance is restarted from the same image:

  • Any resources that its elastic agent must retrieve externally (for example, Maven repository artifacts), must be downloaded in their entirety.
  • Full checkouts must be performed by elastic agents when new Jobs are built.

Therefore, you can use the EBS to store these external resources in an EBS volume and snapshot so that they do not have to be downloaded or source code checked out each time you start up an elastic instance from an image. If your jobs rely heavily on downloading such resources and/or you are not performing clean builds each time, the EBS may significantly improve your build times.

Additionally, the EBS provides an easy mechanism for customising elastic agents, rather than you having to create a custom elastic image from scratch (with your own elastic agent capabilities). For example, you could upload files and scripts to your EBS volume that would install resources such as PostgreSQL databases for your elastic agents, which will be available when the agent's elastic instance is started.

On this page:

Related page:

Creating your first EBS snapshot

To create your first EBS snapshot:

  1. Download Amazon Web Services (AWS) account identifiers to your Bamboo server — You will need to store the AWS private key file and certificate file on your Bamboo server to use Elastic Bamboo with EBS. If you haven't downloaded an AWS private key file or certificate file to your Bamboo server yet, please see Generating your AWS Private Key File and Certificate File for instructions.
  2. Update your Bamboo configuration settings with the location of the AWS account identifier files you have downloaded. This will ensure that these files are uploaded to any new elastic instances started. See the Elastic Instance Settings section on the Configuring Elastic Bamboo for instructions (you will need to update the Upload AWS account identifiers to new elastic instances (mandatory if EBS Snapshot ID specified) checkbox and Account Private Key File and Account Certificate File fields described on this page).
  3. Start a single elastic instance via Bamboo. See Starting an elastic instance for instructions.
  4. Access your elastic instance via SSH (see Accessing an elastic instance for instructions).
  5. As root user, follow the steps below to create an EBS volume and attach it to the elastic instance (steps a & b), upload content to the EBS volume (step c & d), and generate the snapshot (step e & f):
    (info) All the scripts mentioned below are available in /opt/bamboo-elastic-agent/bin on Bamboo stock images. You can also download them from here (choose the latest version). 
    1. Run createInitialVolume.sh <volume size> — This script creates a EBS volume (where <volume size> is the size of the volume), attaches the volume and mounts it on the elastic instance. For example, createInitialVolume.sh 100 will create a 100GB EBS volume and attach and mount it on the elastic instance.
    2. Run rewarmEbsSnapshot.sh — This script sets up the standard structure for Elastic Bamboo on the EBS volume. The directories and files for this standard volume structure are detailed in the Important EBS Directories and Files section below.
    3. (optional) Populate your EBS volume — Your EBS volume can now be populated with any files and scripts that you wish to make available to the elastic instances that use the EBS volume. For example, you may want to upload maven repository data, source code, scripts and files to install databases on your elastic agents, etc. You must upload your files to the /mnt/bamboo-ebs folder or its subfolders, if you want them to be included in the snapshot. We recommend that you read Populating your EBS volume for guidelines on how to populate your EBS volume effectively.
      (info) The EBS volume is attached to the elastic instance, so accessing your elastic instance via SSH will give you full access to the EBS volume (see Important EBS Directories and Files below).
    4. Ensure all uploaded content has the owner bamboo:bamboo — You can set the owner of a file by executing the following command: chown -R bamboo:bamboo <filename>
    5. Execute the killall java command — This command kills all processes on the instance, such as agent processes, so that the volume can be unmounted to be snapshotted.
    6. Run generateSnapshot.sh — This script unmounts and detaches the volume, before creating a snapshot based on the volume. The time taken to create the snapshot will vary depending on the amount of content that you have uploaded to the EBS volume. The Snapshot ID for the snapshot will be available in the logs for the elastic instance. See Accessing an Elastic Instance for instructions on how to access the logs for your elastic instance.
      (info) The device can not unmount if any terminals are currently in the mounted volume.
  6. Shut down your elastic instance. See Shutting down an elastic instance for instructions.

Configuring an elastic image to use an EBS snapshot

Once you have set up an EBS snapshot, the final step is to add the snapshot details to an elastic image configuration, so that any instances started from that image will have EBS volumes attached to them. You can associate different snapshots with different elastic image configurations.

To configure Elastic Bamboo to use an EBS snapshot:

  1. Determine the Snapshot ID of the EBS snapshot you have just created. The Snapshot ID should be recorded in the logs of the elastic instance you created it on. You can also view your EBS snapshots in the AWS Console by clicking the Snapshots menu item.



  2. Click the  icon and select Overview.
  3. Click Image Configurations in the left navigation column (under 'Elastic Bamboo'). 
  4. Click Edit in the 'Operations' column for the elastic image configuration that you would like to add your EBS snapshot to. The 'Edit Elastic Image Configuration - <imagename>' screen will display (see screenshot below).


  5. Check Automatically attach an Amazon Elastic Block Store (EBS) volume to new elastic instances. 
  6. Enter the Snapshot ID of your EBS snapshot in the EBS Snapshot ID field.
  7. Click Save. A new EBS volume will be created from the specified snapshot and attached to any new elastic instances started from that image.

Updating your EBS snapshot

If you are currently using EBS with Elastic Bamboo and want to update your snapshot, follow the instructions below. These are similar to the instructions for creating a new EBS snapshot.

To update your EBS snapshot:

  1. Start a single elastic instance via Bamboo. See Starting an elastic instance for instructions.
  2. (optional) Run a build on the elastic agent of the instance to populate the attached EBS volume. We recommend that you read Populating your EBS volume for guidelines on how to populate your EBS volume effectively.
  3. Access your elastic instance via SSH (see Accessing an elastic instance for instructions) and do the following:
    (info) All the scripts described below are bundled with Bamboo.
    1. (optional) Upload any additional content to the attached EBS volume via Secure Copy (SCP). You must upload your files to the /mnt/bamboo-ebs folder or its subfolders, if you want them to be included in the snapshot.
    2. Execute killall java — This command kills all agent processes, so that nothing is using the mounted volume.
    3. Execute jps -vl — This command displays a list of all java processes running on your instance. There should be no java processes running.
    4. Run generateSnapshot.sh — This script unmounts and detaches the volume, before creating a snapshot based on the volume.
      (info) The device can not unmount if any terminals are currently in the mounted volume.
    5. Check the elastic instance logs for the Snapshot ID of the snapshot you just created. See Accessing an Elastic Instance for instructions on how to access the logs for your elastic instance.
    6. Update the new Snapshot ID in your Elastic Bamboo configuration, as described in Configuring an Elastic Image to use an EBS snapshot above.

Important EBS directories and files

By convention, Bamboo will attach an EBS device at /dev/sdh. This will be mounted at /mnt/bamboo-ebs. The contents of the standard structure are:

  • bin/customiseInstance.sh - This script is run on startup of an elastic instance. We recommend that you do not customise this script, as it is overwritten when rewarmEbsSnapshot.sh is run.
  • bin/customise-extras.sh - This script is run on startup of an elastic instance as the root (as opposed to being run as the Bamboo user). This script is safe to customise, as it will never be overwritten. You can customise this script to automate processes such as setting up your database, move files to custom locations on the instance, etc.
  • profile-extras.sh - This script gets appended to the profile that is run under the Bamboo user (as opposed to being run as the root). It is useful for setting up environment variables.
  • bamboo-agent/bamboo-agent.cfg.xml - This configuration file modifies the build working directory to point to build working directory on the EBS volume.
  • bamboo-agent/build-dir - This is the build working directory.
  • maven/build.properties - This properties file is copied to /home/bamboo on startup of an elastic instance. It points the Maven 1 default repository to /mnt/bamboo-ebs/maven/.maven
  • maven/.m2/settings.xml - This configuration files is copied to /home/bamboo/.m2 on startup of an elastic instance. It points the Maven 2 default repository to /mnt/bamboo-ebs/maven/.m2/repository.
  • tmp-extras - The contents of this directory is copied to /tmp on startup of an elastic instance.

 

 

Except where otherwise noted, content in this space is licensed under a Creative Commons Attribution 2.5 Australia License.