Creating a Custom Elastic Image

Before you begin, you need to ensure that you have set up the following:

• Amazon Web Services (AWS) account with EC2 — if you are already using Elastic Bamboo, you should already have an AWS account with EC2 set up. If not, please read Getting Started with Elastic Bamboo.
• Amazon EC2 API Tools — you must install the EC2 API tools on your local machine, otherwise you will not be able to start and access your AMI instance. You can install the EC2 API tools by executing the following commands:

  wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools-1.3-30349.zip
  unzip ec2-api-tools-1.3-30349.zip

• Environment Variables — you must set up the following environment variables on your local machine before creating a custom elastic image:
  • EC2_HOME — set this to the path to the installed EC2 API Tools
  • EC2_CERT — set this to the path to the certificate assigned to EC2 account
  • EC2_PRIVATE_KEY — set this to the path to the private key assigned to your AWS account
• Registered Key Pair — you need a registered EC2 key pair, which consists of a private key file and certificate file, to use the EC2 API tools with your AMI instance. If you have previously generated and registered an EC2 key pair (e.g. to use the EC2 API tools), you can re-use it. If you need to generate a new key pair, you can use the following command to do so:

  $EC2_HOME/bin/ec2-add-keypair <key_pair_name>

  The content of the private key will be displayed in the command-line output on your console. Save this content in a file , starting with the line "--BEGIN RSA PRIVATE KEY--" and ending with the line "--END RSA PRIVATE KEY--". This private key file will be used to access your AMI instance. Set up the appropriate permissions on the private key file by executing the following command.

  chmod 600 <private_key_file>

We strongly recommend that you select an existing Linux/UNIX AMI to customise, rather than starting with a blank AMI. You can get the list of available AMIs by executing the following command:

$EC2_HOME/bin/ec2-describe-images -a

Please check whether you want to launch 32-bit or 64-bit instances from your custom elastic image before selecting an existing AMI. Elastic Bamboo creates"High-CPU Medium" instances (32-bit) by default.

We recommend the following existing Linux/UNIX AMIs for customisation:

• Atlassian's AMI — this AMI is the default image provided by Atlassian. It is a Fedora 8 image updated and prepared for Bamboo, i.e. you will not have to install any Bamboo prerequisites.
• Amazon's AMI — this is the base Fedora 8 image provided by Amazon. It does not have any Bamboo prerequisites installed.
• Alestic's AMI — this is an Ubuntu 9.04 image provided by Alestic. It does not have any Bamboo prerequisites installed. You can find more Ubuntu images on the Alestic site. Please note, these images are not maintained by Atlassian and you should consult the Alestic site for the name of this AMI.

The names of Atlassian's and Amazon's AMIs (and hence, their IDs) change with each release of Bamboo, including both major and minor (or point) releases.

Therefore, to find out the AMI ID for your version of Bamboo:

1. Open the following URL: https://maven.atlassian.com/content/repositories/atlassian-public/com/atlassian/bamboo/atlassian-bamboo-agent-elastic-image-default/ in a web browser.
2. On the resulting directory page, click on the link that represents the version of Bamboo you are currently running. For example, if you are running Bamboo 2.6.1, click on the '2.6.1' link. Another directory page opens, listing a .ami, a .pom and some additional checksum files.
   Do not click on a version number link that contains 'mX', 'rcX' or 'betaX' (where 'X' is a number), since these relate to publicly available developmental releases of Bamboo.
3. Do one of the following to obtain the name of the AMI you require:
   • Atlassian's AMI — Open the atlassian-bamboo-agent-elastic-image-default-A.B.C.ami file (where A.B.C relate to numbers of your version of Bamboo).
     • The content of this file is the name of this AMI.
   • Amazon's AMI — Open the atlassian-bamboo-agent-elastic-image-default-A.B.C.pom XML file (where A.B.C relate to numbers of your version of Bamboo).
     • Locate the ec2ImageID element in this file. The text content of this element is the name of this AMI.

After you have selected an existing AMI to customise, the next step is to start an instance of the AMI.

3.1 Starting an instance of Atlassian's default AMI

If you chose to customise Atlassian's default AMI, you will have to start the instance from the admin section of Bamboo. See Starting an Elastic Instance.

On start up, the Bamboo agent on Atlassian's AMI checks to see if it was started from a Bamboo server, and immediately shuts itself down if it was not. Because of this, Atlassian's default AMI cannot be started using the command line ec2 tools.

Once started, see Accessing an Elastic Instance for details on how to access the running instance.

3.2 Starting an instance from the command line

Use the ec2-run-instances command to start your instance, as follows:

$EC2_HOME/bin/ec2-run-instances <image_name> -k <key_pair_name>

where <image_name> is the name of the AMI selected in the previous step and <key_pair_name> is the name of the registered key pair generated in '1. Requirements' (the public certificate of this key will be injected into your instance).

For example, if you wanted to start an instance of image ami-e55bbd8c using key pair my-keypair, you would run the following command:

$EC2_HOME/bin/ec2-run-instances ami-e55bbd8c -k my-keypair

This command would produce the following command-line output:

INSTANCE        i-25b86743   ami-e55bbd8c     running   my-keypair

i-25b86743 is the name of your new instance, in the above example. Note this down, as you will need the instance name to access your instance in the next step.

Once your instance is running, you will need to obtain the address of the instance so you can access it. To do this, use the following command:

ec2-describe-instances <instance_name>

For example, if you wanted to find the address of instance i-25b86743, you would enter:

ec2-describe-instances i-25b86743

This command would produce the following command-line output similar to this:

RESERVATION	r-790f7210	121852097033	default
INSTANCE	i-25b86743	ami-e55bbd8c 	ec2-174-129-94-241.compute-1.amazonaws.com
domU-12-31-39-04-38-87.compute-1.internal	running	elasticbamboo	0		m1.small
2009-06-24T12:36:20+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc
monitoring-disabled

The address of your instance in the above example is ec2-174-129-94-241.compute-1.amazonaws.com

You can then use this address to access the instance via SSH. See Accessing an Elastic Instance for instructions. If you are using the example command text from that document, you will need to adjust it as follows:

• replace /opt/bamboo/home/xml-data/configuration/elasticbamboo.pk in the example command text with the private key file you generated in '1. Requirements'.
• replace ec2-68-111-185-197.compute-1.amazonaws.com in the example command text with the address of your instance.

5.1 Installing Bamboo Prerequisite Packages

If you selected ami-e55bbd8c as your base AMI in '2. Selecting an Existing AMI', you can skip this step and go to '5.2 Adding Customisations' as image ami-e55bbd8c has been pre-configured for Bamboo. If you have selected a different AMI, you will need to install the following packages onto your instance using the commands shown below:

1. Amazon EC2 API tools

   wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools-1.3-30349.zip
   unzip ec2-api-tools-1.3-30349.zip
   mv ec2-api-tools-1.3-30349 /usr/local/ec2/ec2-api-tools
   

2. Java 6 and Java 5

   # get Java 6
   wget "http://download.java.net/dlj/binaries/jdk-6u11-dlj-linux-i586.bin"
   mkdir unbundle-jdk-6
   cd unbundle-jdk-6
   sh ../jdk-6u11-dlj-linux-i586.bin --accept-license
   cd ..
   
   # get Java 5
   wget "http://download.java.net/dlj/binaries/jdk-5.0u17-dlj-linux-i586.bin"
   mkdir unbundle-jdk-5
   cd unbundle-jdk-5
   sh ../jdk-5.0u17-dlj-linux-i586.bin --accept-license
   cd ..
   
   # Build the JDKs
   curl "https://jdk-distros.dev.java.net/source/browse/jdk-distros/trunk/utils/construct.sh?content-type=text/plain&rev=148" > construct.sh
   chmod +x construct.sh
   ./construct.sh unbundle-jdk-6 /opt/jdk-6 /opt/jre-6
   ./construct.sh unbundle-jdk-5 /opt/jdk-5 /opt/jre-5
   

3. Ant

   wget "http://www.apache.org/dist/ant/binaries/apache-ant-1.7.1-bin.tar.bz2"
   tar xjC /opt -f apache-ant-1.7.1-bin.tar.bz2
   

4. Maven 2

   wget "http://www.apache.org/dist/maven/binaries/apache-maven-2.0.10-bin.tar.bz2"
   tar xjC /opt -f apache-maven-2.0.10-bin.tar.bz2
   mv /opt/apache-maven-2.0.10 /opt/maven-2.0
   ln -fs /opt/maven-2.0 /opt/apache-maven-2.0.10
   

5. (optional) Maven 1.1

   wget "http://archive.apache.org/dist/maven/binaries/maven-1.1.tar.bz2"
   tar xjC /opt -f maven-1.1.tar.bz2
   

5.2 Adding User Customisations to your Instance

Adding your own customisations is quite a simple process, once you have made it this far.

To add user customisations to your instance,

1. Log into your elastic instance (as previously described in '4. Accessing your Instance').
2. Once you have logged into your elastic instance, you can treat it as a standalone machine and install anything you want. For example, if you want to install Tomcat on an Ubuntu instance you would run 'sudo apt-get install tomcat6', configure it, ensure that your startup scripts are in place, etc, just as you would when installing Tomcat on a standalone machine.
   Please note however, you cannot customise the operating system of a running instance. If you want to create an instance with a customised operating system (e.g. Ubuntu), you will need to select an AMI with that operating system installed (as previously described in '2. Selecting an Existing AMI').
3. Everything that you install will be saved in snapshot image created at the end of these instructions (see '6. Creating an Image of your Customised Instance'). Any instances started from this image will have all of your user customisations automatically installed.

5.3 Deploying Bamboo onto your Instance

Once you have installed the Bamboo pre-requisites on you instance and added your customisations, you can deploy Bamboo onto your instance.
Before you deploy anything however, you need to create a 'bamboo' user on your instance by running the following command:

useradd -m bamboo

After you have created the 'bamboo' user, you need to download the Bamboo agent elastic assembly. The latest version of the Bamboo agent elastic assembly can be downloaded from the Bamboo download center (choose the desired tab and click 'Show all'). If you need an earlier version of the Bamboo agent elastic assembly, you can download it from the Bamboo Archive Downloads page or for a more complete list, our maven repostiory.

Once you have downloaded the Bamboo agent elastic assembly, you need to copy it onto your instance by using the following commands:

cd <BAMBOO_AGENT_ELASTIC_ASSEMBLY_DOWNLOAD_DIRECTORY>/webapp
scp -i <private_key_file> atlassian-bamboo-agent-elastic-assembly-<x.x.x>.tar.gz root@<instance_address>:/mnt

where <private_key_file> is the private key file generated in '1. Requirements', <instance_address> is the address of your instance from '4. Accessing your Instance' and <BAMBOO_STANDALONE_DIRECTORY> is your Bamboo Installation directory

Navigate to the /mnt directory on your instance and install the artifacts, as described below:

1. Bamboo Agent binaries

   tar xzC /opt -f atlassian-bamboo-agent-elastic-assembly-<x.x.x>.tar.gz
   ln -s /opt/bamboo-elastic-agent-<x.x.x> /opt/bamboo-elastic-agent
   

   where <x.x.x> is the Bamboo version you are running (e.g. 2.4.1)
   atlassian-bamboo-agent-elastic-assembly-2.3.tar.gz is available in the webapp sub-directory of the Bamboo installation directory.
2. Bamboo Agent default capabilities definition file

   mv bamboo-elastic-agent/bamboo-capabilities.properties /home/bamboo/

5.4 Instance Configuration

At this stage, you should have a customised instance with Bamboo deployed onto it. The last step in creating a customised instance is to set up an EC2 environment on your instance. Carry out the following steps to set this up:

1. Transfer Amazon private key file and certificate to your instance
   Transfer the key files to your instance by running these commands on your local machine:

   scp -i <private_key_file> $EC2_PRIVATE_KEY root@<instance_address>:/mnt
   scp -i <private_key_file> $EC2_CERT root@<instance_address>:/mnt
   

   where <private_key_file> is the private key file from your local machine created in step 'Registered Key Pair' of 1. Requirements and the <instance_address> is the address of your instance from '4. Accessing your Instance'
2. Set up EC2_HOME and JAVA_HOME environment variables
   Set up these environment variables by running the following commands on your instance:

   export EC2_HOME=/usr/local/ec2/ec2-api-tools
   export EC2_PRIVATE_KEY=/mnt/<ec2_private_key_file>
   export EC2_CERT=/mnt/<ec2_certificate_file>
   export JAVA_HOME=/opt/jdk-5
   

3. Set appropriate permissions for the bamboo user directory
   Run the following command on your instance to set permissions on the bamboo user directory:

   chown -R bamboo:bamboo /home/bamboo/
   

4. Configure path variables
   Create a file profile.sh in your instance's /mnt directory. This file contains the default Elastic Bamboo path configuration settings, as seen below:

   export JAVA_HOME=/opt/jdk-5
   export M2_HOME=/opt/maven-2.0
   export MAVEN_HOME=/opt/maven-1.0.2
   export ANT_HOME=/opt/apache-ant-1.7.1
   export EC2_HOME=/usr/local/ec2/ec2-api-tools
   export EC2_PRIVATE_KEY=/root/pk.pem
   export EC2_CERT=/root/cert.pem
   export PATH=/opt/bamboo-elastic-agent/bin:$EC2_HOME/bin:$JAVA_HOME/bin:$M2_HOME/bin:$MAVEN_HOME/bin:$ANT_HOME/bin:$PATH
   

   If all of the tools on this page were installed in recommended locations, no changes are required. Otherwise, you can update the file as required.
   Once profile.sh is correctly customised for your instance, you need to copy it to the /etc/profile.d directory by running the following command on your instance in the /mnt directory:

   mv profile.sh /etc/profile.d/bamboo.sh
   

5. Configure automatic startup of the Bamboo agent
   You will need to configure your instance to start up the Bamboo agent automatically when the instance is started. You can do this by appending the rc.local file to the one that already exists on your instance, by running the following command on your instance in the mnt directory:

   cat /opt/bamboo-elastic-agent/rc.local >> /etc/rc.d/rc.local

6. Final settings and cleanup
   Finally, create a Bamboo welcome screen and clean up keys on your instance by running the following command:

   echo bamboo-<x.x.x>  >> /opt/bamboo-elastic-agent/motd
   mv /opt/bamboo-elastic-agent/motd /etc/motd
   rm -f /root/firstlogin /etc/ssh/ssh_host_da_key /etc/ssh/ssh_host_dsa_key.pub
   /etc/ssh/ssh_host_key /etc/ssh/ssh_host_key.pub /etc/ssh/ssh_host_rsa_key
   /etc/ssh/ssh_host_rsa_key.pub /root/.ssh/authorized_keys
   touch /root/firstrun
   

   where <x.x.x> is the Bamboo version you are running (e.g. 2.4.1)

The final step is to create an image from your customised instance. To do this, you will require the following information:

• Amazon Account Number
• Access Key ID
• Secret Access Key
• Amazon S3 bucket name that will be used to store image (if you don't have access to Amazon S3, you can sign up on this page.)

You can create an image of your customised instance by using the ec2-bundle-vol command, as follows:

/usr/local/bin/ec2-bundle-vol -c $EC2_CERT -k $EC2_PRIVATE_KEY -u <amazon_account_number> -p <elastic_image_name> --batch --debug

where <elastic_image_name> is the name that you want to assign to your custom image (e.g. 'CustomImage1')

Once the image is created, you need to upload it to Amazon S3 by running the command below:

/usr/local/bin/ec2-upload-bundle -b <s3_bucket_name> -m /tmp/<elastic_image_name>.manifest.xml -a <access_key_id> -s <secret_access_key> --retry --debug

where <s3_bucket_name>, <access_key_id> and <secret_access_key> are the Amazon S3 bucket name, Access Key ID and Secret Access Key described previously, and <elastic_image_name> is the name that you want to assign to your custom image (e.g. 'CustomImage1')

You will then need to register your image with Amazon EC2 by using the ec2-register command:

$EC2_HOME/bin/ec2-register <s3_bucket_name>/<elastic_image_name>.manifest.xml

where <s3_bucket_name> is the Amazon S3 bucket name described previously and <elastic_image_name> is the name that you want to assign to your custom image (e.g. 'CustomImage1')

The output of this command will show the AMI ID of your custom image.

Congratulations, you have successfully set up a custom elastic image!

Now that you have created a custom elastic image, there are two more steps that you will need to complete before you can use it.

First, you will need to associate your custom elastic image with your Bamboo installation by creating an Elastic Image Configuration. Please note the AMI ID of your new custom image and read Managing your Elastic Image Configurations for further instructions.

Secondly, you will need to configure the capabilities of the elastic agents that will run on instances started from your image. This is done by adding the appropriate builder, JDK, Perforce and custom capabilities to your elastic image configuration, so that it reflects what your custom elastic image actually can do. For example, if you have created a custom elastic image with JDK 1.6 and Maven 2 installed, you will need to add capabilities for JDK 1.6 and Maven 2 to the elastic image configuration. Read Configuring Elastic Agent Capabilities for further instructions.

If you need more help, there are a number of resources that you can take advantage of:

• AWS Support Center — if you are having problems with any of your Amazon services, not specifically related to Bamboo, you can obtain basic support from the AWS Support Center. Note, you will need to sign up for Premium Support to get access to web/phone support.
• AWS Resource Center — the AWS Resource Center has links to online documentation, code samples and tools for AWS services.
• Bamboo Developer Forums — please feel free to discuss any useful tips or issues regarding this process in the Bamboo Developer Forums.