Elastic Bamboo allows you to use computing resources from the Amazon Elastic Compute Cloud (EC2) to run builds. Elastic Bamboo uses a remote agent AMI (Amazon Machine Image) to create instances of remote agents in the Amazon EC2. 


To configure your Amazon Web Services (AWS) account details or settings for Elastic Bamboo:

  1. Click the  icon and select Bamboo admin.
  2. Click Configuration in the left navigation panel (under 'Elastic Bamboo').

  3. Click Edit.

  4. Configure settings as described in the sections below.
  5. Click Save when finished.

On this page:

Related pages:

AWS account settings

Before you use Elastic Bamboo for the first time in your Bamboo instance, you must enter your Amazon Web Services (AWS) account details into the Bamboo application. If you do not have an AWS account, you must register for one on the AWS registration page before you can enable Elastic Bamboo.

Before you begin:

  • Please note, Elastic Bamboo dynamically creates and runs remote agents in the Amazon Elastic Compute Cloud (EC2). Hence, if you choose to use Elastic Bamboo, you will be charged by Amazon for your EC2 compute usage (separately to your Bamboo license fee). These charges will be billed to the AWS account that you provide. Please read Elastic Bamboo Costs for more details.
  • Please note, if you change your AWS account details, Bamboo will stop all elastic agents that are currently running.

To set your AWS account details:

  1. Enter or update your AWS Access Key ID (you can get the "AWS Access Key ID" and "AWS Secret Access Key" if you go to your account: "My Account/Console" > Security Credentials).
  2. To enter or update your AWS Secret Access Key, select the Change AWS Secret Access Key? checkbox, and enter or update AWS Secret Access Key.
  3. Click Save.

Note that your AWS Access Key ID and AWS Secret Access Key are used together to identify you when accessing Amazon EC2 services. If you are unsure what your AWS Account ID and AWS Secret Access Key are, please refer to the Amazon documentation on AWS access identifiers.

Global settings

Elastic Bamboo provides you with a number of global configuration options to help you optimise EC2 usage for your Bamboo job builds. These settings control how the Bamboo server operates and how it manages its elastic instances and agents.

Maximum Number of Elastic InstancesThe number of elastic instances that can be running at any one time. You may wish to decrease this value if you are concerned about EC2 compute costs, and you have a large number of concurrent job builds that cannot be supported by your non-elastic agents.
Automatically terminate elastic instance when elastic agent process ends

Controls whether your elastic instances will automatically shut down after the elastic agent processes running on them terminate.

  • Shutdown Delay —controls how long an elastic instance will wait before shutting down, after its elastic agent process terminates.

 

EC2 spot instances

Elastic Bamboo provides support for Amazon EC2 Spot Instances. Amazon spot instances allow you to bid on unused EC2 capacity and use it, as long as your bid exceeds the current "Spot price". You can configure Elastic Bamboo to bid for a spot instance of a particular type, and fall back to a regular instance after a set amount of time if no instances are available.

Enable support for spot instancesSelect this checkbox to enable support for spot instances.
Fallback to a regular instance afterThe time (in minutes) after which Elastic Bamboo will fall back to using a regular instance, if a spot instance has not become available.
Your current bid levels (per hour)Fill out this table with your bids. The bids are categorised by EC2 instance type and operating system.

AWS settings

These settings allow you to specify your AWS configuration settings in Bamboo so that Bamboo can operate elastic instances through your AWS account. This section includes settings that are used to configure elastic instances to work with the Amazon Elastic Block Store (EBS).

Using EBS with your elastic instances can significantly reduce the amount of data transfer required to run a job build, compared with starting a clean elastic instance. To find out more about this feature and how to set it up in Elastic Bamboo, read Configuring elastic instances to use the EBS.

Upload AWS account identifiers to new elastic instancesSelect to upload the AWS Account Private Key File and Account Certificate File to all new elastic instances started. This is mandatory if you wish to use EBS to store job build information in a snapshot. However, you can also check this option if you are not using EBS (e.g. if you wish upload the AWS account identifiers in order to use Amazon's AWS command line tools).
Key files locationChoose how private key and certificate will be provided.
Account Private Key File

You must specify the location of this file to use the Amazon EBS with Elastic Bamboo. This file is generated by Amazon.

Account Certificate FileYou must specify the location of this file to use the Amazon EBS with Elastic Bamboo. This file is generated by Amazon.

(info) If you haven't downloaded an AWS private key file or certificate file to your Bamboo server yet, please see Generating your AWS Private Key File and Certificate File for instructions.

Automatic elastic instance management

The Automatic Elastic Instance Management feature allows Bamboo to start and shut down elastic instances automatically (based on build queue demands), so that you do not have to perform these action manually. This feature reduces Bamboo administration overhead and can help minimise your overall elastic instance usage costs.

If a job's requirements cannot be met by any available online agents, this feature will start any elastic instance whose elastic agent has the capabilities to execute the job, so that the job's build can be generated. Regardless of how an elastic instance was started, all elastic instances will be shut down based on the settings specified below.

Choose from the following elastic instance management presets. Each of these presets define values for the five criteria described in the 'Custom' user-defined options (below). (Bear in mind that both the 'Aggressive' and 'Passive' presets have trade-offs.)

  • Default — Balances build queue clearance rates with elastic instance usage costs.
  • Aggressive — Favours higher build queue clearance rates but with higher elastic instance usage costs.
  • Passive — Favours lower instance usage costs but with lower build queue clearance rates.
  • Custom — Choose your own settings, as described below.
  • Disabled — Disables Bamboo's automatic elastic instance management feature.
Idle Agent Shutdown DelaySpecify the number of minutes that an elastic agent must be idle before Bamboo shuts down the elastic instance running that agent.
(info) Elastic instances running in the Amazon EC2 compute cloud are charged in hourly blocks from the time they are started. To maximise usage of elastic instances in a cost-effective manner, Bamboo only performs these checks just prior to the expiry of each hourly block.
Allowed non-Bamboo instancesThe maximum number of elastic instances allowed on your AWS account that are not controlled by this Bamboo instance. When this limit is exceeded, Bamboo will not start any new instances.
Maximum Number of Instances to Start at OnceThe maximum number of elastic instances that Bamboo can start in one go. Bamboo only starts this maximum number of elastic instances on a 'per minute' basis.
Number of Builds in Queue ThresholdThe total number of builds in a queue. When this and all other thresholds have been reached, new elastic instances will be started.
Number of Elastic Builds in Queue ThresholdThe number of builds in the queue that can be executed on elastic instances. When this and all other thresholds have been reached, new elastic instances will be started.
Average Queue Time ThresholdThe average number of minutes that job builds have been waiting in a queue. When this and all other thresholds have been reached, new elastic instances will be started.
  • No labels

16 Comments

  1. EddieW

    Feb 22, 2013

    "Allowed non-Bamboo instances"
    We are using bamboo in a VPC environment, and there could be any number of instances running that bamboo should not concern itself with.  SHould I just set an arbirtraily high limit, or will 0 ignore.

     

  2. David Witherspoon

    Feb 26, 2013

    I'm curious about "Allowed non-Bamboo instances".  The AWS account is our main vpc and there should be an unlimited number of instance running outside of bamboo control.  Will Bamboo actively attempt to shutdown instances that are already running?  And can we disable it with a value a special value?

  3. Anonymous

    Mar 28, 2013

    I'm also curious about the "Allowed non-Bamboo instances" ... we don't want Bamboo killing our other instances. This is a blocker issue for us.

  4. Jim McCollom

    Apr 04, 2013

    The "Allowed non-Bamboo instances" is a blocker for me as well.  I don't want Bamboo shutting down my production instances; why should it worry about instances outside its control?  The presence of the configuration option makes me uncomfortable enough that I will not proceed with setting up Bamboo until I understand what it will do.

    1. EddieW

      Apr 04, 2013

      It should not terminate your existing instances ( I think there is another option for that) but this setting will prevent Bamboo from starting new agent instances if the limit is exceeded.

  5. David Witherspoon

    Apr 04, 2013

    EddieW, If that is correct,  I still don't understand why this is a "feature" and what it is for.  There is already a way to control the number of instances bamboo can launch.  Do I really need to give information about the number of instance running in our production environment to bamboo?  There are other ways in AWS to limit the total number of instances that can be running concurrently.  Also, Bamboo does have full access to the AWS account so it could hypothetically shutdown instances (it is very difficult to limit access to running ec2 instances via IAM for specific instances).  I need to understand the purpose of this feature before we run it in our production environment.  Note, bamboo works fine in a demo AWS account and is not shutting down other instances, but I'm still not clear if this is because this feature is not fully implemented or that its doing what its suppose to.

  6. EddieW

    Apr 04, 2013

    As the first one to raise the "can we disable this" question I agree, but I do think there is a completely valid use case.

    If our company has a limited budget and is running other tools that use AWS (jenkins, another bamboo, ETL jobs, whatever) I may very well not want to surpass a certain threshold.  THis value will tell bamboo to effectively wait until those other instances are terminated before spinning up agents, thus keeping my entire AWS budget on track.

  7. Oliver Cox

    Jun 06, 2013

    I don't get the "Allowed non-Bamboo instances" feature or at least the documentation is clear as to what the result will be and what use cases would be covered by what values. Is there nobody from Atlassian that can clarify this by answering one of the comments here?

  8. NathanA

    Oct 02, 2013

    Hi guys,

    I'm afraid that I can't answer this directly, but let me pass it up the chain to try and get some clarification for you.

    Cheers,

    Nathan

  9. ArmenA

    Oct 04, 2013

    Regarding the "Allowed non-Bamboo instances" feature, it's about liability. If a user increases that number, and Bamboo runs many instances, it's the user's fault for not monitoring his account and disabling safeguards. If Atlassian disables that feature, the liability falls on Atlassian.

  10. Dax Fohl

    Jan 08, 2014

    ArmenA I don't understand your explanation.  The field is called "Allowed NON-Bamboo Instances", but you're saying it limits the number Bamboo instances.  Is the field named wrong?  Can you provide an exact explanation of what this field does?  Per the description, my understanding is that if this setting is "10", then it'll prevent me from launching an 11th NON-Bamboo AWS instance from the AWS console, outside of Bamboo.  So if I've got a server farm on this same AWS account, completely unrelated to Bamboo, then it'll interfere with me running that farm.  Is that correct?

    1. Przemek Bruski

      Jan 09, 2014

      No, that's not how it works. We can't prevent you from running new instances on your account.

      In your example, once there are more than 10 instances not controlled by Bamboo, Bamboo will cease starting new instances automatically. It will not interfere with your farm.

  11. Alan Cabrera

    Jan 20, 2014

    Armen, your explanation confuses me.  If Atlassian added a field "max bamboo instances" would that not be better?

    Reading between the lines I'm guessing that bamboo instances can wander off the reservation and not be recognized as bamboo instances.  Hence the max instances field being labeled as NON-Bamboo Instances.

    1. Przemek Bruski

      Jan 20, 2014

      That's right. Due to some EC2 API deficiencies (lack of single-step instance tagging and lack of indempotency in spot requests), we were not able to guarantee that Bamboo won't lose control over instances e.g. due to restart or packet retries.

      A couple of versions ago I've implemented a workaround for these deficiences. This also let me add additional safety measures around restart.

      The definition of this field stayed untouched though. Eventually, it will probably change its meaning from "Max running instances that this server does not control" to "Max running instances that this server started but does not control and wasn't able to terminate". The former included completely unrelated instances, the latter will not. With the new definition, the recommended setting will be 0.

      1. Dax Fohl

        Jan 20, 2014

        The definition also needs to contain an explanation of what occurs as a result.  i.e. if I've got it set to 1 and the measured value is 2, then what happens? Does it forcibly shut extra instances down (and how does it choose which one)? Does it email me once per hour? Does it refuse to start new instances? If the latter, does it tell me why it's refusing, or does it just fail silently?

        1. Przemek Bruski

          Jan 20, 2014

          I've already explained in my comment from 9th.

          It will not shut anything down, it will not email you, it will not start new instances. It will list the reason in Elastic Bamboo (visible on the Elastic Instances page).