Parallel step options

Parallel steps allow you to group pipeline steps that can run at the same time (concurrently) to reduce build time.

Overview

Set up or run parallel steps

Parallel steps enable you to build and test faster, by running a set of self-contained steps at the same time. The total number of build minutes used by a pipeline will not change if you make the steps parallel. You'll be able to see results sooner but the total build minutes used will calculate based on the total time taken on each step.

There are several ways in which you can use this feature, however, we recommend following these guidelines:

  1. Set an initial step to build the software.

  2. Add a set of parallel steps to test the software.
    If you are doing a single build step leading to multiple parallel testing steps, cache external build dependencies as much as possible in the first step, so you don't waste time doing it in each of your testing steps. Then, run all your tests concurrently, they can use the cache.

  3. Deploying to multiple similar environments at the same time.

For example:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 pipelines: default: - step: # non-parallel step script: - ./build.sh - parallel: # these 2 steps will run in parallel steps: - step: script: - ./integration-tests.sh --batch 1 - step: script: - ./integration-tests.sh --batch 2 - step: # non-parallel step script: - ./deploy.sh

Default variables for parallel steps

In addition to the standard pipelines variables, parallel step groups also have the following default variables:

  • BITBUCKET_PARALLEL_STEP - zero-based index of the current step in the group (such as 0, 1, 2, …).

  • BITBUCKET_PARALLEL_STEP_COUNT - total number of steps in the group.

Limitations for parallel steps

  • The 100-step limit on pipelines includes each step in a parallel group.

  • All environments in a parallel step set must be of the same type, so don’t try to mix production and test environment types in the same set.

  • Parallel steps can produce and consume artifacts, however, keep in mind the following:

    • Parallel steps can only use artifacts produced by previous steps, not by steps in the same parallel set.

    • Steps after the parallel set will get a combination of all files produced.

    • If parallel steps produce artifacts containing a file at the same location, conflicts are resolved on a per-file basis, and the files generated by the latest step in the YAML file will win.

Parallel step options

The following options can be used to configure parallel step groups:

Parallel

The parallel option allows you to to build and test your code faster by running a list of steps at the same time. The total number of build minutes used by a pipeline will not change if you make the steps parallel, but you'll be able to see the results sooner. The total number of steps you can have in a Pipeline definition is limited to 100, regardless of whether they are running in parallel or serial.

For details on using parallel steps, see Set up or run parallel steps.

Propertyparallel

Required — No

Data type — Block of new-line separated key-value pairs (YAML spec - Block Mapping)

Allowed parent propertiesdefault, branches, pull-requests, tags, and custom

Allowed child propertiessteps (required) and fail-fast

Example — using the parallel option to run two steps at the same time

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 pipelines: default: - step: # sequential step name: Build script: - ./build.sh - step: # sequential step name: Build script: - ./build.sh - parallel: # these 2 steps will run in parallel steps: - step: name: Integration 1 script: - ./integration-tests.sh --batch 1 - step: name: Integration 2 script: - ./integration-tests.sh --batch 2 - step: # non-parallel step script: - ./deploy.sh

Steps

The steps property contains the lists of steps in a stage or a parallel. At least 1 step is required within the steps list.

Propertysteps

Required — Yes

Data type — Block of new-line separated key-value pairs (YAML spec - Block Mapping)

Allowed parent propertiesparallel and stage

Allowed child propertiesstep (required)

Example — using the steps property to list the steps in a parallel group

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 pipelines: default: - step: # sequential step name: Build script: - ./build.sh - step: # sequential step name: Build script: - ./build.sh - parallel: # these 2 steps will run in parallel steps: - step: name: Integration 1 script: - ./integration-tests.sh --batch 1 - step: name: Integration 2 script: - ./integration-tests.sh --batch 2 - step: # non-parallel step script: - ./deploy.sh

Example — using the steps property to list the steps in a stage

1 2 3 4 5 6 7 8 9 10 11 12 13 pipelines: default: - stage: name: Build and test steps: - step: name: Build app script: - sh ./build-app.sh - step: name: Run unit tests script: - sh ./run-tests.sh

Fail-fast

fail-fast can be applied to all parallel steps or to a specific step in a parallel group:

  • If a step has fail-fast: false, then the step can fail without the whole parallel group stopping.

  • If a step has fail-fast: true, then the whole parallel group will stop if the step fails.

Propertyfail-fast

Required — No

Data type — Boolean

Allowed valuestrue or false

Default valuefalse

Allowed parent propertiesstep and parallel

Example — using fail-fast to stop parallel steps when one fails

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 pipelines: default: - step: name: Build script: - ./build.sh - parallel: # these option alows to force stop all running steps if any step fails fail-fast: true steps: - step: name: Integration 1 script: - ./integration-tests.sh --batch 1 - step: name: Integration 2 script: - ./integration-tests.sh --batch 2

Example — exclude a parallel from fail-fast

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 pipelines: default: - step: name: Build script: - ./build.sh - parallel: # these option alows to force stop all running steps if any step fails fail-fast: true steps: - step: name: Integration 1 script: - ./integration-tests.sh --batch 1 - step: name: Integration 2 script: - ./integration-tests.sh --batch 2 - step: # option can be disabled for a step # and its failure won't stop other steps in a group fail-fast: false name: Upload metadata script: - ./upload-metadata.sh

Additional Help