Checking the status of a runner

The runner status can be checked on the Runners page associated with the workspace or repository.

• To find the status of a workspace runner, select Settings on the left sidebar of the workspace, and select Workspace runners under Pipelines on the left sidebar navigation.

• To find the status of a repository runner, select Repository settings on the left sidebar of the repository, and select Runners under Pipelines on the left sidebar navigation.

The status can be one of the following:

• Unregistered: The runner was created but was never run. Note: If a runner isn’t registered within one week, it will be removed.

• Online: The runner is live and available for step scheduling.

• Offline: The runner is not available. It may have been stopped, or there may be network connectivity issues.

• Disabled: The runner was temporarily disabled, and steps will not be scheduled on it until it is enabled again.

Reviewing runner logs

Build and services logs (not including the runners' logs) for a step are removed from a runner host once the step is complete. These logs can be found in the step logs on the Pipeline's user interface.

The location of the runner log files varies between operating systems and the type of runner.

• Linux Docker runners

• Windows PowerShell runners

• macOS shell runners

Identifying the runner UUID to determine log location

The runner UUID is logged as part of a build's step logs in the Pipeline's user interface. This UUID can be used to identify the correct location of runner.log files.

Runner matching labels:
    - linux.shell
Runner name: linux-shell-runner
Runner UUID: {b609961f-891e-c872-c36b-f3f2c315d186}
Runner labels: self.hosted, linux.shell
Runner version:
    current: 1.466
    latest: 1.465

Reviewing log files for Linux Docker runners

1. Access the Linux device hosting the affected runner.

2. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>/<runnerUuid>/runner.log

   For example:

   /tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /tmp directory.

3. Review the end of the log file for errors that need to be addressed.

Reviewing log files for Windows PowerShell runners

1. Access the Windows device hosting the affected runner.

2. Using PowerShell or File Explorer, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where PowerShell was working when the runner was started.
   For example: if PowerShell was working in the users' home directory, such as:

   PS C:\Users\Username>

   The runner installation directory would be C:\Users\Username\atlassian-bitbucket-pipelines-runner\

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>\<runnerUuid>\runner.log

   For example:

   .\atlassian-bitbucket-pipelines-runner\temp\b609961f-891e-c872-c36b-f3f2c315d186\runner.log

   The default working directory for the runner is the \temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. Review the end of the log file for errors that need to be addressed.

Reviewing log files for macOS shell runners

1. Access the macOS device hosting the affected runner.

2. Using Terminal or Finder, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where Terminal was working when the runner was started.
   For example: if Terminal was working in the users' home directory (~/), such as: /Users/Username/; the runner installation directory would be /Users/Username/atlassian-bitbucket-pipelines-runner/

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>/<runnerUuid>/runner.log

   For example:

   ~/atlassian-bitbucket-pipelines-runner/temp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. Review the end of the log file for errors that need to be addressed.

Debugging runner images

Debugging docker image platform 

To debug suspected platform related problems with the images used by your pipeline, there are a few checks you can run against the image to check your platform is supported:

• Check the logs inside of Build Setup and for any service containers used by your pipeline - any warnings about potential platform incompatibilities will be logged.

• Check the documentation provided by the image vendor to see if your platform is supported. In some instances, the image may use a specific tag per platform.

• Run docker manifest inspect <image>:<tag> on the image, looking for a matching entry for your OS and architecture.

• Check the platform when running docker inspect <image>:<tag>, checking that the OS and architecture matches.

Updating a runner version

It is always recommended that you keep your runner updated to receive the latest patches, features, and bug fixes. To see what's included in each update, you can refer to the runner changelog.

If you encounter issues with a runner, updating it to the latest version can help resolve the problem. Follow the steps below to update your runner:

1. Stop the Runner
   Before updating, ensure that the runner is stopped. The method to stop the runner depends on the type of runner you are using:
   
   

   • Linux Docker Runner:

     • Identify the existing Runner container. The below command will return the container ID and name of the runner container:

       docker container ls | grep runner

     • Stop the Runner container using the container ID or name retrieved from the above command:

       docker container stop <container_id_or_name>

     • Remove the Runner container:

       docker container rm <container_id_or_name>

   • Linux Shell, macOS, or Windows Runners:

     • Stop the process that is running the runner. This could be done via terminal, task manager, or service manager, depending on how the runner was originally initiated.

     • For Linux and MacOS, find the runner’s process ID (PID) and kill it:

       ps aux | grep -E 'java.*runner' 
       kill <PID>

     • On Windows, use the Task Manager or PowerShell to stop the process.
       
       

2. Pull the Latest Runner Version
   Use the appropriate command for your runner type to download the latest version of the runner.
   
   Linux Docker Runner

   docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner

   Non-Docker Runners

   • Linux Shell

     curl -O https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner.tar.gz

   • macOS

     curl -O https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner.tar.gz

   • Windows

     Invoke-WebRequest -Uri https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner.zip -OutFile .\atlassian-bitbucket-pipelines-runner.zip

   
   

3. Start the Runner
   Restart the runner using the command provided in the Bitbucket UI when the runner was initially created. If you no longer have this command, you can create a new runner in the UI to retrieve a new command.

Troubleshooting errors when starting a runner

Troubleshooting errors during step execution

A runner is in an unhealthy state and cannot run pipelines

Issue

The status of a runner is Unhealthy and the Runners page is showing the following message:

Cause

At the end of each step in a pipeline, the runner cleans up build directories (folders) before running the next step. If the runner can’t perform this cleanup, it will enter an Unhealthy state. Some possible causes of this issue are:

• The permissions for one or more files (or directories) within a build directory are preventing the runner from deleting files.

• A program or service is accessing one or more files (or directories) in the build directory at the time of cleanup.

Solution

To find out which file or directory is preventing the runner from performing a build cleanup, review the runners' log files. Search for an error, such as:

An error occurred whilst tearing down directories.

A detailed error log that can be used for debugging will be displayed below this error message.

Accessing the log files

The location of the runner log files varies between operating systems and the type of runner.

• Linux Docker runners

• Windows PowerShell runners

• macOS shell runners

Accessing the log files for Linux Docker runners

1. Access the Linux device hosting the affected runner.

2. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    <runner_working_directory>/<runnerUuid>/runner.log 

   For example:

   /tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /tmp directory.

3. Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on Linux

In a terminal; fix the permissions on a single file by adding write permissions for the user (u) and the users' user group (g) with the chmod command. Such as:

chmod ug+w <path/filename> 

To fix the permissions on a directory and its files and subdirectories, add the --recursive option, such as:

chmod ug+w --recursive <path/filename>

The chmod command can also be appended to the script of a pipeline step if the permissions issue occurs with every pipeline executed. For example:

image: atlassian/default-image:3
pipelines:
  default:
    - step:
      name: 'Step that causes permissions issues'
      script:
        - bash ./product-build-script.sh && chmod ug+w --recursive build/

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for Linux.

Accessing the log files for Windows PowerShell runners

1. Access the Windows device hosting the affected runner.

2. Using PowerShell or File Explorer, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where PowerShell was working when the runner was started.
   For example: if PowerShell was working in the users' home directory, such as:

   PS C:\Users\Username>

   The runner installation directory would be C:\Users\Username\atlassian-bitbucket-pipelines-runner\

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>\<runnerUuid>\runner.log 

   For example:

   .\atlassian-bitbucket-pipelines-runner\temp\b609961f-891e-c872-c36b-f3f2c315d186\runner.log 

   The default working directory for the runner is the \temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. 
   

   Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on Windows

To fix file or directory access permissions, either:

• Open the file Properties dialog from File Explorer and update the user permissions for the current user to Full Access for the files or folders preventing cleanup.

• Use the PowerShell Set-Acl command to provide write and delete access to the files or folders preventing cleanup. For information on using the Set-Acl command, visit Microsoft Docs — PowerShell Set-Acl cmdlet.

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for Windows.

Accessing the log files for macOS shell runners

1. Access the macOS device hosting the affected runner.

2. Using Terminal or Finder, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where Terminal was working when the runner was started.
   For example: if Terminal was working in the users' home directory (~/), such as: /Users/Username/; the runner installation directory would be /Users/Username/atlassian-bitbucket-pipelines-runner/

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>/<runnerUuid>/runner.log 

   For example:

   ~/atlassian-bitbucket-pipelines-runner/temp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on macOS

In a terminal; fix the permissions on a single file by adding write permissions for the user (u) and the users' user group (g) with the chmod command. Such as:

chmod ug+w <path/filename> 

To fix the permissions on a directory and its files and subdirectories, add the --recursive option, such as:

chmod ug+w --recursive <path/filename> 

The chmod command can also be appended to the script of a pipeline step if the permissions issue occurs with every pipeline executed. For example:

image: atlassian/default-image:3
pipelines:
  default:
    - step:
      name: 'Step that causes permissions issues'
      script:
        - bash ./product-build-script.sh && chmod ug+w --recursive build/

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for macOS.

Windows PowerShell troubleshooting

Unsigned scripts in Windows PowerShell

Issue

A Windows-based runner has failed to run one or more steps due to an unsigned PowerShell (.ps1) script, resulting in an error similar to the following:

.\start.ps1 : File C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 cannot be loaded. The file
C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 is not digitally signed. You cannot run this script on the
current system. For more information about running scripts and setting execution policy, see about_Execution_Policies at
https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ .\start.ps1 -accountUuid '{924bbdf1-ea18-2c70-4655-2bb23075ddbf}' -re ...
+ ~~~~~~~~~~~
    + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess

Cause

The Windows runner generates PowerShell scripts for cloning the repository and running the script for each step in the pipeline. These scripts are generated when the pipeline is run, preventing them from being digitally signed.

To allow the Windows runners to run unsigned PowerShell scripts, set the PowerShell execution policy of the CurrentUser to either:

• RemoteSigned (recommended)

• unrestricted

• bypass

Solution

To check the execution policy for the CurrentUser:

1. Open Windows PowerShell from the Windows Start menu.

2. Run the following command, which will return the execution policy for the CurrentUser:

   Get-ExecutionPolicy -Scope CurrentUser

To change the execution policy for CurrentUser to RemoteSigned:

1. In Windows PowerShell, run the following command:

   Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser 

2. Verify that the change was successful by running Get-ExecutionPolicy and confirm that the CurrentUser has the RemoteSigned execution policy.

   Get-ExecutionPolicy -Scope CurrentUser

For information on Microsoft PowerShell execution policies, visit Microsoft Docs — PowerShell: about Execution Policies.

Build folder not being cleaned up properly

At the end of a step, the runner will attempt to clean up the build folder. This is a best effort cleanup and in some cases runners are not able to clean up the build folder. The following are the two most common cases that prevent a runner from cleaning up the build folder:

• The step script creates files that runners do not have permission to remove, for example, set FullControl as Deny for all users.

• The step script is creating orphaned processes that a runner is not able to stop, and some of the files in the build are locked by those orphaned processes. For example, a hosted machine had Gradle daemon (which is not suggested by Gradle), and is set to use daemon for all Gradle builds by: org.gradle.daemon=true. In this case, the processes created by Gradle builds will attach to the existing daemon process, and the runner would not able to stop them at the end of the step.

Any of the above cases will prevent the runner from cleaning up the build folder, and will cause the step to fail with an error informing you to check the runner logs and manually clean up the build folder in the host machine. Any following steps that are scheduled to run on this runner will also cause an error because the build folder was not cleaned up properly.

Solution: You need to contact the admin who has access to the runner and check the runner logs to find out which files are preventing the runner from cleaning up the build folder.

For files that have permissions to prevent runners from removing them, the admin needs to fix the permissions for those files, and then remove the build folder and restart runner.

For files that are locked by orphaned processes, the admin needs to find out the process that locked the files, and kill the process. Then remove the build folder and restart runner.

For either of the above solutions, you need to revisit the step script and the current settings for the host machine which will ensure the step won’t lock the build folder next time you run the step, or you will have to do the manual clean up in host machine again. In the case of Gradle daemon, you can add the --no-daemon option in the Gradle build, and it will prevent any child processes attached to the Gradle daemon.

Self-hosted Run Failure Scenarios

• Scenario 1: Pipeline build failed with Currently no online runner available that matches the required labels error
• Scenario 2: Pipeline build failed with Service ‘docker’ exited with exit code: 1 error
• Scenario 3: Pipeline build failed with We couldn't clone the repository. Try rerunning the pipeline error
• Scenario 4: Pipeline build failed with The Docker feature is not supported on this self-hosted runner's platform error
• Scenario 5: Pipeline build failed with An error occurred while processing an artifact error
• Scenario 6: Pipeline build failed with no space left on device error

Scenario 1: Pipeline build failed with Currently no online runner available that matches the required labels error

Platform: Bitbucket Pipeline Self-hosted Runners - All

Possible causes: 

During the setup process, the runner can utilize both custom labels and default labels. The default labels provided to the Runner are determined by the platform being used:

• Linux Shell - self.hosted linux.shell 

• Windows Shell - self.hosted windows

• MacOs Shell - self.hosted macos 

• Linux Docker - self.hosted linux

When a Pipeline step is executed, the parameters specified in the runs-on section are matched with the labels assigned to the Runner. It is important to understand the following points regarding the parameters configured for a step:

• For a step to execute on a Self-hosted runner, it must include the self.hosted parameter.

• The step should also include one of the platform-specific parameters listed below. If none of these parameters are specified, the linux parameter will be applied by default:

  • linux.shell 

  • windows 

  • macos 

  • linux 

• Additionally, the step can incorporate custom parameters.

Errors may arise if there is no Runner available that possesses all the labels specified as parameters in the runs-on section of the Pipeline step.

Troubleshooting Steps:

Examine the labels assigned to the Runner and the parameters included in the runs-on section of the Pipeline step. Ensure that there is a minimum of one Runner that possesses all the labels specified as parameters in the Pipeline step.

Scenario 2: Pipeline build failed with Service ‘docker’ exited with exit code: 1 error

Platform: Bitbucket Pipeline Self-hosted Runners - Linux Docker

Possible causes:

• A change in the docker service being used in the Pipeline step

• The docker service change could be due to the infrastructure changes in the Pipeline, or a change in the Docker service changes at your end if you are using custom image for the Docker service

Troubleshooting steps:

• If you are utilizing a custom Docker image, it is essential to ensure that no recent modifications have been made to the image. You can confirm this by examining the Docker image hash in the Build Setup of the build that recently failed, comparing it to the hash of the build that was functioning correctly earlier.

• If you are not using a custom Docker image, it is advisable to investigate whether the Bitbucket Pipeline infrastructure has implemented any recent changes and to review the solutions provided.

Scenario 3: Pipeline build failed with We couldn't clone the repository. Try rerunning the pipeline error

Platform: Bitbucket Pipeline Self-hosted Runners - All

Possible Causes:

• The host machine may lack the necessary prerequisites as outlined in the Runner specification.

• There could be multiple instances of the runner operating on the same machine.

• The runner might not possess the required permissions for the directory necessary to clone the repository on the host machine.

Troubleshooting Steps:

• Verify whether the host machine meets the necessary prerequisites as outlined by the Runner. Please refer to the Prerequisites section of the relevant Runner documentation to ensure that all requirements are fulfilled.

  • Linux Shell

  • Windows Shell

  • MacOs Shell

  • Linux Docker

• Reconfigure the Runner

  • Stop the Runner.

    • For the Linux Docker Runner:

      • First, identify the existing Runner container. The following command will return the container ID and name of the runner container:

        docker container ls | grep runner

      • Next, stop the Runner container using the container ID or name obtained from the previous command:

        docker container stop <container_id_or_name>

      • Finally, remove the Runner container:

        docker container rm <container_id_or_name>

    • For Linux Shell, MacOS Shell, or Windows Shell Runner:

      • Stop the process that is currently running the runner. This can be accomplished through the terminal, task manager, or service manager, depending on the method used to initiate the runner.

        • For Linux and MacOS, locate the runner’s process ID (PID) and terminate it: 

          ps aux | grep -E 'java.*runner' kill <PID>

        • On Windows, utilize the Task Manager or PowerShell to stop the process.

  • Delete the build  folder.

    • Navigate to the path  /atlassian-bitbucket-pipelines-runner/bin/../temp/ where you will find a folder for each runner on that machine, formatted as <runner-uuid>.

    • For each of the runner's folders, open the folder and check for the existence of a build folder. If it is present, please delete the folder along with its contents.

  • Setup the Runner - Follow the instructions for setting up the Runner according to the specific Runner type.

    • Linux Shell

    • Windows Shell

    • MacOs Shell

    • Linux Docker

• For the Windows Shell Runner, ensure that the Runner host machine has all the necessary permissions as described in the “Allow unsigned scripts to run in PowerShell” section of Windows Shell.

 Scenario 4: Pipeline build failed with The Docker feature is not supported on this self-hosted runner's platform error

Platform: Bitbucket Pipeline Self-hosted Runners - All Shell based

Possible cause:

You may be utilizing runners that currently don’t support the use of service containers, such as the Docker service. This limitation can be confirmed by referring to the respective Runner Unsupported Features section. The following runners don't support service containers:

• Linux Shell

• Windows Shell

• MacOs Shell

Troubleshooting steps:

• To utilize service containers, it is recommended to use Runners that support them, such as Linux Docker.

• If you wish to utilize an existing Runner that does not support service containers, you must remove the service container from the Bitbucket Pipeline YAML file (bitbucket-pipelines.yml).

 Scenario 5: Pipeline build failed with An error occurred while processing an artifact error

Platform: Bitbucket Pipeline Self-hosted Runners - All

Possible causes:

• An outdated version of the Runner is currently in use.

• The latest version of the Runner (greater than 3.0.0) is being utilized, yet the essential modifications required for the Runner have not been implemented.

Troubleshooting Steps:

• Runner Version

  • For Runner versions less than 3.0.0: The Pipeline team has recently announced that Pipelines Runner version 3.0.0 is now available. This version introduces improved artifact and cache management capabilities. It is advisable to upgrade the Runner version to 3.0.0 by referring to the Updating a Runner version section on this page.

  • For Runner versions greater than or equal to 3.0.0: If your self-hosted runners are operating behind a firewall that filters connections based on IP addresses or URLs, it is essential to ensure that you unblock the following for both incoming and outgoing traffic when upgrading to version 3.0.0 or above:

    • If Using IP

      • A comprehensive list of IP addresses that traffic can route to AWS can be found by utilizing the following endpoint. It is important to filter for records where the service equals S3, and specify the us-east-1 and us-west-2 regions.

      • To facilitate the filtering process, the following command can be employed to retrieve the latest list of AWS S3 IPs used by the self-hosted runners:

        • curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select((.region=="us-east-1" or .region=="us-west-2") and .service=="S3") | .ip_prefix'

        • Please note that it is necessary to allowlist all these IPs irrespective of the step size.

    • If Using URL

      micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com

      Additionally, the timeout with the latest version of the self-hosted runners is set to 30 seconds. If you are utilizing the latest version of a self-hosted runner, you can configure this timeout by adjusting the preconfigured command that you use to start the runner.

      If you are using Docker-based runners, please add this to the command that initiates the runner:

      -e S3_READ_TIMEOUT_SECONDS=<secondsvalue>

      For Linux Shell and MacOS Shell runners, please add this to the command that initiates the runner:

      --s3ReadTimeoutSeconds <secondsvalue>

      For Windows Shell runners, please add this to the command that initiates the runner:

      -s3ReadTimeoutSeconds "<secondsvalue>"

      Replace <secondsvalue> with a value in seconds based on your estimation of how long the artifact upload will take, keeping in mind the expected size of the artifact. For example, you may start with 600 (equivalent to 10 minutes) and adjust it to a lower or higher value as necessary.

 Scenario 6: Pipeline build failed with no space left on device error

Platform: Bitbucket Pipeline Self-hosted Runners - All

Possible causes:

• The /tmp directory lacks sufficient disk space.

Troubleshooting Steps:

• Verify the /tmp Directory

  • It is essential to confirm that the /tmp directory, where the Runner is configured, has adequate disk space available.

  • The temporary directory is established during the Runner setup. The directory is specified using the following variables based on the type of Runner:

    • For Linux Docker Runners - WORKING_DIRECTORY 

    • For Other Runners - workingDirectory 

  • To assess the available disk space in the directory:

    • For Windows Shell Runner, refer to Microsoft's article Find out how much storage your PC has.

    • For Other Runners: Execute the following command in the terminal of the host machine to check the available space for each mounted drive and verify if the Runner's working directory (/tmp by default, unless altered during the Runner start) is nearing its limit and how much disk space is allocated to it:

      $ df -h <working directory>

• Free Up Space

  • To free up disk space allocated as the Runner's working directory, consider deleting unnecessary and large files, relocating some files to another disk or storage solution, or increasing the disk size.

  • Alternatively, the Runner's working directory can be changed to a different folder where the host has more disk space available:

    1. Linux Docker Runner:  Changing the working directory of your runner | Linux Docker runner.

    2. Other Runners: Update the --working-directory argument in the Runner start command:

       $ ./start.sh <rest of arguments> --workingDirectory <full path to desired working directory folder>