Checking the status of a runner

The runner status can be checked on the Runners page associated with the workspace or repository.

• To find the status of a workspace runner, select Settings on the left sidebar of the workspace, and select Workspace runners under Pipelines on the left sidebar navigation.

• To find the status of a repository runner, select Repository settings on the left sidebar of the repository, and select Runners under Pipelines on the left sidebar navigation.

The status can be one of the following:

• Unregistered: The runner was created but was never run. Note: If a runner isn’t registered within one week, it will be removed.

• Online: The runner is live and available for step scheduling.

• Offline: The runner is not available. It may have been stopped, or there may be network connectivity issues.

• Disabled: The runner was temporarily disabled, and steps will not be scheduled on it until it is enabled again.

Reviewing runner logs

Build and services logs (not including the runners' logs) for a step are removed from a runner host once the step is complete. These logs can be found in the step logs on the Pipeline's user interface.

The location of the runner log files varies between operating systems and the type of runner.

• Linux Docker runners

• Windows PowerShell runners

• macOS shell runners

Identifying the runner UUID to determine log location

The runner UUID is logged as part of a build's step logs in the Pipeline's user interface. This UUID can be used to identify the correct location of runner.log files.

Runner matching labels:
    - linux.shell
Runner name: linux-shell-runner
Runner UUID: {b609961f-891e-c872-c36b-f3f2c315d186}
Runner labels: self.hosted, linux.shell
Runner version:
    current: 1.466
    latest: 1.465

Reviewing log files for Linux Docker runners

1. Access the Linux device hosting the affected runner.

2. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>/<runnerUuid>/runner.log

   For example:

   /tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /tmp directory.

3. Review the end of the log file for errors that need to be addressed.

Reviewing log files for Windows PowerShell runners

1. Access the Windows device hosting the affected runner.

2. Using PowerShell or File Explorer, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where PowerShell was working when the runner was started.
   For example: if PowerShell was working in the users' home directory, such as:

   PS C:\Users\Username>

   The runner installation directory would be C:\Users\Username\atlassian-bitbucket-pipelines-runner\

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>\<runnerUuid>\runner.log

   For example:

   .\atlassian-bitbucket-pipelines-runner\temp\b609961f-891e-c872-c36b-f3f2c315d186\runner.log

   The default working directory for the runner is the \temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. Review the end of the log file for errors that need to be addressed.

Reviewing log files for macOS shell runners

1. Access the macOS device hosting the affected runner.

2. Using Terminal or Finder, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where Terminal was working when the runner was started.
   For example: if Terminal was working in the users' home directory (~/), such as: /Users/Username/; the runner installation directory would be /Users/Username/atlassian-bitbucket-pipelines-runner/

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>/<runnerUuid>/runner.log

   For example:

   ~/atlassian-bitbucket-pipelines-runner/temp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. Review the end of the log file for errors that need to be addressed.

Debugging runner images

Debugging docker image platform 

To debug suspected platform related problems with the images used by your pipeline, there are a few checks you can run against the image to check your platform is supported:

• Check the logs inside of Build Setup and for any service containers used by your pipeline - any warnings about potential platform incompatibilities will be logged.

• Check the documentation provided by the image vendor to see if your platform is supported. In some instances, the image may use a specific tag per platform.

• Run docker manifest inspect <image>:<tag> on the image, looking for a matching entry for your OS and architecture.

• Check the platform when running docker inspect <image>:<tag>, checking that the OS and architecture matches.

Updating a runner version

Linux Docker runner

If you experience an issue with a runner, update the runner to the latest version.

1. Stop the runner.

2. Pull the latest runner version using the command below:

   docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:3.0.0

3. Restart the runner.

Non-docker runners (Linux shell, macOS, Windows)

The latest runner version will need to be downloaded again. When creating a new runner in the UI, there are several instructions for downloading the runner archive, extracting it, and running the new runner.

In most cases, creating a new runner and using that last command to start the new runner will be the best way to get an updated runner. If an existing runner's starting command including the OAuth client and secret had been saved somewhere previously, the old command can be used to start an existing runner without needing to create a new runner.

The commands to download the latest archive links for each platform are:

• Linux shell

curl https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner-3.0.0.tar.gz --output atlassian-bitbucket-pipelines-runner.tar.gz

• macOS

curl https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner-3.0.0.tar.gz --output atlassian-bitbucket-pipelines-runner.tar.gz

• Windows

Invoke-WebRequest -Uri https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner-3.0.0.zip -OutFile .\atlassian-bitbucket-pipelines-runner.zip

Troubleshooting errors when starting a runner

Troubleshooting errors during step execution

A runner is in an unhealthy state and cannot run pipelines

Issue

The status of a runner is Unhealthy and the Runners page is showing the following message:

Cause

At the end of each step in a pipeline, the runner cleans up build directories (folders) before running the next step. If the runner can’t perform this cleanup, it will enter an Unhealthy state. Some possible causes of this issue are:

• The permissions for one or more files (or directories) within a build directory are preventing the runner from deleting files.

• A program or service is accessing one or more files (or directories) in the build directory at the time of cleanup.

Solution

To find out which file or directory is preventing the runner from performing a build cleanup, review the runners' log files. Search for an error, such as:

An error occurred whilst tearing down directories.

A detailed error log that can be used for debugging will be displayed below this error message.

Accessing the log files

The location of the runner log files varies between operating systems and the type of runner.

• Linux Docker runners

• Windows PowerShell runners

• macOS shell runners

Accessing the log files for Linux Docker runners

1. Access the Linux device hosting the affected runner.

2. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    <runner_working_directory>/<runnerUuid>/runner.log 

   For example:

   /tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /tmp directory.

3. Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on Linux

In a terminal; fix the permissions on a single file by adding write permissions for the user (u) and the users' user group (g) with the chmod command. Such as:

chmod ug+w <path/filename> 

To fix the permissions on a directory and its files and subdirectories, add the --recursive option, such as:

chmod ug+w --recursive <path/filename>

The chmod command can also be appended to the script of a pipeline step if the permissions issue occurs with every pipeline executed. For example:

image: atlassian/default-image:3
pipelines:
  default:
    - step:
      name: 'Step that causes permissions issues'
      script:
        - bash ./product-build-script.sh && chmod ug+w --recursive build/

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for Linux.

Accessing the log files for Windows PowerShell runners

1. Access the Windows device hosting the affected runner.

2. Using PowerShell or File Explorer, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where PowerShell was working when the runner was started.
   For example: if PowerShell was working in the users' home directory, such as:

   PS C:\Users\Username>

   The runner installation directory would be C:\Users\Username\atlassian-bitbucket-pipelines-runner\

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>\<runnerUuid>\runner.log 

   For example:

   .\atlassian-bitbucket-pipelines-runner\temp\b609961f-891e-c872-c36b-f3f2c315d186\runner.log 

   The default working directory for the runner is the \temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. 
   

   Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on Windows

To fix file or directory access permissions, either:

• Open the file Properties dialog from File Explorer and update the user permissions for the current user to Full Access for the files or folders preventing cleanup.

• Use the PowerShell Set-Acl command to provide write and delete access to the files or folders preventing cleanup. For information on using the Set-Acl command, visit Microsoft Docs — PowerShell Set-Acl cmdlet.

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for Windows.

Accessing the log files for macOS shell runners

1. Access the macOS device hosting the affected runner.

2. Using Terminal or Finder, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where Terminal was working when the runner was started.
   For example: if Terminal was working in the users' home directory (~/), such as: /Users/Username/; the runner installation directory would be /Users/Username/atlassian-bitbucket-pipelines-runner/

3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

   <runner_working_directory>/<runnerUuid>/runner.log 

   For example:

   ~/atlassian-bitbucket-pipelines-runner/temp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

   The default working directory for the runner is the /temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

4. Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on macOS

In a terminal; fix the permissions on a single file by adding write permissions for the user (u) and the users' user group (g) with the chmod command. Such as:

chmod ug+w <path/filename> 

To fix the permissions on a directory and its files and subdirectories, add the --recursive option, such as:

chmod ug+w --recursive <path/filename> 

The chmod command can also be appended to the script of a pipeline step if the permissions issue occurs with every pipeline executed. For example:

image: atlassian/default-image:3
pipelines:
  default:
    - step:
      name: 'Step that causes permissions issues'
      script:
        - bash ./product-build-script.sh && chmod ug+w --recursive build/

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for macOS.

Windows PowerShell troubleshooting

Unsigned scripts in Windows PowerShell

Issue

A Windows-based runner has failed to run one or more steps due to an unsigned PowerShell (.ps1) script, resulting in an error similar to the following:

.\start.ps1 : File C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 cannot be loaded. The file
C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 is not digitally signed. You cannot run this script on the
current system. For more information about running scripts and setting execution policy, see about_Execution_Policies at
https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ .\start.ps1 -accountUuid '{924bbdf1-ea18-2c70-4655-2bb23075ddbf}' -re ...
+ ~~~~~~~~~~~
    + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess

Cause

The Windows runner generates PowerShell scripts for cloning the repository and running the script for each step in the pipeline. These scripts are generated when the pipeline is run, preventing them from being digitally signed.

To allow the Windows runners to run unsigned PowerShell scripts, set the PowerShell execution policy of the CurrentUser to either:

• RemoteSigned (recommended)

• unrestricted

• bypass

Solution

To check the execution policy for the CurrentUser:

1. Open Windows PowerShell from the Windows Start menu.

2. Run the following command, which will return the execution policy for the CurrentUser:

   Get-ExecutionPolicy -Scope CurrentUser

To change the execution policy for CurrentUser to RemoteSigned:

1. In Windows PowerShell, run the following command:

   Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser 

2. Verify that the change was successful by running Get-ExecutionPolicy and confirm that the CurrentUser has the RemoteSigned execution policy.

   Get-ExecutionPolicy -Scope CurrentUser

For information on Microsoft PowerShell execution policies, visit Microsoft Docs — PowerShell: about Execution Policies.

Build folder not being cleaned up properly

At the end of a step, the runner will attempt to clean up the build folder. This is a best effort cleanup and in some cases runners are not able to clean up the build folder. The following are the two most common cases that prevent a runner from cleaning up the build folder:

• The step script creates files that runners do not have permission to remove, for example, set FullControl as Deny for all users.

• The step script is creating orphaned processes that a runner is not able to stop, and some of the files in the build are locked by those orphaned processes. For example, a hosted machine had Gradle daemon (which is not suggested by Gradle), and is set to use daemon for all Gradle builds by: org.gradle.daemon=true. In this case, the processes created by Gradle builds will attach to the existing daemon process, and the runner would not able to stop them at the end of the step.

Any of the above cases will prevent the runner from cleaning up the build folder, and will cause the step to fail with an error informing you to check the runner logs and manually clean up the build folder in the host machine. Any following steps that are scheduled to run on this runner will also cause an error because the build folder was not cleaned up properly.

Solution: You need to contact the admin who has access to the runner and check the runner logs to find out which files are preventing the runner from cleaning up the build folder.

For files that have permissions to prevent runners from removing them, the admin needs to fix the permissions for those files, and then remove the build folder and restart runner.

For files that are locked by orphaned processes, the admin needs to find out the process that locked the files, and kill the process. Then remove the build folder and restart runner.

For either of the above solutions, you need to revisit the step script and the current settings for the host machine which will ensure the step won’t lock the build folder next time you run the step, or you will have to do the manual clean up in host machine again. In the case of Gradle daemon, you can add the --no-daemon option in the Gradle build, and it will prevent any child processes attached to the Gradle daemon.