Summary

This knowledge base article is intended to help troubleshoot Agent problems by increasing the logged information on the Server and Agent logs.

Environment

• Bamboo Server
• Bamboo Agents (Remote/Elastic)
• Log4j 1.x and 2.x
• Git Checkout tasks

Solution

• Bamboo 8 and earlier with Log4j 1.x
• Bamboo 9 and later with Log4j 2.x
• Debugging Source Code Git Checkout tasks
• More information

Bamboo 8 and earlier with Log4j 1.x

Option 1: Debug the Bamboo server and Remote Agent (requires an application restart)

1. Edit the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties

2. The debug classes can be added on demand in your log4j.properties file depending on what problem is being troubleshot. Atlassian Support may instruct you on which classes to enable

   Subject	Classes to debug
   Agent Registration	log4j.category.com.atlassian.bamboo.agent=DEBUG log4j.category.com.atlassian.bamboo.v2.build.agent=DEBUG log4j.category.com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl=DEBUG
   Agent load	log4j.category.org.apache.activemq=DEBUG
   External process	log4j.category.com.atlassian.utils.process=DEBUG log4j.logger.com.atlassian.bamboo.executor=DEBUG
   Git	log4j.logger.com.atlassian.bamboo.plugins.git.GitRepository=DEBUG
   Artifact transfer	log4j.logger.com.atlassian.bamboo.v2.build.agent.messages.ArtifactStreams=DEBUG log4j.logger.com.atlassian.bamboo.build.artifact=DEBUG log4j.logger.com.atlassian.bamboo.build.artifact.BambooArtifactHttpTransport=DEBUG

3. Restart both Bamboo Server and Remote Agent (the above-debugging classes will also take effect on Agent logs after the restart) 

Option 2: Debug only the Remote Agent

In case you only need Agent-specific debug logs, or if you can't restart your Bamboo server application, you can enable logging only on the Remote Agent side by following the below steps:

1. Stop the Remote Agent
2. Copy the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties file from the Bamboo Server to the Remote Agent's <bamboo-agent-home>/conf

3. Edit the log4j.properties on the Remote Agent and modify the following property from OFF to ALL or one of the provided log levels: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL. For more information about log levels, check the Log4j API documentation.

   log4j.appender.console.threshold = [FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL]

4. Edit the log4j.properties based on debugging classes suggested in the table above

5. Configure your agent to load the log4j.properties file:

   • Uncomment the line following on <bamboo-agent-home>/conf/wrapper.conf and add the absolute path to the log4j.properties file you just created:

     #wrapper.java.additional.3=-Dlog4j.configuration=

   • It should look like this after the change:

     wrapper.java.additional.3=-Dlog4j.configuration=<bamboo-agent-home>/conf/log4j.properties

6. Restart the Remote Agent

Option 3: Debug the Elastic Agent with Log4j 1.x

To add extra debug flags to the Elastic Agent, create a custom image that would hardcode the necessary Log4j properties and startup flags or use an Instance Startup Script. The instructions below have been tested on Bamboo's stock Linux image running Java 11; you may have to adapt it to a Windows equivalent version.

1. Edit the Image Configuration

2. Add the following Script to the Instance Startup Script:

   cat <<EOF>/home/bamboo/log4j.properties
   <put the log4j 1.x content you want> -- use the Remote Agent's log4j content as an example
   EOF
   
   chown bamboo:bamboo /home/bamboo/log4j.properties
   
   sudo su -c "echo \"export JDK_JAVA_OPTIONS='\"-Dlog4j.configuration=/home/bamboo/log4j.properties\"'\" >> /etc/profile.d/bamboo.sh"

3. The next time you start the image, it will load the log4j.properties file and provide you with the extra debug logs

Bamboo 9 and later with Log4j 2.x

Option 1: Debug the Bamboo server and Remote Agent (requires an application restart)

1. Edit the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j2.properties

2. The debug classes can be added on demand in your log4j2.properties file depending on what problem is being troubleshot. Atlassian Support may let you know which types to enable.

   Subject	Classes to debug
   Agent Registration	logger.agent-added.level=DEBUG logger.agent-added.name=com.atlassian.bamboo.agent logger.buildqueue-added.level=DEBUG logger.buildqueue-added.name=com.atlassian.bamboo.buildqueue logger.buildagent-added.level=DEBUG logger.buildagent-added.name=com.atlassian.bamboo.v2.build.agent
   Agent load	logger.activemq-added.level=DEBUG logger.activemq-added.name=org.apache.activemq
   External process	logger.utils-added.level=DEBUG logger.utils-added.name=com.atlassian.utils.process logger.executor-added.level=DEBUG logger.executor-added.name=com.atlassian.bamboo.executor
   Git	logger.git-added.level=DEBUG logger.git-added.name=com.atlassian.bamboo.plugins.git.GitRepository
   Artifact transfer	logger.artifact-streams.level=DEBUG logger.artifact-streams.name=com.atlassian.bamboo.v2.build.agent.messages.ArtifactStreams logger.artifact.level=DEBUG logger.artifact.name=com.atlassian.bamboo.build.artifact logger.artifact-http-transport.level=DEBUG logger.artifact-http-transport.name=com.atlassian.bamboo.build.artifact.BambooArtifactHttpTransport

3. Restart both Bamboo Server and Remote Agent (the above-debugging classes will also take effect on Agent logs after the restart) 

Option 2: Debug only the Remote Agent

In case you only need Agent-specific debug logs, or if you can't restart your Bamboo server application, you can enable logging only on the Remote Agent side by following the below steps:

1. Stop the Remote Agent
2. Copy the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j2.properties file from the Bamboo Server to the Remote Agent's <bamboo-agent-home>/conf

3. Edit the log4j2.properties on the Agent, and modify the following property from OFF to ALL or one of the provided log levels: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL. For more information about log levels, check the Log4j2 API documentation API documentation.

   appender.console.filter.threshold.level = [FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL]

4. Edit the log4j2.properties based on debugging classes suggested in the table above

5. Configure your agent to load the log4j2.properties file:
   Uncomment the line following on <bamboo-agent-home>/conf/wrapper.conf and add the absolute path to the log4j2.properties file you just copied:

   #wrapper.java.additional.3=-Dlog4j2.configurationFile

   It should look like this after the change:

   wrapper.java.additional.3=-Dlog4j2.configurationFile=<bamboo-agent-home>/conf/log4j2.properties

6. Due to BAM-22466 - Getting issue details... STATUS , add the following line to <bamboo-agent-home>/conf/wrapper.conf to allow the Tanuki Service Wrapper to load extra Bamboo classes so the custom logging works:

   wrapper.java.classpath.3=../classpath/*.jar

7. Restart the Remote Agent

Option 3: Debug the Elastic Agent with Log4j 2.x

To add extra debug flags to the Elastic Agent, create a custom image that would hardcode the necessary Log4j properties and startup flags or use an Instance Startup Script. The instructions below have been tested on Bamboo's stock Linux image running Java 11; you may have to adapt it to a Windows equivalent version.

1. Edit the Image Configuration

2. Add the following Script to the Instance Startup Script:

   cat <<EOF>/home/bamboo/log4j2.properties
   <put the log4j 2.x content you want> -- use the Remote Agent's as an example
   EOF
   
   chown bamboo:bamboo /home/bamboo/log4j2.properties
   
   sudo su -c "echo \"export JDK_JAVA_OPTIONS='\"-Dlog4j2.configurationFile=/home/bamboo/log4j2.properties\"'\" >> /etc/profile.d/bamboo.sh"

3. The next time you start the image, it will load the log4j2.properties file and provide you with the extra debug logs

Debugging Source Code Git Checkout tasks

The following guidelines provide valuable information for troubleshooting Bamboo's Git checkout tasks without using Script Tasks. This is essential for examining problems concerning the Bamboo Agent and the VCS repository while maintaining Bamboo's internal SSH-PROXY authentication active.

It is important to note that debugging a Bamboo checkout task can be complex and may require some modification to succeed. The instructions provided below are tailored for Linux Agents; however, adjustments may be necessary for Windows, using a PowerShell approach.

1. On the Bamboo Agent (or on the Server if you are using Local Agents), create the following wrapper script:

   git.wrapper

   #!/bin/bash
   
   # Bamboo depends on fetch and the version's standard format, don't debug it
   if [[ $1 = "fetch" || $1 = "version" ]]; then
     exec /usr/bin/git $*
   else
     exec /usr/bin/env GIT_TRACE=1 GIT_CURL_VERBOSE=1 GIT_TRANSFER_TRACE=1 GIT_TRACE_SHALLOW=1 /usr/bin/git $*
   fi

2. Set the correct ownership and executable permissions to the Git wrapper script using chmod +x; chown;
3. In Bamboo, modify that specific Agent's capabilities and change Git to point to the wrapper script
4. On the Agent, delete the following folders and their content to force Git to fetch the repository completely:
   • <bamboo-agent-home>/xml-data/build-dir/PROJ-PLAN-JOB
   • <bamboo-agent-home>/xml-data/build-dir/_git-repositories-cache/<cache_folder_for_the_repository>
5. Modify the Plan's requirements to point to that specific Agent. You can also dedicate the Agent to the Plan in case you don't want to modify the Plan structure
6. Run the Plan

The Git debug logs will be very verbose. They will be sent to stdout, meaning they will be available on the Job build logs and will not appear on the Agent logs.

More information

For more info on how to increase log verbosity on remote agents:

• Configuring the level of logging on remote agents

Extra debugging: Investigating packet content

For a deeper investigation of packet content, network traffic dumps between the agent and the server are helpful. Please enable the network packet capturing on the server side and reproduce the problem. Please refer to Capturing HTTP traffic using Wireshark or Fiddler for more details on capturing the network packet.