How to increase debug logging to investigate Agent problems

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

This knowledge base article is intended to help troubleshoot Agent problems by increasing the logged information on the Server and Agent logs.

Environment

  • Bamboo Server
  • Bamboo Agents (Remote/Elastic)
  • Log4j 1.x and 2.x
  • Git Checkout tasks

Solution

Bamboo 8 and earlier with Log4j 1.x

Option 1: Debug the Bamboo server and Remote Agent (requires an application restart)

  1. Edit the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties
  2. The debug classes can be added on demand in your log4j.properties file depending on what problem is being troubleshot. Atlassian Support may instruct you on which classes to enable

    SubjectClasses to debug
    Agent Registration

    log4j.category.com.atlassian.bamboo.agent=DEBUG
    log4j.category.com.atlassian.bamboo.v2.build.agent=DEBUG
    log4j.category.com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl=DEBUG

    Agent load

    log4j.category.org.apache.activemq=DEBUG

    External process

    log4j.category.com.atlassian.utils.process=DEBUG
    log4j.logger.com.atlassian.bamboo.executor=DEBUG

    Git

    log4j.logger.com.atlassian.bamboo.plugins.git.GitRepository=DEBUG

    Artifact transfer

    log4j.logger.com.atlassian.bamboo.v2.build.agent.messages.ArtifactStreams=DEBUG
    log4j.logger.com.atlassian.bamboo.build.artifact=DEBUG
    log4j.logger.com.atlassian.bamboo.build.artifact.BambooArtifactHttpTransport=DEBUG

  3. Restart both Bamboo Server and Remote Agent (the above-debugging classes will also take effect on Agent logs after the restart) 

Option 2: Debug only the Remote Agent

In case you only need Agent-specific debug logs, or if you can't restart your Bamboo server application, you can enable logging only on the Remote Agent side by following the below steps:

  1. Stop the Remote Agent
  2. Copy the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties file from the Bamboo Server to the Remote Agent's <bamboo-agent-home>/conf
  3. Edit the log4j.properties on the Remote Agent and modify the following property from OFF to ALL or one of the provided log levels: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL. For more information about log levels, check the Log4j API documentation.

    log4j.appender.console.threshold = [FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL]
  4. Edit the log4j.properties based on debugging classes suggested in the table above
  5. Configure your agent to load the log4j.properties file:

    • Uncomment the line following on <bamboo-agent-home>/conf/wrapper.conf and add the absolute path to the log4j.properties file you just created:

      #wrapper.java.additional.3=-Dlog4j.configuration=
    • It should look like this after the change:

      wrapper.java.additional.3=-Dlog4j.configuration=<bamboo-agent-home>/conf/log4j.properties
  6. Restart the Remote Agent

Option 3: Debug the Elastic Agent with Log4j 1.x

To add extra debug flags to the Elastic Agent, create a custom image that would hardcode the necessary Log4j properties and startup flags or use an Instance Startup Script. The instructions below have been tested on Bamboo's stock Linux image running Java 11; you may have to adapt it to a Windows equivalent version.

  1. Edit the Image Configuration
  2. Add the following Script to the Instance Startup Script:

    cat <<EOF>/home/bamboo/log4j.properties
    <put the log4j 1.x content you want> -- use the Remote Agent's log4j content as an example
    EOF
    
    chown bamboo:bamboo /home/bamboo/log4j.properties
    
    sudo su -c "echo \"export JDK_JAVA_OPTIONS='\"-Dlog4j.configuration=/home/bamboo/log4j.properties\"'\" >> /etc/profile.d/bamboo.sh"
  3. The next time you start the image, it will load the log4j.properties file and provide you with the extra debug logs

Bamboo 9 and later with Log4j 2.x

Option 1: Debug the Bamboo server and Remote Agent (requires an application restart)

  1. Edit the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j2.properties
  2. The debug classes can be added on demand in your log4j2.properties file depending on what problem is being troubleshot. Atlassian Support may let you know which types to enable.

    SubjectClasses to debug
    Agent Registration

    logger.agent-added.level=DEBUG
    logger.agent-added.name=com.atlassian.bamboo.agent

    logger.buildqueue-added.level=DEBUG
    logger.buildqueue-added.name=com.atlassian.bamboo.buildqueue

    logger.buildagent-added.level=DEBUG
    logger.buildagent-added.name=com.atlassian.bamboo.v2.build.agent

    Agent load

    logger.activemq-added.level=DEBUG
    logger.activemq-added.name=org.apache.activemq

    External process

    logger.utils-added.level=DEBUG
    logger.utils-added.name=com.atlassian.utils.process

    logger.executor-added.level=DEBUG
    logger.executor-added.name=com.atlassian.bamboo.executor

    Git

    logger.git-added.level=DEBUG
    logger.git-added.name=com.atlassian.bamboo.plugins.git.GitRepository

    Artifact transfer

    logger.artifact-streams.level=DEBUG
    logger.artifact-streams.name=com.atlassian.bamboo.v2.build.agent.messages.ArtifactStreams

    logger.artifact.level=DEBUG
    logger.artifact.name=com.atlassian.bamboo.build.artifact

    logger.artifact-http-transport.level=DEBUG
    logger.artifact-http-transport.name=com.atlassian.bamboo.build.artifact.BambooArtifactHttpTransport

  3. Restart both Bamboo Server and Remote Agent (the above-debugging classes will also take effect on Agent logs after the restart) 

Option 2: Debug only the Remote Agent

In case you only need Agent-specific debug logs, or if you can't restart your Bamboo server application, you can enable logging only on the Remote Agent side by following the below steps:

  1. Stop the Remote Agent
  2. Copy the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j2.properties file from the Bamboo Server to the Remote Agent's <bamboo-agent-home>/conf
  3. Edit the log4j2.properties on the Agent, and modify the following property from OFF to ALL or one of the provided log levels: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL. For more information about log levels, check the Log4j2 API documentation API documentation.

    appender.console.filter.threshold.level = [FATAL, ERROR, WARNING, INFO, DEBUG, TRACE, ALL]
  4. Edit the log4j2.properties based on debugging classes suggested in the table above
  5. Configure your agent to load the log4j2.properties file:
    Uncomment the line following on <bamboo-agent-home>/conf/wrapper.conf and add the absolute path to the log4j2.properties file you just copied:

    #wrapper.java.additional.3=-Dlog4j2.configurationFile

    It should look like this after the change:

    wrapper.java.additional.3=-Dlog4j2.configurationFile=<bamboo-agent-home>/conf/log4j2.properties
  6. Due to BAM-22466 - Getting issue details... STATUS , add the following line to <bamboo-agent-home>/conf/wrapper.conf to allow the Tanuki Service Wrapper to load extra Bamboo classes so the custom logging works:

    wrapper.java.classpath.3=../classpath/*.jar
  7. Restart the Remote Agent

Option 3: Debug the Elastic Agent with Log4j 2.x

To add extra debug flags to the Elastic Agent, create a custom image that would hardcode the necessary Log4j properties and startup flags or use an Instance Startup Script. The instructions below have been tested on Bamboo's stock Linux image running Java 11; you may have to adapt it to a Windows equivalent version.

  1. Edit the Image Configuration
  2. Add the following Script to the Instance Startup Script:

    cat <<EOF>/home/bamboo/log4j2.properties
    <put the log4j 2.x content you want> -- use the Remote Agent's as an example
    EOF
    
    chown bamboo:bamboo /home/bamboo/log4j2.properties
    
    sudo su -c "echo \"export JDK_JAVA_OPTIONS='\"-Dlog4j2.configurationFile=/home/bamboo/log4j2.properties\"'\" >> /etc/profile.d/bamboo.sh"
  3. The next time you start the image, it will load the log4j2.properties file and provide you with the extra debug logs

Debugging Source Code Git Checkout tasks

The following guidelines provide valuable information for troubleshooting Bamboo's Git checkout tasks without using Script Tasks. This is essential for examining problems concerning the Bamboo Agent and the VCS repository while maintaining Bamboo's internal SSH-PROXY authentication active.

It is important to note that debugging a Bamboo checkout task can be complex and may require some modification to succeed. The instructions provided below are tailored for Linux Agents; however, adjustments may be necessary for Windows, using a PowerShell approach.

  1. On the Bamboo Agent (or on the Server if you are using Local Agents), create the following wrapper script:

    git.wrapper
    #!/bin/bash
    
    # Bamboo depends on fetch and the version's standard format, don't debug it
    if [[ $1 = "fetch" || $1 = "version" ]]; then
      exec /usr/bin/git $*
    else
      exec /usr/bin/env GIT_TRACE=1 GIT_CURL_VERBOSE=1 GIT_TRANSFER_TRACE=1 GIT_TRACE_SHALLOW=1 /usr/bin/git $*
    fi
  2. Set the correct ownership and executable permissions to the Git wrapper script using chmod +x; chown;
  3. In Bamboo, modify that specific Agent's capabilities and change Git to point to the wrapper script
  4. On the Agent, delete the following folders and their content to force Git to fetch the repository completely:
    • <bamboo-agent-home>/xml-data/build-dir/PROJ-PLAN-JOB
    • <bamboo-agent-home>/xml-data/build-dir/_git-repositories-cache/<cache_folder_for_the_repository>
  5. Modify the Plan's requirements to point to that specific Agent. You can also dedicate the Agent to the Plan in case you don't want to modify the Plan structure
  6. Run the Plan

The Git debug logs will be very verbose. They will be sent to stdout, meaning they will be available on the Job build logs and will not appear on the Agent logs.

More information

For more info on how to increase log verbosity on remote agents:

Extra debugging: Investigating packet content

For a deeper investigation of packet content, network traffic dumps between the agent and the server are helpful. Please enable the network packet capturing on the server side and reproduce the problem. Please refer to Capturing HTTP traffic using Wireshark or Fiddler for more details on capturing the network packet.

Last modified on Aug 27, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.