Summary

Byteman is a tool which makes it easy to trace, monitor and test the behaviour of Java applications. It injects Java code into the application methods or into Java runtime methods without the need to recompile, repackage or even redeploy the application. Atlassian Support may ask you to enable Byteman while troubleshooting issues for a number of reasons (e.g. introduce additional logging, change the behaviour of the application, and etc).

This article explains how to enable Byteman and covers the most common issues when doing so and how to address them.

Solution

Enabling Byteman

1. Stop Bamboo (Server or Data Center).
2. Open the <bamboo-install>/bin/setenv.sh file.

3. Locate the message "# Uncomment the line below to enable the Byteman agent" and uncomment the following line:

   JVM_SUPPORT_RECOMMENDED_ARGS="$JVM_SUPPORT_RECOMMENDED_ARGS -javaagent:../tools/byteman/lib/byteman.jar=listener:true"

    Please make sure that the path to tools/byteman is correct. By default the path should be <bamboo-install>/tools/byteman/lib/byteman.jar

   If you're running Bamboo as a Service please refer to the Configuring your system properties documentation for steps on how to add the Java property above to Bamboo.

4. Start Bamboo.

   You must restart Bamboo when enabling Byteman for the first time. Once it's been turned on you will be able to apply rules and patches without the need for another restart.

Loading Byteman rules

1. Open the byteman folder (<bamboo-install>/tools/byteman).
2. Create a file in the byteman directory to store your rule(s) e.g. <bamboo-install>/tools/byteman/myrule.btm.
3. Copy the rule provided by Atlassian Support into the newly created file e.g. myrule.btm.

4. Run the following command from the <bamboo-install>/tools/byteman folder to load the rule:
   

   ./load-rule.sh myrule.btm

Unloading Byteman rules

If you wish to unload a rule that is currently in effect you should run the following command from the <bamboo-install>/tools/byteman folder to unload the rule:

./unload-rule.sh myrule.btm

Updating logging levels

Atlassian Support may also ask you to set some classes to log in DEBUG level in Bamboo. This can be done through the following steps:

1. Go to the Bamboo administration > Overview > Log settings page.
2. Add the DEBUG classes that will be provided by Atlassian Support.

Troubleshooting Byteman

Known bug when using Java 11 : BAM-21569 - Byteman 3.x doesn't work on Java 11 which has been fixed in 8.1.2, 8.0.6, 8.2.0 (or later)

Connection refused

You may get "Connection refused" errors when trying to load your Byteman rules:

$:/opt/atlassian/bamboo/tools/byteman# ./load-rules.sh myrule.btm
Failed to process request: java.net.ConnectException: Connection refused (Connection refused)
-- Args were: [rule.txt]
java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/java.net.Socket.connect(Socket.java:558)
	at java.base/java.net.Socket.<init>(Socket.java:454)
	at java.base/java.net.Socket.<init>(Socket.java:231)
	at org.jboss.byteman.agent.submit.Submit$Comm.<init>(Submit.java:881)
	at org.jboss.byteman.agent.submit.Submit.submitRequest(Submit.java:787)
	at org.jboss.byteman.agent.submit.Submit.addScripts(Submit.java:603)
	at org.jboss.byteman.agent.submit.Submit.addRulesFromFiles(Submit.java:555)
	at org.jboss.byteman.agent.submit.Submit.main(Submit.java:1147)

Solution 1

Use netstat to check if port 9091 is being utilized by another process on the server. This is the port used by the Byteman agent by default. Here's an example on how to check this on Linux:

netstat -anpe | grep "tcp" | grep "LISTEN"

If there's another application, service, and etc running on port 9091 already you can change the port used by Byteman using the following steps:

1. Stop Bamboo.
2. Open the <bamboo-install>/bin/setenv.sh file.

3. Locate the following line:

   JVM_SUPPORT_RECOMMENDED_ARGS="$JVM_SUPPORT_RECOMMENDED_ARGS -javaagent:../tools/byteman/lib/byteman.jar=listener:true"

4. Add the following attribute to the end of the -javaagent property (you don't have to use port 9999, this is just an example):

   port:9999

5. The -javaagent property will look like this after the change:

   JVM_SUPPORT_RECOMMENDED_ARGS="$JVM_SUPPORT_RECOMMENDED_ARGS -javaagent:../tools/byteman/lib/byteman.jar=listener:true,port:9999"

6. Start Bamboo.

Solution 2

Make sure the environment variable has been correctly set on the <bamboo-install>/bin/setenv.sh file. If there's a typo or if the property has not been properly set follow the steps under the Enabling Byteman section of this document again and restart Bamboo.

Solution 3

Use ps to check if you have more than one Bamboo (Java) process of the same instance running on the server. This can happen if an old Java process was not properly removed from the server after a Bamboo restart. Byteman will not start up properly if there are multiple Bamboo (Java) processes of the same instance running:

ps -fax | grep bamboo

If you see multiple processes for the same Bamboo (Java) instance you should:

1. Stop Bamboo.
2. Run the ps command again until all of the Bamboo (Java) processes are gone.
   1. It may be necessary to issue a kill -9 <PID> command to remove the zombie processes.
3. Start Bamboo.
4. Run the ps command again to check and ensure there aren't multiple Java processes for the same Bamboo instance.