All Remote Agent disconnecting or Transport Connection errors
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Intermittent All Remote Agents drop from Bamboo. Or intermittent Transport Connection Channel was inactive for too (>300000) long errors and evidence of a stuck ActiveMQ VMTransport thread.
Diagnosis
All Remote Agent intermittently drop from Bamboo and Intermittent errors in logs like the following:
WARN [ActiveMQ InactivityMonitor Worker] [Transport] Transport Connection to: tcp://10.10.10.10:49705 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>300000) long: tcp://10.10.10.10:49705The $BAMBOO_HOME/logs/emergency-bamboo.log thread dumps will contain instances where one of the VMTransport threads are stuck in a send:
"ActiveMQ VMTransport: vm://bamboo#1-623" daemon prio=5 tid=0x00000000009ca35f nid=0x9ca35f timed_waiting [496289df]
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:192)
at org.apache.activemq.transport.nio.NIOOutputStream.flush(NIOOutputStream.java:132)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194)
at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335)
at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:94)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1479)
at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:977)
at org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:933)
at org.apache.activemq.broker.TransportConnection.dispatchAsync(TransportConnection.java:943)
at org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:713)
at org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:121)
at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:48)
at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:775)
at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:556)
at org.apache.activemq.broker.region.Topic.send(Topic.java:484)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:508)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:459)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.util.TimeStampingBrokerPlugin.send(TimeStampingBrokerPlugin.java:131)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:578)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:275)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Cause
When Bamboo and a Remote Agent have a network issue, Bamboo checks every 5 minutes if the TCP session is still active. Bamboo does not timeout the session for write operations. Eventually, the session does timeout depending on other settings. There are ways that Bamboo can restart the time even with no action.
Solution
From Bamboo 9.4.0, new installations will default to the Agent timeout settings proposed on this KB article. Upgraded Bamboo instances will still carry the previous settings unchanged. We highly recommend you update them following the instructions below. Please check BAM-20858 - Getting issue details... STATUS for more information.
Adding transport.soWriteTimeout under the Bamboo UI > Cog > Overview > System> General configuration > Broker URL, will allow Bamboo to shut down the sessions if the TCP session has timed out.
- We now check dead transports every 90 seconds by reducing the
maxInactivityDuration - If something gets stuck in a socket write, now it will be timed out in 45 seconds by adding
transport.soWriteTimeout
What we are recommending:
- Edit the
$BAMBOO_HOME/bamboo.cfg.xml Update the
bamboo.jms.broker.uriand include the changes tomaxInactivityDurationand transport.soWriteTimeout. Keep the specified protocol (nio,tcp,ssl, etc) unchanged.<property name="bamboo.jms.broker.uri">nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=90000&transport.soWriteTimeout=45000</property>- Restart Bamboo for the changes to take affect
Please note, if you are making these changes directly in the UI configuration and not within the XML, there is no need to escape the ampersand with & Bamboo will still need to be restarted
Still having problems?
Contact Atlassian Support. We'll be happy to advise you. Please include as much detail as possible, where applicable:
- Logs from the affected application (if applicable)
- Screenshots of the error message (if they're not covered in logs)
- Information about the steps you've taken previously