Remote agents are disconnected due to the ActiveMQ broker blocking all queues

Summary

Prior to Bamboo 9.4, a remote agent’s default socket inactivity timeout period was 5 minutes. This meant that if a remote agent terminated abruptly, Bamboo wouldn’t close the transport to the terminated agent for up to 5 minutes. If the dead transport was used, ActiveMQ could potentially close all queues, disconnecting all remote agents.

In Bamboo 9.4, the issue was addressed for fresh instances by reducing the maximum inactivity duration to 90 seconds and introducing a socket timeout of 45 seconds on the transport. The configuration of active instances upgraded to Bamboo 9.4 must be updated manually.

Diagnosis

All remote agents disconnect abruptly.

Cause

The default socket inactivity timeout causes dead ActiveMQ transports to be kept open for up to 5 minutes after the abrupt termination of a remote agent.

Solution

If you’ve just installed and configured a fresh instance of Bamboo 9.4, no manual intervention is required. In Bamboo 9.4, the new inactivity duration and socket timeout values are applied to the broker and broker client by default.

If you’ve just upgraded an active instance to Bamboo 9.4, manually apply the inactivity duration and socket timeout values to the broker and broker client URLs. You can do that in the Bamboo web interface or by editing the bamboo.cfg.xml configuration file.

Apply the changes to the broker and broker client URLs through the Bamboo web interface

To apply the changes to the broker and broker client URLs through the Bamboo web interface:

1. From the upper-right corner of the screen, select Administration > Overview.

2. In the menu on the left, under System, select General configuration.

3. In the Broker URL field, set the wireFormat.maxInactivityDuration parameter to 90000 and add the soWriteTimeout parameter with a value of 45000. For example:

   1 nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=90000&transport.soWriteTimeout=45000

4. In the Broker client URL field, set the wireFormat.maxInactivityDuration parameter to 90000. For example:

   1 failover:(nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=90000)?initialReconnectDelay=15000&maxReconnectAttempts=10
   The broker client URL doesn’t expect the transport.soWriteTimeout property.

5. Select Save.

6. Restart the Bamboo server.

Apply the changes to the broker and broker client URLs in the bamboo.cfg.xml file

To apply the changes to the broker URL in the bamboo.cfg.xml file:

1. Shut down the Bamboo server and agents.

2. Open the bamboo.cfg.xml file in a text editor.

3. Change the bamboo.jms.broker.uri property value by setting the wireFormat.maxInactivityDuration parameter to 90000 and adding the soWriteTimeout parameter with a value of 45000. For example:

   1 nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=90000&transport.soWriteTimeout=45000

4. Change the bamboo.jms.broker.client.uri property value by setting the wireFormat.maxInactivityDuration parameter to 90000. For example:

   1 failover:(nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=90000)?initialReconnectDelay=15000&maxReconnectAttempts=10
   The broker client URL doesn’t expect the transport.soWriteTimeout property.

5. Save your changes to the bamboo.cfg.xml file.

6. Start up the Bamboo server and agents.