Bamboo broker ActiveMQ service stopped with error: KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND java.io.IOException: Async Writer Thread Shutdown

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

Bamboo ActiveMQ service stopped causing agents to be unresponsive.

Environment

Observed on Bamboo 8.2.5 but might apply to other versions as well.

Diagnosis

The following can be seen in the Bamboo logs:

ERROR [ActiveMQ Journal Checkpoint Worker] [MessageDatabase] Checkpoint failed
java.io.IOException: Permission denied
        at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
        at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
        at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
2022-10-25 15:48:23,236 ERROR [ActiveMQ NIO Worker 30] [MessageDatabase] KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND
java.io.IOException: Async Writer Thread Shutdown
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.enqueue(DataFileAppender.java:164)
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:129)
        at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:961)
        at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1152)
        at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1134)
        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:566)
        at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addMessage(KahaDBTransactionStore.java:407)
        at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.addMessage(KahaDBTransactionStore.java:161)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:888)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:295)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
        at org.apache.activemq.broker.util.TimeStampingBrokerPlugin.send(TimeStampingBrokerPlugin.java:132)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:580)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:769)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:335)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
        at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
        at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:162)
        at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:81)
        at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:98)
        at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
INFO [ActiveMQ Journal Checkpoint Worker] [DefaultIOExceptionHandler] Stopping BrokerService[bamboo] due to exception, java.io.IOException: Permission denied
java.io.IOException: Permission denied
        at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
        at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
        at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)

Agents will then become unresponsive because the broker service stopped:

WARN [scheduler_Worker-2] [RemoteAgentManagerImpl] Detected that remote agent 'bamboo-agent1' has been inactive since <date>
WARN [scheduler_Worker-2] [RemoteAgentManagerImpl] Marking remote agent 'bamboo-agent1' as unresponsive

Cause

Kahadb lock files are not accessible because of permission issues.

Solution

Check the permissions for the <Bamboo-Home>/shared/jms-store/bamboo/KahaDB/ directory and confirm if the Bamboo user is able to access this directory.

In case of a cluster configuration with the shared home deployed on an NFS system, please check if the NFS system has the correct permissions and access lists.

Last modified on Nov 16, 2022

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.