Bamboo broker ActiveMQ service stopped with error: KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND java.io.IOException: Async Writer Thread Shutdown
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bamboo ActiveMQ service stopped causing agents to be unresponsive.
Environment
Observed on Bamboo 8.2.5 but might apply to other versions as well.
Diagnosis
The following can be seen in the Bamboo logs:
ERROR [ActiveMQ Journal Checkpoint Worker] [MessageDatabase] Checkpoint failed
java.io.IOException: Permission denied
at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
2022-10-25 15:48:23,236 ERROR [ActiveMQ NIO Worker 30] [MessageDatabase] KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND
java.io.IOException: Async Writer Thread Shutdown
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.enqueue(DataFileAppender.java:164)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:129)
at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:961)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1152)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1134)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:566)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addMessage(KahaDBTransactionStore.java:407)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.addMessage(KahaDBTransactionStore.java:161)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:888)
at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:295)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.util.TimeStampingBrokerPlugin.send(TimeStampingBrokerPlugin.java:132)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:580)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:769)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:335)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:162)
at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:81)
at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:98)
at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
INFO [ActiveMQ Journal Checkpoint Worker] [DefaultIOExceptionHandler] Stopping BrokerService[bamboo] due to exception, java.io.IOException: Permission denied
java.io.IOException: Permission denied
at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
Agents will then become unresponsive because the broker service stopped:
WARN [scheduler_Worker-2] [RemoteAgentManagerImpl] Detected that remote agent 'bamboo-agent1' has been inactive since <date>
WARN [scheduler_Worker-2] [RemoteAgentManagerImpl] Marking remote agent 'bamboo-agent1' as unresponsiveCause
Kahadb lock files are not accessible because of permission issues.
Solution
Check the permissions for the <Bamboo-Home>/shared/jms-store/bamboo/KahaDB/ directory and confirm if the Bamboo user is able to access this directory.
In case of a cluster configuration with the shared home deployed on an NFS system, please check if the NFS system has the correct permissions and access lists.
Last modified on Nov 16, 2022
Powered by Confluence and Scroll Viewport.