Bamboo remote agents lose connection because of 'No space left on device' error
Problem
Bamboo remote agents lose connection and unable to reconnect.
The following appears in the atlassian-bamboo-agent.log
INFO | jvm 1 | 2012/02/07 23:58:48 | 2012-02-07 23:58:48,514 WARN [Scheduler_Worker-7] [JmsTemplateMessageSender] JmsException when trying to send message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@115ea42. Retrying...
INFO | jvm 1 | 2012/02/07 23:58:48 | org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
...
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: java.lang.Throwable: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
...
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.io.RandomAccessFile.writeBytes(Native Method)
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.io.RandomAccessFile.write(RandomAccessFile.java:482)
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:387)
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:216)The following appears in the atlassian-bamboo.log
2020-12-04 16:47:51,239 ERROR [ActiveMQ Transport: tcp:///1.1.1.1:49810@8080] [MessageDatabase] KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND
java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
...
2020-12-04 16:47:51,526 ERROR [ActiveMQ Connection Executor: vm://bamboo#0] [BambooServerSimpleMessageListenerContainer] Encountered non-recoverable JMSException
javax.jms.JMSException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1960)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:278)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)
... 6 more
Caused by: org.apache.activemq.broker.BrokerStoppedException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:205)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.aCause
There's no space left on the disk where $BAMBOO_HOME is located. This will cause the Bamboo messaging broker to shutdown with an unrecoverable exception as its unable to journal messages.
Resolution
Bamboo must be restarted to recover the broker, but first the disk space issue should be addressed.
- Stop Bamboo
- Clean up or add more storage to the Bamboo server where
$BAMBOO_HOMEis located. - Start Bamboo
In versions Bamboo 6.10.5 or greater, you may also change the location of the ActiveMQ Journal with the below system property if you want to store it on a different location to where $BAMBOO_HOME is:
-Dbamboo.jms.store.directory=/path/to/jms-store
Last modified on Dec 10, 2020
Powered by Confluence and Scroll Viewport.