Bamboo Remote Agents running on JDK 21, JDK 17, JDK 11 and JDK 8u341 or later fails to start with "Software caused connection abort: recv failed" message when using a secure connection (SSL)

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

When installing or running a Bamboo Remote agent on JDK version 11 or 8u341 and later, it refuses to connect to the Bamboo's ActiveMQ JMS broker service port with a "Software caused connection abort: recv failed"  (Java 11) or without an exception and instead with a generic "Connection or outbound has closed, continuing to retry." (Java 8) or message when over SSL.

Java 11, 17 and 21:

Agent logs 
INFO   | jvm 8    | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,601 INFO [AgentRunnerThread] [AgentRegistrationBean] Registering agent on the server,
INFO   | jvm 8    | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,602 ERROR [ActiveMQ Task-1] [FailoverTransport] Failed to connect to [ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=300000] after: 10 attempt(s)
INFO   | jvm 8    | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,617 INFO [ActiveMQ Connection Executor: unconnected] [PooledConnectionFactory] Expiring connection ActiveMQConnection {id=ID:EC2AMAZ-JQCAG7H-49860-1658475932964-1:1,clientId=ID:EC2AMAZ-JHKSJDH-49860-1658475932964-0:1,started=false} on IOException: Software caused connection abort: recv failed

Java 8u341+:

Agent logs 
INFO   | jvm 8    | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,601 INFO [AgentRunnerThread] [AgentRegistrationBean] Registering agent on the server,
INFO   | jvm 8    | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,617 WARN [ActiveMQ Task-1] [FailoverTransport] Failed to connect to [ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=300000] after: 1 attempt(s) with Connection or outbound has closed, continuing to retry.
Server logs 
2022-06-22 15:47:08,577 WARN [ActiveMQ BrokerService[bamboo] Task-2049] [TransportConnector] Could not accept connection from tcp://X.X.X.X:49871: No available authentication scheme (No available authentication scheme)
2022-06-22 15:47:22,365 WARN [ActiveMQ BrokerService[bamboo] Task-2049] [TransportConnector] Could not accept connection from tcp://X.X.X.X:49874: Remote host terminated the handshake (SSL peer shut down incorrectly)

Environment

  • Bamboo Server 8+ with secure agents (SSL) and the bamboo.manage.jms.ssl property is set to true (default)
  • Bamboo Remote Agent running over an updated JDK 21, JDK 17, JDK 11 or JDK 8u341 or later
  • The Bamboo server comes from older releases such as 5 or 6 up to 8 and contains an old <bamboo-home>/shared/configuration/broker.ks keystore file with an old certificate

Diagnosis

The agent tries to connect to the Bamboo Server ActiveMQ JMS port and fails the SSL negotiation.

After enabling the following debug properties on the Bamboo Agent application, along with Java's -Djavax.net.debug=ssl,handshake on the Agent's wrapper.conf:

For Bamboo version 8 and earlier

conf/log4j.properties 
log4j.rootLogger=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d %p [%t] [%c{1}] %m%n
log4j.category.com.atlassian.bamboo=DEBUG
log4j.category.com.atlassian.util.profiling.UtilTimerStack=ALL
log4j.category.org.apache.sshd=DEBUG
log4j.category.org.apache.activemq=DEBUG
conf/wrapper.conf 
wrapper.java.additional.4=-Dlog4j.configuration=<bamboo-agent-home>/conf/log4j.properties
wrapper.java.additional.5=-Djavax.net.debug=ssl,handshake

For Bamboo version 9 and later

conf/log4j2.properties 
packages = com.atlassian.bamboo.log
status = warn
rootLogger=INFO, filelog, console

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{DEFAULT} %p [%t] [%C{1}] %m%n
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = OFF

appender.filelog.type = RollingFile
appender.filelog.name = filelog
appender.filelog.fileName = ../atlassian-bamboo-agent.log
appender.filelog.filePattern = atlassian-bamboo.log-agent.%i
appender.filelog.layout.type = PatternLayout
appender.filelog.layout.pattern = %d{DEFAULT} %p [%t] [%C{1}] %m%n
appender.filelog.policies.type = Policies
appender.filelog.policies.size.type = SizeBasedTriggeringPolicy
appender.filelog.policies.size.size = 100MB
appender.filelog.strategy.type = DefaultRolloverStrategy
appender.filelog.strategy.max = 5
appender.filelog.strategy.fileIndex = min

logger.bamboo-added.level=DEBUG
logger.bamboo-added.name=com.atlassian.bamboo

logger.bamboo-profiling.level=ALL
logger.bamboo-profiling.name=com.atlassian.util.profiling.UtilTimerStack

logger.bamboo-sshd.level=DEBUG
logger.bamboo-sshd.name=org.apache.sshd

logger.bamboo-activemq.level=DEBUG
logger.bamboo-activemq.name=org.apache.activemq
conf/wrapper.conf 
wrapper.java.additional.4=-Dlog4j2.configurationFile=<bamboo-agent-home>/conf/log4j2.properties
wrapper.java.additional.5=-Djavax.net.debug=ssl,handshake


We can see the following SSL handshake issue:

INFO   | jvm 10   | 2022/06/22 17:06:58 | 2022-06-22 17:06:58,549 INFO [AgentRunnerThread] [AgentRegistrationBean] Current agent remote definition: agent7 Remote agent on host agent7
INFO   | jvm 10   | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,632 INFO [AgentRunnerThread] [AgentRegistrationBean] Registering agent on the server,
INFO   | jvm 10   | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,801 DEBUG [ActiveMQ Task-1] [FailoverTransport] urlList connectionList:[ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000], from: [ssl://bamboo.example.net:46804?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000]
INFO   | jvm 10   | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,898 DEBUG [ActiveMQ Task-1] [FailoverTransport] Attempting 0th connect to: ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000
INFO   | jvm 10   | 2022/06/22 17:06:59 | javax.net.ssl|DEBUG|4C|ActiveMQ Task-1|2022-06-22 17:06:59.908 AEST|Utilities.java:73|the previous server name in SNI (type=host_name (0), value=bamboo.example.net) was replaced with (type=host_name (0), value=bamboo.example.net)
INFO   | jvm 10   | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,913 DEBUG [ActiveMQ Task-1] [WireFormatNegotiator] Sending: WireFormatInfo { version=12, properties={TcpNoDelayEnabled=true, SizePrefixDisabled=false, CacheSize=1024, ProviderName=ActiveMQ, StackTraceEnabled=true, PlatformDetails=Java, CacheEnabled=true, TightEncodingEnabled=true, MaxFrameSize=9223372036854775807, MaxInactivityDuration=90000, MaxInactivityDurationInitalDelay=10000, ProviderVersion=5.16.3}, magic=[A,c,t,i,v,e,M,Q]}
(...)
INFO   | jvm 10   | 2022/06/22 17:07:00 | javax.net.ssl|ERROR|4C|ActiveMQ Task-1|2022-06-22 17:07:00.090 AEST|TransportContext.java:345|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking (
INFO   | jvm 10   | 2022/06/22 17:07:00 | "throwable" : {
INFO   | jvm 10   | 2022/06/22 17:07:00 |   javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1696)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:921)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1291)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:123)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1019)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/java.lang.Thread.run(Thread.java:829)
INFO   | jvm 10   | 2022/06/22 17:07:00 |   Caused by: java.io.EOFException: SSL peer shut down incorrectly
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
INFO   | jvm 10   | 2022/06/22 17:07:00 |       ... 19 more}
INFO   | jvm 10   | 2022/06/22 17:07:00 | 
INFO   | jvm 10   | 2022/06/22 17:07:00 | )
INFO   | jvm 10   | 2022/06/22 17:07:00 | 2022-06-22 17:07:00,097 DEBUG [ActiveMQ Task-1] [FailoverTransport] Connect fail to: ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000, reason: {}

We can also observe that Bamboo's ActiveMQ Keystore contains a certificate created long ago:

Bamboo's ActiveMQ broker.ks 
# keytool -list -keystore shared/configuration/broker.ks -storepass bamboo -v
Keystore type: jks
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: jmsbrokerkey
Creation date: 24/03/2016
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=Bamboo ActiveMQ Broker
Issuer: CN=Bamboo ActiveMQ Broker
Serial number: 1ff5b90bc6c5aa
Valid from: Thu Mar 24 15:14:33 AEDT 2016 until: Wed Jun 22 14:14:33 AEST 2016
Certificate fingerprints:
	 SHA1: 43:AD:5D:CA:44:61:DD:79:GG:8A:14:DA:A0:C2:78:94:1C:64:4D:4F
	 SHA256: 7C:5B:C7:67:61:93:29:11:AB:85:B6:99:67:2E:C8:23:06:BA:40:B4:07:94:46:C5:3A:10:FD:62:DA:DB:6A:1B
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 1024-bit DSA key (weak)
Version: 3

*******************************************
*******************************************

Warning:
<jmsbrokerkey> uses a 1024-bit DSA key which is considered a security risk. This key size will be disabled in a future update.
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore shared/configuration/broker.ks.very-old.certificate -destkeystore shared/configuration/broker.ks.very-old.certificate -deststoretype pkcs12".

Cause

JDK 21, JDK17, JDK 11 and JDK 8u341 or later use TLSv1.3 whenever possible. On certain occasions, when the Bamboo Server's ActiveMQ broker.ks keystore and its certificate were created using older Java versions and DSA algorithms, JDK versions 21, 17, 11 and 8u341 or later will fail to negotiate a connection over TLSv1.3, even though it is advertised as compatible.

Solution

This is a mandatory step if the current ActiveMQ Broker certificate is of type DSA. To find out if the Bamboo Server uses a DSA certificate please run the following command and look for Public Key Algorithm: dsaEncryption:

  • openssl s_client -connect <bamboo_server>:<jms_port> <<< "Q" | sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | openssl x509 -text | grep 'Public Key Algorithm:'

Once confirmed, there are two possible solutions to this:

Option 1: Regenerate the Bamboo Server's ActiveMQ certificate

This solution (Option 1) requires you to delete the existing trust store files on every remote agent that has already connected to your Bamboo Server so that it can trust the new certificate which can be a significant amount of manual work depending on how many agents you have and how they are provisioned.

For extra detail, please see How to generate a new Bamboo ActiveMQ Broker Certificate

Summary of the steps:

  1. Completely stop Bamboo
  2. Backup and delete the <bamboo-home>/shared/configuration/broker.ks file
  3. Start the Bamboo Server; a new broker.ks file will be created
  4. On each Remote Agent, delete the <bamboo-agent-home>/configuration/jmsclient.* files so they can request the new certificate and properties from the Server upon reconnection
  5. Make sure that the Java version running on the Remote Agents align with the version running on the Bamboo Server
  6. Restart the Remote Agents

Option 2: Configure Bamboo's ActiveMQ JMS broker service to listen to TLSv1.2 only

  1. Completely stop Bamboo
  2. Apply the transport.enabledProtocols changes as suggested on How to restrict cryptographic protocols used by Bamboo's JMS broker for remote agent communication (item #3) to instruct the Agents to use TLSv1.2
  3. Start the Bamboo Server
  4. Restart the Remote Agents in case they don't reconnect automatically


Last modified on Nov 21, 2024

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.