Bamboo unable to connect to mail server due to IPv6 issues on JVM

Still need help?

The Atlassian Community is here for you.

Ask the community

Symptoms

Bamboo is not able to connect to a mail server or mail queue keeps getting overloaded as each message takes a long time to leave the queue. Test emails work as intended. 

Diagnosis

  1. Enable Mail Debug logging in Bamboo
  2. Generate Thread dumps for Bamboo
  3. Looking at Bamboo debug logs, we can see the message "451 Timeout waiting for data from client":
DEBUG SMTP: trying to connect to host "email.server.com", port 587, isSSL false
....
DEBUG SMTP: connected to host "email.server.com", port: 587

EHLO email.server.com
451 Timeout waiting for data from client.
HELO email.server.com
DEBUG SMTP: EOF: [EOF]
2019-05-27 15:41:40,214 WARN [mail-queue-processor] [EmailService] Uncaught exception while processing mail queue
com.atlassian.mail.MailException: javax.mail.MessagingException: [EOF]
	at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:225)
	at com.atlassian.mail.server.impl.SMTPMailServerImpl.send(SMTPMailServerImpl.java:162)
	at com.atlassian.bamboo.mail.EmailService.sendEmailSynchronously(EmailService.java:115)
	at com.atlassian.bamboo.mail.EmailService$MailQueueProcessor.run(EmailService.java:67)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.mail.MessagingException: [EOF]
	at com.sun.mail.smtp.SMTPTransport.issueCommand(SMTPTransport.java:2168)
	at com.sun.mail.smtp.SMTPTransport.helo(SMTPTransport.java:1552)
	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:703)
	at javax.mail.Service.connect(Service.java:386)
	at javax.mail.Service.connect(Service.java:245)
	at javax.mail.Service.connect(Service.java:194)
	at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:185)
	... 4 more

4. The thread dumps will reveal the mail threads are taking a long time in the method java.net.Inet6AddressImpl.lookupAllHostAddr:

"mail-queue-processor" #49 daemon prio=5 os_prio=0 tid=0x00007ff727563000 nid=0x7a63 runnable [0x00007ff7232f3000]
   java.lang.Thread.State: RUNNABLE
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
	at java.net.InetAddress.getLocalHost(InetAddress.java:1501)
	- locked <0x00000000c0dfb8a0> (a java.lang.Object)
	at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:253)
	- locked <0x00000000e00ae788> (a com.sun.mail.smtp.SMTPTransport)
	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:701)
	- locked <0x00000000e00ae788> (a com.sun.mail.smtp.SMTPTransport)

Cause

The IPv6 lookup method can enter an infinite loop when a certain race condition occurs. This is due to DNS servers or the local network not being configured for IPv6. It can occur infrequently and may cause deadlocks where some threads end up blocked.

Resolution

  • Set the following parameter on Bamboo startup:

    -Djava.net.preferIPv4Stack=true
    

    With the parameter stated above set to true, Java will not execute IPv6 lookup methods and the problem should be avoided.


Last modified on Jul 23, 2019

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.