Jira notifications piling up in the mail queue due to IPv6 issues on JVM

Still need help?

The Atlassian Community is here for you.

Ask the community

Symptoms

Jira is not able to connect to a specific mail server or mail queue keeps getting overloaded as each message takes a long time to leave the queue. As a result all notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue and are sent with a very long delay.

Diagnosis

Looking at atlassian-jira-outgoing-mail debug logs, we can see the message is always taking longer between these two messages below:

2014-05-06 15:36:06,848 DEBUG [jira.local@domain.com] Sending mailitem To='user@domain.com' Subject='(XXX-1) Summary' From='null' FromName='JIRA updates' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='null' MessageId='null' ServiceRunner    Mail Queue Service DEBUG SMTP: connected to host "mail.domain.net", port: 25
2014-05-06 15:36:18,860 DEBUG [jira.local@domain.com] Sending mailitem To='user@domain.com' Subject='(XXX-1) Summary' From='null' FromName='JIRA updates' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='null' MessageId='null' ServiceRunner    Mail Queue Service EHLO jira.domain.net

The following error can also appear on the outgoing mail log files:

2015-02-04 21:00:26,605 WARN [Mail Server - JIRA] QuartzScheduler_Worker-0 ServiceRunner    TEST TEST[10100]: 
javax.mail.MessagingException: Connect failed while connecting to host 'mailserver.example.com' as user 'user@example.com' via protocol 'smtp': javax.mail.MessagingException: Connect failed;
  nested exception is:
	java.net.UnknownHostException: mailserver.example.com
javax.mail.MessagingException: Connect failed;


Taking thread dumps while the problem is happening will reveal the mail threads are taking a long time in the method java.net.Inet6AddressImpl.lookupAllHostAddr:

java.lang.Thread.State: RUNNABLE
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(Unknown Source)
	at java.net.InetAddress.getAddressesFromNameService(Unknown Source)
	at java.net.InetAddress.getLocalHost(Unknown Source)
	- locked <0x00000007802e1cc8> (a java.lang.Object)
	at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:235)
	- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:646)
	- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
	at javax.mail.Service.connect(Service.java:295)
	- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
(...) 

Cause

The IPv6 lookup method can enter an infinite loop when a certain race condition occurs. This occurs infrequently, however can cause deadlocks where some threads end up blocked.

Please check the Java bug: https://bugs.openjdk.java.net/browse/JDK-8158038 for more information.

Resolution

  • Set the following parameter on JIRA startup:

    -Djava.net.preferIPv4Stack=true
    

    With the parameter stated above set to true, Java will not execute IPv6 lookup methods and the problem should be avoided.


  • Please ensure you have your JIRA server hosts file updated with JIRA server hostname and IP addresses, both IPv4 and IPv6.

Last modified on Dec 4, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.