Jira notifications piling up in the mail queue due to a server hostname resolution issue

Still need help?

The Atlassian Community is here for you.

Ask the community

Symptoms

All notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue and are sent with a very long delay.

Diagnosis

  • Looking at atlassian-jira-outgoing-mail debug logs, we can see that each email is being sent with a long delay (20 seconds in the example below):

    grep 'was sent with Message-Id' atlassian-jira-outgoing-mail.log:
    
    2020-07-27 04:53:41,256+0200 DEBUG [] Sending mailitem To='email1@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@24d56930' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32599.1595818401102@Atlassian.JIRA>
    2020-07-27 04:54:01,456+0200 DEBUG [] Sending mailitem To='email2@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@3b62f1d5' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32600.1595818441274@Atlassian.JIRA>
    2020-07-27 04:54:21,639+0200 DEBUG [] Sending mailitem To='email3@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@19310596' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32601.1595818481472@Atlassian.JIRA>
    
  • The root cause from Jira notifications piling up in the mail queue due to IPv6 issues on JVM has already been checked. Basically, the customer is already using the IPv4 look up method by using the following JVM parameter:

    -Djava.net.preferIPv4Stack=true
    
  • Taking thread dumps while the problem is happening will reveal that the mail threads are taking a long time in the method java.net.Inet4AddressImpl.lookupAllHostAddr:

    "Sending mailitem To='bok@moyastacja.pl' Subject='[ASD-47415] - POTWIERDZENIE PRZELEWU - ADAMEX ADAM CZAJKOWSKI ' From='null' FromName='Damian Kowalski' Cc='null' Bcc='null' ReplyTo='pomoc@anwim.pl' InReplyTo='null' MimeType='text/plain' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@2cb1caa8' MessageId='null' ExcludeSubjectPrefix=true'" #241 daemon prio=5 os_prio=0 tid=0x00007ff58eba6000 nid=0x4361 runnable [0x00007ff54665a000]
       java.lang.Thread.State: RUNNABLE
    	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
    	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    	at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
    	- locked <0x00000004d2ea2e30> (a java.lang.Object)
    	at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:274)
    	- locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport)
    	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:730)
    	- locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport)
    	at javax.mail.Service.connect(Service.java:388)
    	- locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport)
    	at javax.mail.Service.connect(Service.java:246)
    	at javax.mail.Service.connect(Service.java:195)
    	at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:185)
    	at com.atlassian.mail.queue.SingleMailQueueItem.send(SingleMailQueueItem.java:44)
    
  • Download this java class onto the Jira Server,  run the following command from the same folder where this class was downloaded and check the time it takes for the Jira server to resolve its own hostname. In the example below, the "real" value is showing "20 seconds", which is extremely slow would delay the sending of each email by at least 20 seconds:

    time <JAVA_BIN_FOLDER>/java GetLocalHostName -Djava.net.preferIPv4Stack=true
    
    Your current IP address : jiraserver/172.10.10.10
    Your current Hostname : jiraserver
    
    real    0m20.148s
    user    0m0.091s
    sys     0m0.063s
    

Cause

In order to send notification emails, Jira needs to perform a handshake with the SMTP server, during which it needs to resolve the server hostname. Since the local DNS server is taking longer than it should to resolve Jira server's hostname, this adds a huge delay to the sending of each single Jira notification (since each Jira opens a new connection with the SMTP server to send each single notification).

Resolution

Engage with the Jira server admin and the network admin to investigate why the DNS resolution takes so much time on the Jira server.

Workaround

A possible workaround is to have your Jira server hosts file updated with the hostname and IP address of the server Jira is running on:

  • To know which hostname and IP address to add to the hosts file, check the hostname and IP address returned by the command time <JAVA_BIN_FOLDER>/java GetLocalHostName -Djava.net.preferIPv4Stack=true during the diagnosis steps above. Then add the following line in the hosts file (/etc/hosts for Linux):

    <IP_ADDRESS> <SERVER_HOSTNAME>
    
  • If we take the example provided in the diagnosis steps, the line to add to the hosts file will be:

    172.10.10.10 jiraserver
    


Last modified on Dec 4, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.