Jira notifications piling up in the mail queue due to SMTP server infinite timeout setting

Summary

Symptoms

All notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue and are not sent automatically at all.

Diagnosis

• Emails can be sent by manually flushing the queue from the page ⚙ > System > Mail Queue

• After enabling the debugging package com.atlassian.jira.service, you should not see any trace of the Mail Queue Service in atlassian-jira.log, indicating that the job is either stuck, or not scheduled at all:

  1 2 3 grep 'Mail Queue Service' atlassian-jira.log -> no result

• When running the following query, you’ll see 2 rows (1 with “S” and an old start time, and 1 with “A”). This indicates that the mail queue service has been stuck for a while:

  1 2 3 4 5 select * from rundetails where job_id = 'com.atlassian.jira.service.JiraService:10000'; "ID","JOB_ID","START_TIME","RUN_DURATION","RUN_OUTCOME","INFO_MESSAGE" 24491947,com.atlassian.jira.service.JiraService:10000,"2020-07-27 11:28:00.0",0,S,"" 24614357,com.atlassian.jira.service.JiraService:10000,"2020-08-07 11:12:00.0",0,A,Already running

• Taking Generating a Thread Dump while the problem is happening will show that the same mail thread keeps showing in any thread dump (in the RUNNABLE state):

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 "Sending mailitem To='name.surname@atlassian.com' Subject='(ISSUEID-1300) Issue Description: Jira Notifications piling up in the queue' From='null' FromName='Name Surname (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@10f21874' MessageId='null' ExcludeSubjectPrefix=false'" daemon prio=5 tid=0x000000000000009c nid=0 runnable java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:126) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x0000000008586485> (a java.io.BufferedInputStream) at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:104) at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2425) at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2172) at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:726) - locked <0x0000000008fd87a1> (a com.sun.mail.smtp.SMTPTransport) at javax.mail.Service.connect(Service.java:366) - locked <0x0000000008fd87a1> (a com.sun.mail.smtp.SMTPTransport) at javax.mail.Service.connect(Service.java:246) at javax.mail.Service.connect(Service.java:195) at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:182) at com.atlassian.mail.queue.SingleMailQueueItem.send(SingleMailQueueItem.java:44) at com.atlassian.mail.queue.MailQueueImpl.sendBufferUnderLock(MailQueueImpl.java:103) at com.atlassian.mail.queue.MailQueueImpl.sendBuffer(MailQueueImpl.java:56) at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:51) at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:48)

• Check if the SMTP mail server is configured with an infinite timeout. To check it, go to the page ⚙> System > Outgoing Mails, click Edit next to the mail server:

  • If there is nothing specified in the JNDI location field, then check if connection timeout field is set to 0. If it's set to 0, then it means that no timeout value is set (infinite timeout):

    

  • If there is something specific in the JNDI location field, then it means that the SMTP server is not configure in the UI, but in the JIRA_INSTALL/conf/server.xml file

    

    • In this case, look into the server.xml file and check if the parameters mail.smtp.connectiontimeout and mail.smtp.timeout are set. If either or both of these 2 parameters are missing, then it means that the SMTP mail server is configured with no timeout value (=infinite timeout):

      1 2 3 4 5 6 7 8 9 10 11 12 <Resource name="mail/Office365SmtpServer" auth="Container" type="javax.mail.Session" mail.smtp.host="smtp.office365.com" mail.smtp.port="587" mail.smtp.auth="true" mail.smtp.user="test@test.com" password="Sanitized by Support Utility" mail.smtp.starttls.enable="true" mail.smtp.ssl.protocols="TLSv1.2" mail.transport.protocol="smtps" />

  • Either way, if you see that no timeout value is set (=infinite timeout) for the SMTP Mail Server, then the resolution from this KB applies

Cause

When the Outgoing Mail Server (SMTP mail server) is configured with a connection timeout set to 0 or no connection timeout set at all, the Mail Queue Service is waiting infinitely to get a connection from SMTP server. If for any reason the connection is unstable while the mail queue is in the middle of sending a notification, it might get stuck in that infinite state. The consequence of that is that the mail queue service is no longer scheduled every minute, and the mail queue indefinitely piles up.

Solution

Resolution

• Change the Outgoing Mail server timeout value to a non 0 value (for example 10000):

  • If the mail server is configured in the UI in the page ⚙> System > Outgoing Mails:

    • Set the connection timeout to the default value (10000)

  • If the mail server is configured in the server.xml file (JNDI location):

    • Add the 2 following parameters to the SMTP mail server config in the server.xml file:

      1 2 mail.smtp.connectiontimeout="10000" mail.smtp.timeout="10000"

• After you make this change, a Jira restart is required