Notification mail slow to arrive or having huge delay

Still need help?

The Atlassian Community is here for you.

Ask the community

This article only applies to Atlassian's server products. Learn more about the differences between cloud and server.

Problem

The notification mail for all events are slow to reach the users, sometimes can reach up to hours to reach the intended recipient. No warning or errors is spotted on the mail queue, mails are leaving gradually but slowly.

This does not apply for the mail queue not flushed automatically, for that please refer our Mail Queue Never Flushed Automatically KB. Additionally, the Troubleshooting JIRA Service Desk KB may offer further clarification on the way JIRA applications handle their services.

Diagnosis

There could be multiple causes for this to happen, you will need to diagnose following the steps below to see if which could be the cause to the observation:

Mail Queue Service Delay

Mail Queue Services are scheduled to be running by default at once per minute. But sometimes due to user preference, it can be altered to different delay and this will take direct effect on how long it will be delayed to reach the end user. This can be verify by navigating to Administration > System > Advanced > Services and check if the mail queue is set to different value, revert back to 1 minute and see if this helps. Proceed on with the Diagnosis if this value is set as default.

Mail Queue Error Message Stuck

There could be times where there is error message not able to being flush and therefore JIRA application attempts to flush the mail again and again causing other mail items to be delayed on sending. But this can be identified from the logs itself or visually monitor the mail queue. Navigate to mail queue and perform manual flush and see if there is any mail not leaving the queue, if you manage to identify which mail is causing the delay, perhaps you could reach out to https://support.atlassian.com for further assistance.

Services and Third Party Plugins Services

By default, JIRA applications services are running on either 2 or 4 QuartzWorker threads - this depends upon the version of JIRA applications. There are some obstruction or delay on the threads executing, the next service may not be executed despite having the services schedule to run. To diagnose, follow steps below:

  1. Navigate to Administration > Troubleshooting and Support > Logging and Profiling.
  2. Look for the Default Loggers and click on the Configured logging level for another package:

    If JIRA application is restarted, these extra logging levels will be disabled and will need to be re-enabled upon each startup.

    1. com.atlassian.jira.service.services.mail
    2. com.atlassian.mail.queue (this will not appear when you check on the step below, but it will still be triggered).
  3. Enable the Outgoing Mail log and the Incoming Mail Log.
  4. Verify that the added parameter is appearing at the list below (hint: try to search using the browser search function, it should appear as DEBUG after you enabled.)
  5. Keep the logs running for 48 hours or a period of time depending on your preference, as long as it captures the peak hour activities.


  6. From the logs generated ($JIRA_HOME/log Directory) filter for the string "Attempting to run mail queue service" (hint: Linux based machine can use grep to print the result into a file, for Windows, utilize the findstr to print the result into a file). You will notice something like below as example: 

    2013-06-24 07:26:40,565 QuartzWorker-1 DEBUG ServiceRunner    Mail Queue Service [service.services.mail.MailQueueService] Attempting to run mail queue service
    2013-06-24 07:28:40,570 QuartzWorker-1 DEBUG ServiceRunner    Mail Queue Service [service.services.mail.MailQueueService] Attempting to run mail queue service
    2013-06-24 07:30:41,587 QuartzWorker-0 DEBUG ServiceRunner    Mail Queue Service [service.services.mail.MailQueueService] Attempting to run mail queue service
    2013-06-24 07:32:40,570 QuartzWorker-0 DEBUG ServiceRunner    Mail Queue Service [service.services.mail.MailQueueService] Attempting to run mail queue service
    2013-06-24 07:59:45,790 QuartzWorker-0 DEBUG ServiceRunner    Mail Queue Service [service.services.mail.MailQueueService] Attempting to run mail queue service
  7. Compare the timing of the execution, from the example given above, you will notice that the service is executed 2 minutes once and sometimes it overshoot to almost 30 minutes. If your result is consistently 1 minute, it shows that most likely it could be caused by other factors, for example the reason below:

Network and Mail Server

  1. Verify that you are able to reach the mail server by ping and check if the latency is high on delivery.
  2. If the latency is high this will causes the delay for the JavaMail used by JIRA applications to receive the response in timely manner.
  3. It could also be the load that the mail server is overloading the SMTP and therefore unable to complete the requests on time.
  4. Consult the network administrators and mail server administrators if you encountered such problem.

Incoming Mail Handlers

We have also seen issues where Incoming Mail Handlers take a long time to scan mailboxes connected to JIRA due to the mailbox housing a very large amount of messages. This causes the Caesium threads to not be released as often as they would, increasing the time it takes for the Mail Queue Service to run.

Reverse DNS

Every time mail is attempted to be sent, it will perform a reverse DNS lookup for the JIRA application server hostname. If the hostname isn't reachable, JIRA application will have to wait for for a timeout which can be a long period of time (20-40 seconds). 

Filter subscriptions to big lists or for slow JQL

There are could be subscriptions being sent to huge user groups or slow queries being run as part of the filters that are subscribed to. 

Resolution

Depending what is the reason behind the delay of the mail service, follow the resolution below:

Mail Queue Service Delay

Deduct the mail queue delay on the service to a shorter delay that fits your need.

Mail Queue Error Message Stuck

  • Check if the stuck mail contains any information that might prompt towards the root cause (Constant stuck on one particular recipient, or notification for one particular issue, or one particular type of event)
  • Provide your observation and data collected to https://support.atlassian.com for further analysis on the root cause (Please help to generate a full support zip (Administration > System > Atlassian Support Tool > Support Zip) for better diagnosis at our end)

Services and Third Party Plugins Services

  • Check the services running, if you have more than 5 services running at the same delay as the mail queue (default 1 minute), it could delay the service to be executed as it will need to wait for other service to be completed on time. Set proper delay for services so it can be scheduled to run at different time, example: mail handler service to run 5 minute once and so forth.
  • Adjust the delay higher for other services at the mean time to determine which service is causing the problem (larger the delay will enable clearer view in the logs), see if the mail queue is getting executed on time. Readjust after you have determined which is the root cause behind the delay.
  • Feedback to Third Party Plugin vendor if you found a particular service on that is provided by the Third Party takes longer time to execute and complete.

Network and Mail Server

  • Consult the network administrator or mail server administrator to further troubleshoot on this.

Incoming Mail Handlers

Clearing the mailboxes used to get mail into JIRA can also help with speeding the mail fetch process, which will end up allowing the Mail Queue Service to run as often as it should.

Reverse DNS

  • Verify the hosts file is correct.
  • Verify the hostname of the server is correct.
  • Ping the hostname:

    ping <HOSTNAME>
  • Perform a nslookup on the IP that is returned:

    nslookup <IP_ADDRESS_FROM_PING>

     

  • (Linux only): Perform a DIG lookup:

    dig -x IP <IP_ADDRESS_FROM_PING>

Filter subscriptions to big lists or for slow JQL

This problem may be caused by the bug below:

JRA-61543 - Getting issue details... STATUS

The query below can be used to retrieve the list of filters, their JQL query, the groups subscribed to those filters and the number of members in the group:

SELECT sr.filtername, sr.reqcontent, fs.groupname, count(1) as group_members_count
FROM filtersubscription fs
JOIN searchrequest sr on sr.id = fs.filter_i_d
JOIN cwd_membership cm on cm.parent_name = fs.groupname
WHERE membership_type = 'GROUP_USER'
GROUP BY sr.filtername, sr.reqcontent, fs.groupname
ORDER BY 4 DESC;

You may also check the atlassian-jira-slow-queries.log file if the JQL used by the offending e-mail subscription is taking a long time to complete.

Additionally, our Troubleshooting JIRA Services may offer assistance here as well.

Description

This page covers the steps needed to resolve the following issue. The notification mail for all events are slow to reach the users, sometimes can reach up to hours to reach the intended recipient. No warning or errors is spotted on the mail queue, mails are leaving gradually but slowly.

Product Jira



Last modified on Jan 31, 2019

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.