Jira Server or Data Center mail handler and notification email functionalities are running slowly due to an add-on taking all the Caesium threads

Summary

Symptoms

The 2 following symptoms might be observed simultaneously:

• All notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue. The emails might be eventually sent, but with a very long delay

• If Jira (or Jira Service Management) is configured with mail handlers, the mail handlers are not running on schedule (every 1 minute by default) and incoming mails are not processed, or processed with a very long delay

Diagnosis

• If the debugging package com.atlassian.jira.service has been enabled in the page ⚙ > System > Logging and Profiling, we should see in the file atlassian-jira.log that:

  • the Mail Queue Service (responsible to empty the mail queue) is not scheduled every 1 min as it should. Instead, it is scheduled at random intervals (or it's never running):

    1 2 3 4 5 6 7 grep 'Mail Queue Service' atlassian-jira.log 2020-07-01 12:00:38,278+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service 2020-07-01 12:16:23,597+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service 2020-07-01 12:35:49,036+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service 2020-07-01 12:48:40,567+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service

• If the debugging package com.atlassian.jira.service has been enabled in the page ⚙ > System > Logging and Profiling, and if logging and debugging has been enabled for Incoming Mail, we should see in the file atlassian-jira-incoming-mail.log that:

  • The Jira Mail Handler(s) are not running at all, or are running at random times with a big delay:

    1 2 3 grep -h 'Execution id: ' atlassian-jira-incoming-mail.log* | sort 2021-01-22 14:03:13,492+0300 DEBUG [Incoming Mail] Caesium-1-1 anonymous Mail Handler Execution id: 36300 2021-01-22 16:37:04,126+0300 DEBUG [Incoming Mail] Caesium-1-1 anonymous Mail Handler Execution id: 36301

  • The Service Desk Mail Handler(s) are not running at all, or are running at random times with a big delay:

    1 2 3 4 grep -h 'Start running MailProcessor ...' atlassian-jira-incoming-mail.log* | sort 2021-01-22 17:36:42,092+0300 DEBUG [] Caesium-1-1 anonymous Start running MailProcessor ... 2021-01-22 18:39:08,527+0300 DEBUG [] Caesium-1-1 anonymous Start running MailProcessor ... 2021-01-22 22:40:11,904+0300 DEBUG [] Caesium-1-1 anonymous Start running MailProcessor ...

• Taking Generating a Thread Dump while the problem is happening, you should see that most (or all) of the 4 Caesium threads are used by some method starting with com.intenso.jira.plugins.admanager, in various statuses (blocked, waiting, or runnable). Below is an example of thread dumps showing 3 Caesium threads in BLOCKED status and 1 Caesium thread in WAITING status, all used by com.intenso.jira.plugins.admanager:

  • Caesium thread 1:

    1 2 3 4 5 "Caesium-1-1" daemon prio=5 tid=0x0000000000000197 nid=0 waiting for monitor entry java.lang.Thread.State: BLOCKED (on object monitor) at com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl.process(SyncJobRunnerImpl.java:44) - waiting to lock <0x00000000115a47cc> (a com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl) owned by Caesium-1-3 id=0x0000000000000199

  • Caesium thread 2:

    1 2 3 4 5 "Caesium-1-2" daemon prio=5 tid=0x0000000000000198 nid=0 waiting for monitor entry java.lang.Thread.State: BLOCKED (on object monitor) at com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl.process(SyncJobRunnerImpl.java:44) - waiting to lock <0x00000000115a47cc> (a com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl) owned by Caesium-1-3 id=0x0000000000000199

  • Caesium thread 3:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 "Caesium-1-3" daemon prio=5 tid=0x0000000000000199 nid=0 waiting on condition java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000006304431e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at com.sun.jndi.ldap.LdapRequest.getReplyBer(Unknown Source) at com.sun.jndi.ldap.Connection.readReply(Unknown Source) at com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source) at com.sun.jndi.ldap.LdapClient.search(Unknown Source) at com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source) at com.sun.jndi.ldap.LdapCtx.searchAux(Unknown Source) at com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source) at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(Unknown Source) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source) at javax.naming.directory.InitialDirContext.search(Unknown Source) at com.intenso.jira.plugins.admanager.service.ADIntegrationConServiceImpl.getReffToUser(ADIntegrationConServiceImpl.java:

  • Caesium thread 4:

    1 2 3 4 5 "Caesium-1-4" daemon prio=5 tid=0x000000000000019a nid=0 waiting for monitor entry java.lang.Thread.State: BLOCKED (on object monitor) at com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl.process(SyncJobRunnerImpl.java:44) - waiting to lock <0x00000000115a47cc> (a com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl) owned by Caesium-1-3 id=0x0000000000000199

• The add-on Active Directory Attributes Sync is installed in the Jira instance. When disabling it from ⚙ > Manage Apps > Manage Apps, the issue goes away:

  • the mail queue is automatically emptied

  • the Jira (and Service Desk) Mail Handlers process incoming mails on schedule

• When enabling plugin safe mode in Jira, the issue goes away

Cause

Jira only has 4 Caesium threads used to run scheduled jobs (mail queue service, jira xml backup service, mail handler services, etc...).

If for any reason the 4 Caesium threads are constantly busy running services from add-ons, these Caesium threads are never or rarely available for the Jira Mail Queue service and Mail Handlers to execute their tasks. As a consequence, the outgoing mail queue keeps piling up and incoming mails are not processed.

Note that any party add-on that uses scheduled services could be excessively using the Caesium threads. Some add-ons which have been known to cause this issue are:

• The add-on Active Directory Attributes Sync

• The add-on Version & Component Sync for Jira

• The add-on Insight Device42 Integration

Enabling plugin safe mode in Jira will help you confirm that there is a 3rd party add-on which is causing this issue. You can identify which add-on is the problematic one by quitting the safe mode, and then disabling each add-on 1 by 1 until the issue goes away.

Solution

Workaround

Short Term workaround

• Disable the problematic add-on (for example Active Directory Attributes Sync, if this is the one that you identified to be the problem)

Long Term workaround

• If you need to keep this add-on enabled, open a support ticket on the add-on provider support platform to get further assistance