Jira incoming mail and outgoing mail functionalities are not running due to an add-on taking all the Caesium resources

Still need help?

The Atlassian Community is here for you.

Ask the community

Symptoms

The 2 following symptoms might be observed simultaneously:

  • All notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue. The emails might be eventually sent, but with a very long delay
  • If Jira (or Jira Service Management) is configured with mail handlers, the mail handlers are not running on schedule (every 1 minute by default) and incoming mails are not processed, or processed with a very long delay

Diagnosis

  • If the debugging package com.atlassian.jira.service has been enabled in the page ⚙ > System > Logging and Profiling, we should see in the file atlassian-jira.log that:

    • the Mail Queue Service (responsible to empty the mail queue) is not scheduled every 1 min as it should. Instead, it is scheduled at random intervals (or it's never running):

      grep 'Mail Queue Service' atlassian-jira.log
      
      
      2020-07-01 12:00:38,278+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service 
      2020-07-01 12:16:23,597+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service 
      2020-07-01 12:35:49,036+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service 
      2020-07-01 12:48:40,567+0000 Caesium-1-1 DEBUG anonymous Mail Queue Service [c.a.j.s.services.mail.MailQueueService] Attempting to run mail queue service
  • If the debugging package com.atlassian.jira.service has been enabled in the page ⚙ > System > Logging and Profiling, and if logging and debugging has been enabled for Incoming Mail, we should see in the file atlassian-jira-incoming-mail.log that:

    • The Jira Mail Handler(s) are not running at all, or are running at random times with a big delay:

      grep -h 'Execution id: ' atlassian-jira-incoming-mail.log* | sort
      2021-01-22 14:03:13,492+0300 DEBUG [Incoming Mail] Caesium-1-1 anonymous    Mail Handler Execution id: 36300
      2021-01-22 16:37:04,126+0300 DEBUG [Incoming Mail] Caesium-1-1 anonymous    Mail Handler Execution id: 36301
    • The Service Desk Mail Handler(s) are not running at all, or are running at random times with a big delay: 

      grep -h 'Start running MailProcessor ...' atlassian-jira-incoming-mail.log* | sort
      2021-01-22 17:36:42,092+0300 DEBUG [] Caesium-1-1 anonymous     Start running MailProcessor ...
      2021-01-22 18:39:08,527+0300 DEBUG [] Caesium-1-1 anonymous     Start running MailProcessor ...
      2021-01-22 22:40:11,904+0300 DEBUG [] Caesium-1-1 anonymous     Start running MailProcessor ...
      
  • Taking thread dumps while the problem is happening, you should see that most (or all) of the 4 Caesium threads are used by some method starting with com.intenso.jira.plugins.admanager, in various statuses (blocked, waiting, or runnable). Below is an example of thread dumps showing 3 Caesium threads in BLOCKED status and 1 Caesium thread in WAITING status, all used by com.intenso.jira.plugins.admanager:

    • Caesium thread 1:

      "Caesium-1-1" daemon prio=5 tid=0x0000000000000197 nid=0 waiting for monitor entry 
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl.process(SyncJobRunnerImpl.java:44)
      	- waiting to lock <0x00000000115a47cc> (a com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl)
      	 owned by Caesium-1-3 id=0x0000000000000199
    • Caesium thread 2:

      "Caesium-1-2" daemon prio=5 tid=0x0000000000000198 nid=0 waiting for monitor entry 
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl.process(SyncJobRunnerImpl.java:44)
      	- waiting to lock <0x00000000115a47cc> (a com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl)
      	 owned by Caesium-1-3 id=0x0000000000000199
    • Caesium thread 3:

      "Caesium-1-3" daemon prio=5 tid=0x0000000000000199 nid=0 waiting on condition 
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for <0x000000006304431e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
      	at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
      	at com.sun.jndi.ldap.LdapRequest.getReplyBer(Unknown Source)
      	at com.sun.jndi.ldap.Connection.readReply(Unknown Source)
      	at com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source)
      	at com.sun.jndi.ldap.LdapClient.search(Unknown Source)
      	at com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source)
      	at com.sun.jndi.ldap.LdapCtx.searchAux(Unknown Source)
      	at com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source)
      	at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(Unknown Source)
      	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source)
      	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source)
      	at javax.naming.directory.InitialDirContext.search(Unknown Source)
      	at com.intenso.jira.plugins.admanager.service.ADIntegrationConServiceImpl.getReffToUser(ADIntegrationConServiceImpl.java:
    • Caesium thread 4:

      "Caesium-1-4" daemon prio=5 tid=0x000000000000019a nid=0 waiting for monitor entry 
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl.process(SyncJobRunnerImpl.java:44)
      	- waiting to lock <0x00000000115a47cc> (a com.intenso.jira.plugins.admanager.scheduler.SyncJobRunnerImpl)
      	 owned by Caesium-1-3 id=0x0000000000000199
  • The add-on Active Directory Attributes Sync is installed in the Jira instance. When disabling it from ⚙ > Manage Apps > Manage Apps, the issue goes away:

    • the mail queue is automatically emptied
    • the Jira (and Service Desk) Mail Handlers process incoming mails on schedule
  • When enabling plugin safe mode in Jira, the issue goes away

Cause

Jira only has 4 Caesium threads used to run scheduled jobs (mail queue service, jira xml backup service, mail handler services, etc...).

If for any reason the 4 Caesium threads are constantly busy running services from add-ons, these Caesium threads are never or rarely available for the Jira Mail Queue service and Mail Handlers to execute their tasks. As a consequence, the outgoing mail queue keeps piling up and incoming mails are not processed.

Note that any party add-on that uses scheduled services could be excessively using the Caesium threads. Some add-ons which have been known to cause this issue are:

Enabling plugin safe mode in Jira will help you confirm that there is a 3rd party add-on which is causing this issue. You can identify which add-on is the problematic one by quitting the safe mode, and then disabling each add-on 1 by 1 until the issue goes away.

Workaround

Short Term workaround

Long Term workaround

  • If you need to keep this add-on enabled, open a support ticket on the add-on provider support platform to get further assistance




Last modified on Mar 23, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.