Symptoms

Some Jira Incoming Mail Handlers configured in ⚙ > System > Incoming Mails > Mail Handlers stop processing emails. This issue is intermittent.

The purpose of this KB article is to describe a possible root cause for this type of issue, how to identify it, and how to fix it. If after following the diagnosis steps, you find that this article is not relevant, then a different root cause might be behind this issue. In such case, we advise that you reach out to Atlassian Support for further assistance.

Note that this KB article only applies to the Jira Incoming Mail Handler feature, and not the Jira Service Management Mail Channel feature.

Diagnosis

• The IMAP (or POP3) Mail Server associated to the stuck mail handlers are configured with a timeout value set to 0 in ⚙ > System > Incoming Mails > Mail Servers
  
  

• After enabling the debugging package com.atlassian.jira.service, if you grep the Mail Handler name in the log file atlassian-jira.log, you should see:

  • that the last execution of this mail handler never completed (there is a "Running Service" message, but no "Finished Running Service" message afterwards):

    2021-04-29 15:36:55,081+0200 Caesium-1-3 DEBUG anonymous    Some Mail Handler [c.a.jira.service.ServiceRunner] Finished Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
    2021-04-29 15:37:52,638+0200 Caesium-1-3 DEBUG ServiceRunner    Some Mail Handler [c.a.jira.service.ServiceRunner] Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
    2021-04-29 15:37:55,100+0200 Caesium-1-3 DEBUG anonymous    Some Mail Handler [c.a.jira.service.ServiceRunner] Finished Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
    2021-04-29 15:38:52,639+0200 Caesium-1-4 DEBUG ServiceRunner    Some Mail Handler [c.a.jira.service.ServiceRunner] Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
    

  • If you are using Jira Data Center, then you will see that any recent execution of the mail handler is skipped, because this mail handler is currently stuck:

    2021-05-05 00:40:02,381+0200 DEBUG [] Caesium-1-1 anonymous    Some Mail Handler MailFetcherService run() method has been called
    2021-05-05 00:40:02,382+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Execution id: 40000
    2021-05-05 00:40:02,383+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.43301 lock. Skipping the run.
    2021-05-05 00:42:02,437+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Execution id: 40000
    2021-05-05 00:42:02,437+0200 DEBUG [] Caesium-1-1 anonymous    Some Mail Handler MailFetcherService run() method has been called
    2021-05-05 00:42:02,439+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.43301 lock. Skipping the run.
    2021-05-05 00:48:02,484+0200 DEBUG [] Caesium-1-1 anonymous    Some Mail Handler MailFetcherService run() method has been called
    2021-05-05 00:48:02,485+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Execution id: 40000
    

• Taking thread dumps while the mail handler is stuck will show that the same mail thread keeps showing in any thread dump in the RUNNABLE state, indicating that the mail handler is stuck trying to connect to the Mail Server to fetch incoming mails:

  "Caesium-1-3" daemon prio=5 tid=0x00000000000001fa nid=0 runnable 
     java.lang.Thread.State: RUNNABLE
  	at java.net.SocketInputStream.socketRead0(Native Method)
  	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  	at java.net.SocketInputStream.read(SocketInputStream.java:171)
  	at java.net.SocketInputStream.read(SocketInputStream.java:141)
  	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
  	at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
  	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
  	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
  	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
  	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
  	- locked <0x00000000698b5834> (a sun.security.ssl.TransportContext)
  	at com.sun.mail.util.SocketFetcher.configureSSLSocket(SocketFetcher.java:602)
  	at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:376)
  	at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:214)
  	at com.sun.mail.pop3.Protocol.<init>(Protocol.java:103)
  	at com.sun.mail.pop3.POP3Store.getPort(POP3Store.java:249)
  	- locked <0x000000007b102adc> (a com.sun.mail.pop3.POP3SSLStore)
  	at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:191)
  	at javax.mail.Service.connect(Service.java:342)
  	- locked <0x000000007b102adc> (a com.sun.mail.pop3.POP3SSLStore)
  	at javax.mail.Service.connect(Service.java:222)
  	at javax.mail.Service.connect(Service.java:243)
  	at com.atlassian.mail.server.InternalAuthenticationContext.connectService(InternalAuthenticationContext.java:58)
  	at com.atlassian.mail.server.AbstractMailServer.smartConnect(AbstractMailServer.java:156)
  	at com.atlassian.jira.service.services.mail.MailServicesHelper$$Lambda$2229/184044302.connect(Unknown Source)
  	at com.atlassian.jira.service.services.mail.MailServicesHelper.lambda$connectUsing$2(MailServicesHelper.java:81)
  	at com.atlassian.jira.service.services.mail.MailServicesHelper$$Lambda$2233/571355549.apply(Unknown Source)
  	at java.util.Optional.map(Optional.java:215)
  	at com.atlassian.jira.service.services.mail.MailServicesHelper.handleAuthAwareMailServer(MailServicesHelper.java:52)
  	at com.atlassian.jira.service.services.mail.MailServicesHelper.getConnectedStore(MailServicesHelper.java:46)
  	at com.atlassian.jira.service.services.mail.MailFetcherService.getConnectedStore(MailFetcherService.java:397)
  	at com.atlassian.jira.service.services.mail.MailFetcherService$MessageProviderImpl.getAndProcessMail(MailFetcherService.java:166)
  	at com.atlassian.jira.service.services.mail.MailFetcherService.processMessages(MailFetcherService.java:361)
  	at com.atlassian.jira.service.services.mail.MailFetcherService.runImpl(MailFetcherService.java:353)
  	at com.atlassian.jira.service.services.file.AbstractMessageHandlingService.run(AbstractMessageHandlingService.java:229)
  	at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:68)
  	at com.atlassian.jira.service.ServiceRunner.runService(ServiceRunner.java:62)
  	at com.atlassian.jira.service.ServiceRunner.runServiceId(ServiceRunner.java:44)
  	at com.atlassian.jira.service.ServiceRunner.runJob(ServiceRunner.java:32)
  	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
  	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
  	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$$Lambda$2029/185588655.accept(Unknown Source)
  	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
  	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
  	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
  	at java.lang.Thread.run(Thread.java:748)

Cause

When the Incoming Mail Server  is configured with a connection timeout set to 0 or no connection timeout set at all, the Mail Handler is waiting infinitely to get a connection from the POP3 (or IMAP) server. If for any reason the connection is unstable while the Mail Handler is trying to connect to the mail server, it might get stuck in an infinite state. The consequence of that is that the mail handler will be completely stuck and the only way to fix this situation is either to re-start Jira or to delete and re-create the mail handler.

Resolution

• Change the Incoming Mail server timeout value to a non 0 value (for example the default value 10000 ms). This workaround will prevent any mail handler from being stuck in the future.
• If any mail handler was already stuck before changing the time out value, then you will need to use either of the 2 methods below to reset this mail handler:
  • Method 1: re-start the Jira application
  • Method 2: delete the mail handler and re-create it