Jira Mail Handlers get intermittently stuck due to the POP3/IMAP server infinite timeout setting

Still need help?

The Atlassian Community is here for you.

Ask the community

Symptoms

Some Jira Mail Handlers configured in ⚙ > System > Incoming Mails > Mail Handlers stop processing emails. This issue is intermittent.

Diagnosis

  • The IMAP (or POP3) Mail Server associated to the stuck mail handlers are configured with a timeout value set to 0 in ⚙ > System > Incoming Mails > Mail Servers

  • After enabling the debugging package com.atlassian.jira.service, if you grep the Mail Handler name in the log file atlassian-jira.log, you should see:

    • that the last execution of this mail handler never completed (there is a "Running Service" message, but no "Finished Running Service" message afterwards):

      2021-04-29 15:36:55,081+0200 Caesium-1-3 DEBUG anonymous    Some Mail Handler [c.a.jira.service.ServiceRunner] Finished Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
      2021-04-29 15:37:52,638+0200 Caesium-1-3 DEBUG ServiceRunner    Some Mail Handler [c.a.jira.service.ServiceRunner] Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
      2021-04-29 15:37:55,100+0200 Caesium-1-3 DEBUG anonymous    Some Mail Handler [c.a.jira.service.ServiceRunner] Finished Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
      2021-04-29 15:38:52,639+0200 Caesium-1-4 DEBUG ServiceRunner    Some Mail Handler [c.a.jira.service.ServiceRunner] Running Service [Container: com.atlassian.jira.service.services.mail.MailFetcherService  delay [60000ms]]
      
    • If you are using Jira Data Center, then you will see that any recent execution of the mail handler is skipped, because this mail handler is currently stuck:

      2021-05-05 00:40:02,381+0200 DEBUG [] Caesium-1-1 anonymous    Some Mail Handler MailFetcherService run() method has been called
      2021-05-05 00:40:02,382+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Execution id: 40000
      2021-05-05 00:40:02,383+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.43301 lock. Skipping the run.
      2021-05-05 00:42:02,437+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Execution id: 40000
      2021-05-05 00:42:02,437+0200 DEBUG [] Caesium-1-1 anonymous    Some Mail Handler MailFetcherService run() method has been called
      2021-05-05 00:42:02,439+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Unable to acquire com.atlassian.jira.service.services.mail.MailFetcherService.43301 lock. Skipping the run.
      2021-05-05 00:48:02,484+0200 DEBUG [] Caesium-1-1 anonymous    Some Mail Handler MailFetcherService run() method has been called
      2021-05-05 00:48:02,485+0200 DEBUG [Some Mail Server] Caesium-1-1 anonymous    Some Mail Handler Execution id: 40000
      
  • Taking thread dumps while the mail handler is stuck will show that the same mail thread keeps showing in any thread dump in the RUNNABLE state, indicating that the mail handler is stuck trying to connect to the Mail Server to fetch incoming mails:

    "Caesium-1-3" daemon prio=5 tid=0x00000000000001fa nid=0 runnable 
       java.lang.Thread.State: RUNNABLE
    	at java.net.SocketInputStream.socketRead0(Native Method)
    	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    	at java.net.SocketInputStream.read(SocketInputStream.java:171)
    	at java.net.SocketInputStream.read(SocketInputStream.java:141)
    	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
    	at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
    	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
    	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
    	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
    	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
    	- locked <0x00000000698b5834> (a sun.security.ssl.TransportContext)
    	at com.sun.mail.util.SocketFetcher.configureSSLSocket(SocketFetcher.java:602)
    	at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:376)
    	at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:214)
    	at com.sun.mail.pop3.Protocol.<init>(Protocol.java:103)
    	at com.sun.mail.pop3.POP3Store.getPort(POP3Store.java:249)
    	- locked <0x000000007b102adc> (a com.sun.mail.pop3.POP3SSLStore)
    	at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:191)
    	at javax.mail.Service.connect(Service.java:342)
    	- locked <0x000000007b102adc> (a com.sun.mail.pop3.POP3SSLStore)
    	at javax.mail.Service.connect(Service.java:222)
    	at javax.mail.Service.connect(Service.java:243)
    	at com.atlassian.mail.server.InternalAuthenticationContext.connectService(InternalAuthenticationContext.java:58)
    	at com.atlassian.mail.server.AbstractMailServer.smartConnect(AbstractMailServer.java:156)
    	at com.atlassian.jira.service.services.mail.MailServicesHelper$$Lambda$2229/184044302.connect(Unknown Source)
    	at com.atlassian.jira.service.services.mail.MailServicesHelper.lambda$connectUsing$2(MailServicesHelper.java:81)
    	at com.atlassian.jira.service.services.mail.MailServicesHelper$$Lambda$2233/571355549.apply(Unknown Source)
    	at java.util.Optional.map(Optional.java:215)
    	at com.atlassian.jira.service.services.mail.MailServicesHelper.handleAuthAwareMailServer(MailServicesHelper.java:52)
    	at com.atlassian.jira.service.services.mail.MailServicesHelper.getConnectedStore(MailServicesHelper.java:46)
    	at com.atlassian.jira.service.services.mail.MailFetcherService.getConnectedStore(MailFetcherService.java:397)
    	at com.atlassian.jira.service.services.mail.MailFetcherService$MessageProviderImpl.getAndProcessMail(MailFetcherService.java:166)
    	at com.atlassian.jira.service.services.mail.MailFetcherService.processMessages(MailFetcherService.java:361)
    	at com.atlassian.jira.service.services.mail.MailFetcherService.runImpl(MailFetcherService.java:353)
    	at com.atlassian.jira.service.services.file.AbstractMessageHandlingService.run(AbstractMessageHandlingService.java:229)
    	at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:68)
    	at com.atlassian.jira.service.ServiceRunner.runService(ServiceRunner.java:62)
    	at com.atlassian.jira.service.ServiceRunner.runServiceId(ServiceRunner.java:44)
    	at com.atlassian.jira.service.ServiceRunner.runJob(ServiceRunner.java:32)
    	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
    	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
    	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
    	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
    	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
    	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
    	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
    	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$$Lambda$2029/185588655.accept(Unknown Source)
    	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
    	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
    	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
    	at java.lang.Thread.run(Thread.java:748)

Cause

When the Incoming Mail Server  is configured with a connection timeout set to 0 or no connection timeout set at all, the Mail Handler is waiting infinitely to get a connection from the POP3 (or IMAP) server. If for any reason the connection is unstable while the Mail Handler is trying to connect to the mail server, it might get stuck in an infinite state. The consequence of that is that the mail handler will be completely stuck and the only way to fix this situation is either to re-start Jira or to delete and re-create the mail handler.

Resolution

  • Change the Incoming Mail server timeout value to a non 0 value (for example the default value 10000 ms). This workaround will prevent any mail handler from being stuck in the future.
  • If any mail handler was already stuck before changing the time out value, then you will need to use either of the 2 methods below to reset this mail handler:
    • Method 1: re-start the Jira application
    • Method 2: delete the mail handler and re-create it



Last modified on May 5, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.