Jira/Service Managements notifications are piling up in the mail queue due to the bug JRASERVER-62072

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.

    

Summary

All types of notification emails from Jira/Service Management (batched notifications, non-batched notifications, customer notifications) are piling up in the mail queue and are not sent automatically:



Please note that the symptoms will be a bit different, depending on the type of environment among the 2 types listed below:

  • Jira Server and Jira Data Center single node
    • The Mail Queue keeps piling up until it is manually flushed by a Jira admin user
    • Notifications (of any type) will completely stop being sent
  • Jira Data Center multi-nodes
    • The Mail Queue will keep piling on only on the impacted Jira nodes (each Jira node is managing its own Mail Queue), but is automatically emptied on the other Jira nodes
    • Notifications (of any type) will intermittently not being sent, depending on which Jira node the notification was triggered from:
      • If the notification was triggered from a node with a functioning Mail Queue service, it will be sent as expected
      • If the notification was triggered from a node with a non-functioning Mail Queue service (due to this bug), it will not be sent and be stuck in the mail queue until it's manually flushed by a Jira admin user

Environment

Any Jira version 7.x/8.x, and any type of Jira environment (Jira Server, Jira Data Center single node, Jira Data Center multi-nodes)

Diagnosis

  • The issue impacts any type of notification from either Jira Software or Jira Service Management projects (Jira batched notifications, Jira non-batched notifications, Service Management Customer Notifications)
  • The mail queue can be emptied by manually flushing it from the UI from the page ⚙ > System > Mail Queue (it's just not flushed automatically)
  • Diagnosis step only for Jira Server and Jira Data Center single node:

    • Run the following SQL query to check the running status of the Mail Queue Service (referred to in the database as com.atlassian.jira.service.JiraService:10000):

      select * from rundetails where job_id = 'com.atlassian.jira.service.JiraService:10000';
    • Check the output of the SQL query. If in the output, you get 1 single row with the outcome "S", and this row has a timestamp from a very long time ago (hours ago, days ago...), then you might be impacted by the bug. In the example below, the query was executed on Jan 18th 2022, and the row is from Dec 26th 2021, showing that the Mail Queue Service has not been scheduled for weeks:

      id","job_id","start_time","run_duration","run_outcome","info_message"
      "50376528","com.atlassian.jira.service.JiraService:10000","2021-12-26 01:13:00-05","1","S","
  • Look into the Jira application logs and check if you can find:

    • an error showing that the Jira scheduler failed to queue the Mail Queue Service (com.atlassian.jira.service.JiraService:10000) due to some SQL exception (note that the timestamp of the error should be close to the timestamp of the oldest email that is stuck in the Mail Queue):

      2021-12-26 01:14:00,729 Caesium-1-3 ERROR ServiceRunner     [c.a.s.caesium.impl.SchedulerQueueWorker] Unhandled exception thrown by job QueuedJob[jobId=com.atlassian.jira.service.JiraService:10000,deadline=1646509320000]
      com.opensymphony.module.propertyset.PropertyImplementationException: Unable to load values for CacheKey[entityName=jira.properties,entityId=1]
      	at com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.propEx(CachingOfBizPropertyEntryStore.java:374)
      	at com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.resolve(CachingOfBizPropertyEntryStore.java:128)
      	at com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.getEntry(CachingOfBizPropertyEntryStore.java:151)
      	at com.atlassian.jira.propertyset.CachingOfBizPropertySet.get(CachingOfBizPropertySet.java:189)
      	at com.opensymphony.module.propertyset.AbstractPropertySet.getString(AbstractPropertySet.java:305)
      	at com.atlassian.jira.config.properties.ApplicationPropertiesStore.getStringFromDb(ApplicationPropertiesStore.java:234)
      	at com.atlassian.jira.config.properties.ApplicationPropertiesImpl.getString(ApplicationPropertiesImpl.java:53)
      	at com.atlassian.jira.scheduler.JiraCaesiumSchedulerConfiguration.getDefaultTimeZone(JiraCaesiumSchedulerConfiguration.java:30)
      	at com.atlassian.scheduler.caesium.impl.RunTimeCalculator.getTimeZone(RunTimeCalculator.java:115)
      	at com.atlassian.scheduler.caesium.impl.RunTimeCalculator.nextRunTime(RunTimeCalculator.java:96)
      	at com.atlassian.scheduler.caesium.impl.RunTimeCalculator.nextRunTime(RunTimeCalculator.java:70)
      	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.calculateNextRunTime(CaesiumSchedulerService.java:444)
      	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:401)
      	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:380)
      	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
      	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
      	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: com.atlassian.cache.CacheException: com.atlassian.jira.exception.DataAccessException: com.microsoft.sqlserver.jdbc.SQLServerException: Cannot open database "XXXXXXXX" requested by the login. The login failed. ClientConnectionId:XXXXXXXXXXXXX
      	at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:113)
      	at com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.resolve(CachingOfBizPropertyEntryStore.java:126)
      	... 16 more
    • or any database connection error around the same day/time from the SQL query, similar to the example below (note that the timestamp of the error should be close to the timestamp of the oldest email that is stuck in the Mail Queue):

      2021-12-26 01:14:00,337-0500 periodic-de-index-fixer-0 ERROR      [c.a.jira.database.DatabaseAccessorImpl] Unable to rollback SQL connection.
      org.postgresql.util.PSQLException: This connection has been closed.
      	at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:782)
      	at org.postgresql.jdbc.PgConnection.rollback(PgConnection.java:789)
      	at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:488)
      	at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:488)
      	at org.ofbiz.core.entity.jdbc.interceptors.connection.DelegatingConnection.rollback(DelegatingConnection.java:61)
      	at com.atlassian.jira.ofbiz.sql.ConnectionWrapper.rollback(ConnectionWrapper.java:72)
      	at com.atlassian.jira.database.DatabaseAccessorImpl.executeQuery(DatabaseAccessorImpl.java:78)
      	at com.atlassian.jira.database.DatabaseAccessorImpl.runInTransaction(DatabaseAccessorImpl.java:100)

Cause

If Jira experiences a temporary database connectivity issue (for example, during a DB server maintenance or re-start) while it is trying to schedule a job/service (such as the Mail Queue Service which is responsible to empty the Mail Queue), such service might never recover from this outage, and the Jira scheduler will never try to execute this job anymore. This behavior is tracked in the following bug ticket:

JRASERVER-62072 - Getting issue details... STATUS

Solution

Since the bug has not been fixed yet, the only way to ensure that the Mail Queue Service can be automatically scheduled again is to re-start the Jira application by following the steps below:

  • For Jira Server / Jira Data Center (JDC) single node
    • (Optional) Go to ⚙ > System > Mail Queue, and manually flush the queue (otherwise all the pending notifications will be lost after the Jira re-start)
    • Restart the Jira application
    • Go back to ⚙ > System > Mail Queue, and verify that the Mail Queue is regularly emptied (every 1 min by default)
  • For Jira Data Center (JDC) multi node
    • (Optional) Go to the ⚙ > System > Mail Queue page of the impacted Jira node(s), and manually flush the queue (otherwise all the pending notifications will be lost after the Jira re-start)
    • Restart the impacted Jira node(s)
    • Go back to the ⚙ > System > Mail Queue page of the impacted Jira node(s), and verify that the Mail Queue is regularly emptied (every 1 min by default)


Last modified on Mar 21, 2022

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.