Full Reindex failing with Out Of Memory due to the size of cache net.sf.ehcache.store.chm.SelectableConcurrentHashMap

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

Full Reindex failing due to OutofMemory caused by SLA timeline calculation.

Environment

4.20.13
.

Diagnosis

  • Enable all the required Java arguments to generate a Heapdump when an Out of Memory occurs. Reference: Analyze OutofMemory errors in Jira server with Heap Dumps and reproduce the problem, running a full reindex;
  • Using a tool to analyze the Heapdump, we could observe a big amount of data related to the SLA timeline. They are loaded into the cache net.sf.ehcache.store.chm.SelectableConcurrentHashMap:

  • A piece of the SLA timeline observed on this cache:
{"timeline":{"events":[{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]}


  • Stack Trace:
Thread Stack 
IssueIndexer:thread-11
  at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I (Native Method)
  at java.net.SocketInputStream.socketRead(Ljava/io/FileDescriptor;[BIII)I (SocketInputStream.java:116)
  at java.net.SocketInputStream.read([BIII)I (SocketInputStream.java:171)
  at java.net.SocketInputStream.read([BII)I (SocketInputStream.java:141)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(I)V (ReadAheadInputStream.java:101)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary([BII)I (ReadAheadInputStream.java:144)
  at com.mysql.jdbc.util.ReadAheadInputStream.read([BII)I (ReadAheadInputStream.java:174)
  at com.mysql.jdbc.MysqlIO.readFully(Ljava/io/InputStream;[BII)I (MysqlIO.java:2966)
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3427)
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3417)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3860)
  at com.mysql.jdbc.MysqlIO.sendCommand(ILjava/lang/String;Lcom/mysql/jdbc/Buffer;ZLjava/lang/String;I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:2524)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;Ljava/lang/String;Lcom/mysql/jdbc/Buffer;IIIZLjava/lang/String;[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetInternalMethods; (MysqlIO.java:2675)
  at com.mysql.jdbc.ConnectionImpl.execSQL(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;ILcom/mysql/jdbc/Buffer;IIZLjava/lang/String;[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (ConnectionImpl.java:2465)
  at com.mysql.jdbc.PreparedStatement.executeInternal(ILcom/mysql/jdbc/Buffer;ZZ[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (PreparedStatement.java:1915)
  at com.mysql.jdbc.PreparedStatement.executeQuery()Ljava/sql/ResultSet; (PreparedStatement.java:2023)
  at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
  at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
  at com.atlassian.jira.ofbiz.sql.PreparedStatementWrapper.executeQuery()Ljava/sql/ResultSet; (PreparedStatementWrapper.java:42)
  at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.lambda$executeQuery$5()Ljava/sql/ResultSet; (DiagnosticPreparedStatement.java:59)
  at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement$$Lambda$2806.execute()Ljava/lang/Object; (Unknown Source)
  at com.atlassian.diagnostics.internal.platform.monitor.db.DefaultDatabaseDiagnosticsCollector.recordExecutionTime(Lcom/atlassian/diagnostics/internal/platform/monitor/db/SqlOperation;Ljava/lang/String;)Ljava/lang/Object; (DefaultDatabaseDiagnosticsCollector.java:70)
  at com.atlassian.jira.diagnostic.connection.DatabaseDiagnosticsCollectorDelegate.recordExecutionTime(Lcom/atlassian/diagnostics/internal/platform/monitor/db/SqlOperation;Ljava/lang/String;)Ljava/lang/Object; (DatabaseDiagnosticsCollectorDelegate.java:55)
  at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DiagnosticPreparedStatement.java:59)
  at com.querydsl.sql.AbstractSQLQuery.iterateSingle(Lcom/querydsl/core/QueryMetadata;Lcom/querydsl/core/types/Expression;)Lcom/mysema/commons/lang/CloseableIterator; (AbstractSQLQuery.java:370)
  at com.querydsl.sql.AbstractSQLQuery.iterate()Lcom/mysema/commons/lang/CloseableIterator; (AbstractSQLQuery.java:342)
  at com.querydsl.core.group.GroupByMap.transform(Lcom/querydsl/core/FetchableQuery;)Ljava/util/Map; (GroupByMap.java:54)
  at com.querydsl.core.group.GroupByMap.transform(Lcom/querydsl/core/FetchableQuery;)Ljava/lang/Object; (GroupByMap.java:35)
  at com.querydsl.core.support.FetchableQueryBase.transform(Lcom/querydsl/core/ResultTransformer;)Ljava/lang/Object; (FetchableQueryBase.java:55)
  at com.atlassian.servicedesk.internal.sla.customfield.JIRACustomFieldValueStore.lambda$getAllSLATextValues$0(Ljava/lang/Long;Lcom/atlassian/pocketknife/api/querydsl/DatabaseConnection;)Ljava/util/Map; (JIRACustomFieldValueStore.java:42)
  at com.atlassian.servicedesk.internal.sla.customfield.JIRACustomFieldValueStore$$Lambda$3208.apply(Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.lambda$execute$0(Ljava/util/function/Function;Ljava/sql/Connection;)Ljava/lang/Object; (DatabaseAccessorImpl.java:69)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl$$Lambda$3179.execute(Ljava/sql/Connection;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.executeInternal(Ljava/sql/Connection;Lcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (DefaultTransactionalExecutor.java:87)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.lambda$execute$0(Lcom/atlassian/sal/api/rdbms/ConnectionCallback;Ljava/sql/Connection;)Ljava/lang/Object; (DefaultTransactionalExecutor.java:43)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor$$Lambda$1289.execute(Ljava/sql/Connection;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor$$Lambda$1290.apply(Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.jira.database.DatabaseAccessorImpl.runInManagedTransaction(Ljava/util/function/Function;)Ljava/lang/Object; (DatabaseAccessorImpl.java:131)
  at sun.reflect.GeneratedMethodAccessor138.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Method.java:498)
  at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (ContextClassLoaderSettingInvocationHandler.java:26)
  at com.sun.proxy.$Proxy313.runInManagedTransaction(Ljava/util/function/Function;)Ljava/lang/Object; (Unknown Source)
  at sun.reflect.GeneratedMethodAccessor138.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Method.java:498)
  at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (HostComponentFactoryBean.java:130)
  at com.sun.proxy.$Proxy313.runInManagedTransaction(Ljava/util/function/Function;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.runInStartedOrExistingTransaction(Lcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (JiraHostConnectionAccessor.java:130)
  at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.execute(ZZLcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (JiraHostConnectionAccessor.java:60)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.execute(Lcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (DefaultTransactionalExecutor.java:40)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.execute(Ljava/util/function/Function;ZZLcom/atlassian/pocketknife/api/querydsl/util/OnRollback;)Ljava/lang/Object; (DatabaseAccessorImpl.java:67)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.runInTransaction(Ljava/util/function/Function;Lcom/atlassian/pocketknife/api/querydsl/util/OnRollback;)Ljava/lang/Object; (DatabaseAccessorImpl.java:43)
  at com.atlassian.servicedesk.internal.sla.customfield.JIRACustomFieldValueStore.getAllSLATextValues(Ljava/lang/Long;)


  • SQL statistics collected by the DBA while running a reindex are showing high response time for SLA table:
RankQuery IDResponse time%CallsR/CallV/MItem
10x8845FDA210EBA1B41693978.716338.2%5922861.450587...SELECT UNION AO_?E_SLAAUDITLOG


  • During the reindex, we can observe warnings about issues that will not have the SLA timeline indexed because they are marked as too big:
2022-11-14 09:40:50,189-0800 IssueIndexer:thread-2 WARN xxxx /secure/admin/IndexReIndex!reindex.jspa [c.a.s.i.sla.searcher.SLACustomFieldIndexer] Not indexing timeline for xxxx-99999, because it is too big.


Cause

These SLA timelines are big due to the SLA configuration, causing the issues to have a long history of start/stop/pause events. 
Although the cache is configured to automatically clear items out after 10 minutes,  the EHCache documentation says "elements can stay in the cache after expiration and actually expired elements are only evicted when it is accessed or resource constraints are reached".

We've run into customers experiencing out-of-memory issues previously as we used to allow an unlimited size for the cache, however, that was resolved as part of the ticket JSDSERVER-5468 - Getting issue details... STATUS .

A default maximum value of 2 million items (where items are the number of issues) was added for nodes in cluster environments, and it was made configurable by the system property servicedesk.slavalue.global.cache.size.limit. This value can be configured to a smaller value, for instance, 250k, reducing the number of issues loaded into the cache.


Solution

Edit the jira-config.properties file and add the system property into the jira-config.properties file (Reference: Edit the jira-config.properties file in Jira server)

  • Shut down your JIRA applications;
  • Open the jira-config.properties file (located at the root of your JIRA application home directory) in a text editor;

 This file may not exist if you are using a new installation or an upgraded installation where your previous JIRA application versions had never been customized. If this file does not exist, create it using a text editor.

  • Add the property servicedesk.slavalue.global.cache.size.limit=250000 in this file;
  • Save your modifications to the jira-config.properties file;
  • Restart your JIRA applications.


Last modified on Jan 3, 2023

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.