Re-index on Jira fails due to an OutOfMemoryError when connected to a MySQL database

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

When attempting to perform a re-index on the instance connected to a MySQL database, it fails due to an OutOfMemoryError, even if the heap size is adequated based on our Jira Sizing Guide. The following appears in the atlassian-jira.log:

2019-01-07 12:49:13,367 JiraTaskExectionThread-1 WARN admin@admin.com 763x101x1 1plfa7f 192.168.1.1 /secure/admin/IndexReIndex.jspa [c.a.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '1897157'
2019-01-07 12:49:13,367 JiraTaskExectionThread-1 WARN admin@admin.com  763x101x1 1plfa7f 192.168.1.1 /secure/admin/IndexReIndex.jspa [c.a.jira.index.AccumulatingResultBuilder] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.atlassian.jira.index.FutureResult.await(FutureResult.java:29)
    at com.atlassian.jira.index.AccumulatingResultBuilder.collectResult(AccumulatingResultBuilder.java:93)
    at com.atlassian.jira.index.AccumulatingResultBuilder.addInternal(AccumulatingResultBuilder.java:58)
    at com.atlassian.jira.index.AccumulatingResultBuilder.add(AccumulatingResultBuilder.java:35)
    at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:1014)
    at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:991)
    at com.atlassian.jira.issue.index.DefaultIndexManager.lambda$reIndexAll$0(DefaultIndexManager.java:328)
    at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:377)

This issue has not been observed yet on different database types.

Diagnosis

A detailed analysis of a heap dump prior to the crash illustrates that the top consumers on the dominator tree are related to indexing threads, as shown on example below:

Class Name                                                   | Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x67ec03900  IssueIndexer:thread-7 Thread |          120 | 1,862,908,776 |     29.55%
java.lang.Thread @ 0x67ec0a278  IssueIndexer:thread-6 Thread |          120 | 1,000,382,352 |     15.87%
java.lang.Thread @ 0x67ec0c530  IssueIndexer:thread-3 Thread |          120 |   800,610,288 |     12.70%
java.lang.Thread @ 0x67ec08270  IssueIndexer:thread-9 Thread |          120 |   585,851,264 |      9.29%
java.lang.Thread @ 0x67ec01328  IssueIndexer:thread-1 Thread |          120 |   585,616,720 |      9.29%
java.lang.Thread @ 0x67ec05dc8  IssueIndexer:thread-10 Thread|          120 |   322,734,384 |      5.12%
java.lang.Thread @ 0x67ea705c0  IssueIndexer:thread-8 Thread |          120 |   322,068,320 |      5.11%
java.lang.Thread @ 0x67ebfeec0  IssueIndexer:thread-4 Thread |          120 |   136,338,712 |      2.16%
java.lang.Thread @ 0x67e885448  IssueIndexer:thread-5 Thread |          120 |   101,507,576 |      1.61%
java.lang.Thread @ 0x67ebfc968  IssueIndexer:thread-2 Thread |          120 |   100,766,272 |      1.60%

By checking the equivalent thread dump for the top consumer object we can see the following stack trace:

IssueIndexer:thread-7
  at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I (Native Method)
  at java.net.SocketInputStream.socketRead(Ljava/io/FileDescriptor;[BIII)I (SocketInputStream.java:116)
  at java.net.SocketInputStream.read([BIII)I (SocketInputStream.java:171)
  at java.net.SocketInputStream.read([BII)I (SocketInputStream.java:141)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(I)V (ReadAheadInputStream.java:101)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary([BII)I (ReadAheadInputStream.java:144)
  at com.mysql.jdbc.util.ReadAheadInputStream.read([BII)I (ReadAheadInputStream.java:174)
  at com.mysql.jdbc.MysqlIO.readFully(Ljava/io/InputStream;[BII)I (MysqlIO.java:3008)
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3469)
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3459)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3900)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket()Lcom/mysql/jdbc/Buffer; (MysqlIO.java:873)
  at com.mysql.jdbc.MysqlIO.nextRow([Lcom/mysql/jdbc/Field;IZIZZZLcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/ResultSetRow; (MysqlIO.java:1996)
  at com.mysql.jdbc.MysqlIO.readSingleRowSet(JIIZ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/RowData; (MysqlIO.java:3410)
  at com.mysql.jdbc.MysqlIO.getResultSet(Lcom/mysql/jdbc/StatementImpl;JIIIZLjava/lang/String;Z[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:470)
  at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:3112)
  at com.mysql.jdbc.MysqlIO.readAllResults(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:2341)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;Ljava/lang/String;Lcom/mysql/jdbc/Buffer;IIIZLjava/lang/String;[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetInternalMethods; (MysqlIO.java:2736)
  at com.mysql.jdbc.ConnectionImpl.execSQL(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;ILcom/mysql/jdbc/Buffer;IIZLjava/lang/String;[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (ConnectionImpl.java:2484)
  at com.mysql.jdbc.PreparedStatement.executeInternal(ILcom/mysql/jdbc/Buffer;ZZ[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (PreparedStatement.java:1858)
  at com.mysql.jdbc.PreparedStatement.executeQuery()Ljava/sql/ResultSet; (PreparedStatement.java:1966)
  at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
  at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
  at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery()Ljava/sql/ResultSet; (SQLProcessor.java:527)
  at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(Lorg/ofbiz/core/entity/jdbc/SQLProcessor;Ljava/lang/String;Lorg/ofbiz/core/entity/EntityFindOptions;Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/List;Ljava/util/List;Ljava/util/List;Lorg/ofbiz/core/entity/GenericDAO$TableCleanUp;)Lorg/ofbiz/core/entity/EntityListIterator; (GenericDAO.java:877)
  at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(Lorg/ofbiz/core/entity/model/ModelEntity;Lorg/ofbiz/core/entity/EntityCondition;Lorg/ofbiz/core/entity/EntityCondition;Ljava/util/Collection;Ljava/util/List;Lorg/ofbiz/core/entity/EntityFindOptions;)Lorg/ofbiz/core/entity/EntityListIterator; (GenericDAO.java:857)
  at org.ofbiz.core.entity.GenericDAO.selectByAnd(Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericDAO.java:729)
  at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericHelperDAO.java:166)
  at org.ofbiz.core.entity.GenericDelegator.findByAnd(Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericDelegator.java:909)
  at org.ofbiz.core.entity.GenericDelegator.findByAnd(Ljava/lang/String;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericDelegator.java:887)
  at org.ofbiz.core.entity.GenericDelegator.getRelated(Ljava/lang/String;Ljava/util/Map;Ljava/util/List;Lorg/ofbiz/core/entity/GenericValue;)Ljava/util/List; (GenericDelegator.java:1505)
  at org.ofbiz.core.entity.GenericDelegator.getRelatedByAnd(Ljava/lang/String;Ljava/util/Map;Lorg/ofbiz/core/entity/GenericValue;)Ljava/util/List; (GenericDelegator.java:1452)
  at org.ofbiz.core.entity.GenericValue.getRelatedByAnd(Ljava/lang/String;Ljava/util/Map;)Ljava/util/List; (GenericValue.java:317)
  at com.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssue(Ljava/lang/String;Lorg/ofbiz/core/entity/GenericValue;)Ljava/util/List; (DefaultIssueManager.java:395)
  at com.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssueObject(Ljava/lang/String;Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (DefaultIssueManager.java:410)
  at com.atlassian.jira.issue.managers.RequestCachingIssueManager.getEntitiesByIssueObject(Ljava/lang/String;Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (RequestCachingIssueManager.java:155)
  at com.atlassian.jira.issue.comments.CommentSearchManager.getComments(Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (CommentSearchManager.java:106)
  at com.atlassian.jira.issue.comments.DefaultCommentManager.getComments(Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (DefaultCommentManager.java:174)
  at com.atlassian.jira.issue.index.DefaultCommentRetriever.apply(Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (DefaultCommentRetriever.java:38)
  at com.atlassian.jira.issue.index.DefaultCommentRetriever.apply(Ljava/lang/Object;)Ljava/lang/Object; (DefaultCommentRetriever.java:29)
  at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(Lcom/atlassian/jira/issue/Issue;)Ljava/util/Collection; (DefaultIssueIndexer.java:573)
  at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (DefaultIssueIndexer.java:570)
  at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(Lcom/atlassian/jira/issue/Issue;Lcom/atlassian/jira/issue/index/IssueIndexingParams;)Lcom/atlassian/jira/issue/index/DefaultIssueIndexer$Documents; (DefaultIssueIndexer.java:562)
  at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(Lcom/atlassian/jira/issue/Issue;Lcom/atlassian/jira/task/context/Context$Task;)Lcom/atlassian/jira/index/Index$Result; (DefaultIssueIndexer.java:383)
  at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(Ljava/lang/Object;Lcom/atlassian/jira/task/context/Context$Task;)Lcom/atlassian/jira/index/Index$Result; (DefaultIssueIndexer.java:372)
  at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$null$2(Lcom/atlassian/jira/issue/index/DefaultIssueIndexer$IndexOperation;Lcom/atlassian/jira/issue/Issue;Lcom/atlassian/jira/task/context/Context$Task;)Lcom/atlassian/jira/index/Index$Result; (DefaultIssueIndexer.java:311)
  at com.atlassian.jira.issue.index.DefaultIssueIndexer$$Lambda$2337.get()Ljava/lang/Object; (Unknown Source)
  at com.atlassian.jira.index.SimpleIndexingStrategy.get(Lcom/atlassian/jira/util/Supplier;)Lcom/atlassian/jira/index/Index$Result; (SimpleIndexingStrategy.java:7)
  at com.atlassian.jira.index.SimpleIndexingStrategy.get(Ljava/lang/Object;)Ljava/lang/Object; (SimpleIndexingStrategy.java:5)
  at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call()Lcom/atlassian/jira/index/Index$Result; (MultiThreadedIndexingStrategy.java:33)
  at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call()Ljava/lang/Object; (MultiThreadedIndexingStrategy.java:31)
  at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call()Ljava/lang/Object; (BoundedExecutor.java:68)
  at java.util.concurrent.FutureTask.run()V (FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:624)
  at java.lang.Thread.run()V (Thread.java:748)

We can see on stack trace called the com.atlassian.jira.issue.comments.DefaultCommentManager.getComments method. This suggests that while building this index record, Jira is obtaining the comments on issues individually from the database and it eventually causes the OutOfMemoryError.

Contextually, JIRA's comments are stored in the jiraaction database table, where the field that stores the comment in MySQL is of type longtext. As clarified in MySQL's documentation for 11.1.3 String Type Overview, a LONGTEXT can store up to a 4GB value.

To summarize, we need to check the database in order to understand the data size there. To diagnose this, the following SQL query can be run:

select concat(p.pkey,'-',ji.issuenum) as IssueID, p.pname as ProjectName, SUM(length(ja.actionbody)) as TotalLengthBytes from project p, jiraissue ji, jiraaction ja  where p.ID = ji.PROJECT and ji.ID = ja.issueid  group by IssueID order by TotalLengthBytes desc limit 30;

This query might take some time to run, due to its complexity.

The purpose of this SQL query is to show the length of the comments in a Jira instance's database, sorted in descending in order to show the largest comments on top. It'll show just the top 30 in length, and that value can be adjusted if needed.

Cause

Huge comments objects are stored on the database and the indexing operation fails to allocate these, leading to the OutOfMemoryError seen. The output of the SQL query above may indicate the size (in bytes) of the objects are stored for single issues. The example below illustrate that the greatest object has a size greater than 3GB.

+------------------+------------------+------------------+
| IssueID          | ProjectName      | TotalLengthBytes |
+------------------+------------------+------------------+
| EX-1 | Example Project |       3261804965 |
| EX-2 | Example Project |       1440212209 |
| TT-3 | Test Project    |       1239449727 |
+------------------+------------------+------------------+

Workaround

This is recommended to attempt deleting the issues listed on the SQL output that have 1GB or more of total size.

Due to the expensive delete operation, the UI may be unresponsive and a re-index is required on such scenarios, to avoid data inconsistencies.

Another alternative is to use the equivalent REST API call to perform this operation.



DescriptionOOM Error when performing a re-index on an instance connected to MySQL
ProductJIRA Server
Last modified on Mar 6, 2019

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.