File conversion service causes out of memory errors in Confluence 5.7 or above

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

Confluence experiences out of memory errors when complex Office and PDF files are uploaded. 

One or more of the following errors appear in the Confluence logs at <Confluence-home>/logs/atlassian-confluence.log 

ERROR [conversion-thread-0] [plugins.conversion.impl.TimeoutConversionRunnable] run Cannot convert. Execution error: java.lang.OutOfMemoryError: Java heap space
2015-01-15 11:48:34,898 ERROR [conversion-thread-0] [plugins.conversion.impl.TimeoutConversionRunnable] run Cannot convert. Execution error: java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: GC overhead limit exceeded

Cause

The new preview experience for files, introduced in Confluence 5.7, relies on a file conversion service that runs in the JVM with Confluence. The file conversion service has been known to cause out of memory errors when very complex Office and PDF files are uploaded and converted.  This issue is being tracked in  CONF-38233 - Getting issue details... STATUS .

Diagnosis

To help diagnose this problem, you can turn on additional logging for the file conversion service. This will enable you to see how many conversions are happening, when they start and finish, as well as thread and heap information. 

To enable additional logging:

  1. Go to  > General Configuration > Logging and Profiling
  2. Add a new entry for the package com.atlassian.confluence.plugins.conversion.impl.runnable and set the level to DEBUG

The logs will now show information about each conversion, like this:

2015-01-19 17:45:55,516 DEBUG [conversion-thread-6-internal] [conversion.impl.runnable.ConversionRunnable] debug Starting conversion (THUMBNAIL)
2015-01-19 17:45:55,521 DEBUG [conversion-thread-7-internal] [conversion.impl.runnable.ConversionRunnable] debug Starting conversion (THUMBNAIL)
2015-01-19 17:45:59,047 DEBUG [conversion-thread-7-internal] [conversion.impl.runnable.ConversionRunnable] debug Finished conversion
2015-01-19 17:46:00,174 DEBUG [conversion-thread-6-internal] [conversion.impl.runnable.ConversionRunnable] debug Finished conversion
2015-01-19 17:46:03,862 DEBUG [conversion-thread-8-internal] [conversion.impl.runnable.ConversionRunnable] debug Starting conversion (DOCUMENT)
2015-01-19 17:46:34,840 DEBUG [conversion-thread-8
-internal] [conversion.impl.runnable.ConversionRunnable] debug Finished conversion

and also the number of threads the file conversion is using and the available heap space, like this:

2015-01-19 17:57:15,004 DEBUG [scheduler_Worker-6] [conversion.impl.runnable.ConversionQueueMonitor] doExecute Adjusted in-JVM file conversion thread pool to 1 threads (Free heap: 98MB/910MB, Cores: 8)
2015-01-19 17:57:30,003 DEBUG [scheduler_Worker-7] [conversion.impl.runnable.ConversionQueueMonitor] doExecute Adjusted in-JVM file conversion thread pool to 6 threads (Free heap: 423MB/910MB, Cores: 8)

Sometimes it's a specific file that causes the problem. If you can identify the file, either remove the attachment, or zip the file and re-upload the zip. 

You should also check the attachment size limit for your instance.  If it's very large (for example, 100MB), you can either reduce it – meaning your users won't be able to upload such large files – or adjust your memory settings to suit. 

Resolutions

There are several things you can do to resolve this issue:

We recommend increasing the memory available to the JVM and adjusting the number of threads first. In most cases this will resolve the issue.

Increase PermGen (Java 7)

If you're seeing the following error, you should increase your PermGen. The recommended PermGen for Confluence 5.7 is 384mb. Confluence 5.8 and above uses Java 8 which manages PermGen automatically.

2015-01-15 11:48:34,898 ERROR [conversion-thread-0] [plugins.conversion.impl.TimeoutConversionRunnable] run Cannot convert. Execution error: java.lang.OutOfMemoryError: PermGen space

Go to <installation-directory>/bin/setenv.bat or setenv.sh and change the value of -XX:MaxPermSize parameter from 256m to 384m, or, if you're running Confluence as a Windows service, you can follow these steps to update the setting.

If you're using Java 8, please ignore this section as PermGen is no longer supported. More info here.


Increase Heap

If you're seeing the following error, you should increase the size of the heap.

ERROR [conversion-thread-0] [plugins.conversion.impl.TimeoutConversionRunnable] run Cannot convert. Execution error: java.lang.OutOfMemoryError: Java heap space

See How to fix out of memory errors by increasing available memory for details on how to do this for your operating system.

We recommend starting with the following values and increasing them in small increments.

-Xms1024m -Xmx1024m 


Change the number of threads

The number of threads allocated to the file conversion service is calculated dynamically based on the amount of memory assigned to the instance and the number of CPU cores. It generally sits around 4 to 6 threads.

Decreasing the number of threads means that more memory is available to each thread, but documents will sit in the conversion queue a little longer.

Add the following argument in the JVM options to adjust number of threads. In this example we'll reduce the number of threads to 3.

-Dconfluence.document.conversion.threads=3

 Lowering the thread count to 1 will mean that files will sit in the queue for longer, but the maximum available memory will be used to process them.

Disable the file conversions service

If you still experience out of memory errors after increasing your heap memory and adjusting the threads, you can choose to disable the file conversion service. This means that only image and PDF files will appear in the preview. Users won't be able to preview or comment on Office files.

To disable the file conversion service:

  1. Go to > Add-ons
  2. Locate the Confluence Document Conversion Library system add-on
  3. Disable the File Conversions (file-conversions) module

Once disabled, Office files won't appear in the preview (even if they were previously uploaded), and thumbnails for existing Office and PDF files will revert back to being placeholder thumbnails (with an icon, rather than a preview of the document). You also won't be able to view any existing inline comments on the file (as the preview is disabled).

Data Center: reroute file conversion traffic to designated node

If you are on Confluence Data Center, you can take advantage by routing file conversion traffic to a designated node (ideally a dark node). This way, we can isolate the outage to a specific node only. 

In Apache, you can add the following line on the Load balancer configuration

# route document conversion request to a node (change the ip-address to designated node's address)
ProxyPass /rest/documentConversion/ http://192.168.10.148:18090/rest/documentConversion/   
ProxyPassReverse /rest/documentConversion/ http://192.168.10.148:18090/rest/documentConversion/   
 
# route document preview request to a node (change the ip-address to designated node's address)
ProxyPass /pages/worddav/preview.action http://192.168.10.148:18090/pages/worddav/preview.action   
ProxyPassReverse /pages/worddav/preview.action http://192.168.10.148:18090/pages/worddav/preview.action )


Effect on end users. In case of OOME, users will see a broken thumbnail for Office document and 503 error in the browser console until conversion node is restarted. Other operations are not affected. 

Last modified on Feb 20, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.