Summary

The Confluence Cloud Migration Assistant (CCMA) has resulted in an OOME exception while migrating spaces to Confluence Cloud. 

Environment

Confluence Server and CCMA 3.2 and higher.

Diagnosis

The following exception or similar is seen in atlassian-confluence.log or Tomcat catalina.out logs indicating a problem while running a space migration for space key ALPHA: 

2021-07-15 17:25:27,196 ERROR [pool-49-thread-3] [service.stepexecutor.attachment.AttachmentMigrationExecutor] error Attachment uploader finished with exception
 -- planName: Confluence Cloud Migration | cloudId: cddb44fc-a422-4a22-9a97-8a6bfaedcee7 | planId: 4e4f661a-fde4-46d4-9af9-9608c61456db | cloudUrl: https://acmeinc.atlassian.net | taskName: Migrate space ALPHA | taskId: 311e0b39-c6a3-4f85-9d87-af6f48285d26
java.util.concurrent.CompletionException: net.jodah.failsafe.FailsafeException: java.lang.OutOfMemoryError: unable to create new native thread

Cause

Migration plans including any spaces with many attachments may create thread contention on the source host. To provide concurrency in Confluence Server (meaning the ability to do multiple things at once), Java will spawn operating system threads and use them to perform given tasks. Hard limits can be set at the system level on the number of threads that can be created, so if the application is requesting more threads than the OS is willing to provide, the above error is thrown.

This can also happen if the operating system has no native memory left to allocate threads (say the 32-bit Java process space has been reached, or the OS virtual memory is fully depleted), or the maximum number of open files has been reached.

Solution

Setting the maximum running user processes and number of opened files permanently is recommended, and the implementation of this is operating system specific. Consult with your System Administrator / hosting provider about the most appropriate limits to set. 

1. Edit the $CONFLUENCE_INSTALL/bin/setenv.sh file and add the below to the top of the file:

ulimit -u 4096
ulimit -n 4096

These can be set in Debian / Ubuntu in limits.conf by adding the below. In this example below, confluence indicates the system user that is running Confluence. This value will need to match your system user that runs Confluence in case it is not the same. 

confluence      soft   nofile  4096
confluence      hard   nofile  8192
confluence      soft   nproc   4096        
confluence      hard   nproc   8192

Other Notes

For more information about this OOME exception, consult this documentation: Confluence crashes and throws 'OutOfMemory: unable to create new native thread' error