Backup client restore fails due to Out of Memory or GC overhead limit exceeded

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Symptoms

The backup client fails during the restore due to the default Java heap space not being enough for the operation to be performed.

One of the logs below can be shown in the atlassian-bitbucket-YYYY-MM-DD.log:

2015-03-09 15:50:58,740 INFO [main] c.a.s.i.b.c.DefaultRestoreClient Processed 726000 changes of 1293334
2015-03-09 15:53:26,155 INFO [main] c.a.s.i.b.c.DefaultRestoreClient Processed 727000 changes of 1293334
2015-03-09 15:53:54,477 ERROR [main] c.a.s.i.backup.client.RestoreMain test-bitbucket-20150304-021150-568.tar could not be restored. Reason: Failed to execute change: Insert Row; nested exception is java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
com.atlassian.bitbucket.internal.backup.liquibase.LiquibaseChangeExecutionException: Failed to execute change: Insert Row; nested exception is java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseDao.insert(DefaultLiquibaseDao.java:274) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DatabaseUpdater.endElement(DatabaseUpdater.java:81) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) ~[na:1.7.0_71]
at com.atlassian.security.xml.RestrictedXMLReader.parse(RestrictedXMLReader.java:103) ~[atlassian-secure-xml-3.2.4.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseMigrationDao.parse(DefaultLiquibaseMigrationDao.java:206) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseDao.withLock(DefaultLiquibaseDao.java:301) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseMigrationDao.restore(DefaultLiquibaseMigrationDao.java:168) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restoreFile(DefaultRestoreClient.groovy:309) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.withDataSource(DefaultRestoreClient.groovy:266) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restoreDatabase(DefaultRestoreClient.groovy:218) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restoreSystem(DefaultRestoreClient.groovy:252) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restore(DefaultRestoreClient.groovy:83) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.spring.SpringMain.run(SpringMain.groovy:28) [bitbucket-backup-core-1.6.0.jar:na]
at com.atlassian.bitbucket.internal.backup.client.RestoreMain.main(RestoreMain.groovy:55) [bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.JavaVersionCheckingShim.invokeClientClassMain(JavaVersionCheckingShim.java:71) [bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.JavaVersionCheckingShim.run(JavaVersionCheckingShim.java:85) [bitbucket-restore-client.jar:1.6.0]
... 100 frames suppressed

OR

2015-03-10 14:01:29,052 INFO         Processed 806000 changes of 1293334
2015-03-10 14:04:55,066 INFO         Processed 807000 changes of 1293334
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.lang.reflect.Method.copy(Method.java:151)
        at java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136)
        at sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
        at java.lang.Class.copyMethods(Class.java:2934)
        at java.lang.Class.getDeclaredMethods(Class.java:1860)
        at org.codehaus.groovy.reflection.CachedClass$3$1.run(CachedClass.java:84)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:81)
        at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:79)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:46)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:37)
        at org.codehaus.groovy.reflection.CachedClass.getMethods(CachedClass.java:250)
        at org.codehaus.groovy.reflection.CachedMethod.find(CachedMethod.java:58)
        at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2474)
        at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
        at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
        at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:251)
        at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:259)
        at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:855)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.createPojoSite(CallSiteArray.java:122)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:163)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at com.atlassian.bitbucket.internal.backup.client.liquibase.LoggingLiquibaseRestoreMonitor.onAppliedChange(LoggingLiquibaseRestoreMonitor.groovy:26)
        at com.atlassian.bitbucket.internal.backup.liquibase.DatabaseUpdater.endElement(DatabaseUpdater.java:83)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2973)

Cause

The default MaxHeapSize defined for the Java in your system is not enough for performing the restore. On the output below, for example, this value is 1GB:

$ java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
    uintx AdaptivePermSizeWeight                    = 20              {product}
     intx CompilerThreadStackSize                   = 0               {pd product}
    uintx ErgoHeapSizeLimit                         = 0               {product}
    uintx HeapSizePerGCThread                       = 87241520        {product}
    uintx InitialHeapSize                          := 62733632        {product}
    uintx LargePageHeapSizeThreshold                = 134217728       {product}
    uintx MaxHeapSize                              := 1004535808      {product}
    uintx MaxPermSize                               = 174063616       {pd product}
    uintx PermSize                                  = 21757952        {pd product}
     intx ThreadStackSize                           = 1024            {pd product}
     intx VMThreadStackSize                         = 1024            {pd product}
java version "1.7.0_71"
OpenJDK Runtime Environment (rhel-2.5.3.1.el6-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
 


Resolution

While executing the restore, add a parameter to the restore command, increasing the heap size for that execution by passing a higher maxHeapSize parameter. For this specific example, we raised it to -Xmx2g.

According to the link above, that's how you'd normally do:

$ java -Djdbc.override=true -Djdbc.driver=org.postgresql.Driver -Djdbc.url=jdbc:postgresql://HOSTNAME:PORT/DATABASE -Djdbc.user=bitbucketuser -Djdbc.password=password -Dbitbucket.home="path/to/bitbucket/home" -jar /path/to/bitbucket-restore-client.jar /path/to/original/backup/file

Change the command below into:

$ java -Xmx2g -Djdbc.override=true -Djdbc.driver=org.postgresql.Driver -Djdbc.url=jdbc:postgresql://HOSTNAME:PORT/DATABASE -Djdbc.user=bitbucketuser -Djdbc.password=password -Dbitbucket.home="path/to/bitbucket/home" -jar /path/to/bitbucket-restore-client.jar /path/to/original/backup/file
Last modified on Apr 6, 2016

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.