java.lang.OutOfMemoryError: Java heap space encountered while upgrading a Bitbucket mirror to version 7.1 or later
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Upgrading a Bitbucket mirror to version 7.1 or later fails due to a java.lang.OutOfMemoryError: Java heap space error.
Environment
This affects Bitbucket mirror versions < 7.1 that are being upgraded to version >= 7.1.
Diagnosis
The logs show a java.lang.OutOfMemoryError: Java heap space error while executing the SQL statement: UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0.
2021-06-13 09:31:18,602 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set liquibase/r7_1/upgrade.xml::BBSDEV-21559-2::acarlton:
Reason: liquibase.exception.DatabaseException: Out of memory.; SQL statement:
UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0 [90108-176] [Failed SQL: UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:723)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.lang.Thread.run(Thread.java:748)
... 18 frames trimmed
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set liquibase/r7_1/upgrade.xml::BBSDEV-21559-2::acarlton:
Reason: liquibase.exception.DatabaseException: Out of memory.; SQL statement:
UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0 [90108-176] [Failed SQL: UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
... 5 common frames omitted
Caused by: liquibase.exception.MigrationFailedException: Migration failed for change set liquibase/r7_1/upgrade.xml::BBSDEV-21559-2::acarlton:
Reason: liquibase.exception.DatabaseException: Out of memory.; SQL statement:
UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0 [90108-176] [Failed SQL: UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0]
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:625)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:78)
at liquibase.Liquibase.update(Liquibase.java:202)
at liquibase.Liquibase.update(Liquibase.java:179)
at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:317)
at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:39)
... 5 common frames omitted
Suppressed: org.h2.jdbc.JdbcSQLException: The database has been closed [90098-176]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:178)
at org.h2.message.DbException.get(DbException.java:154)
at org.h2.message.DbException.get(DbException.java:143)
at org.h2.engine.Database.checkPowerOff(Database.java:490)
at org.h2.command.Command.executeUpdate(Command.java:274)
at org.h2.jdbc.JdbcConnection.rollbackInternal(JdbcConnection.java:1500)
at org.h2.jdbc.JdbcConnection.rollback(JdbcConnection.java:488)
at com.zaxxer.hikari.pool.ProxyConnection.close(ProxyConnection.java:257)
at com.atlassian.stash.internal.jdbc.ProxyingConnectionTracker$ConnectionInvocationHandler.invoke(ProxyingConnectionTracker.java:104)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:385)
at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:40)
... 5 common frames omitted
Caused by: liquibase.exception.DatabaseException: Out of memory.; SQL statement:
UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0 [90108-176] [Failed SQL: UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0]
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:356)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:57)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:125)
at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1229)
at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1211)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:600)
... 11 common frames omitted
Caused by: org.h2.jdbc.JdbcSQLException: Out of memory.; SQL statement:
UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0 [90108-176]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.convert(DbException.java:288)
at org.h2.command.Command.executeUpdate(Command.java:260)
at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:186)
at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:160)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:95)
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:352)
... 16 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181)
at java.util.ArrayList.grow(ArrayList.java:265)
at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:239)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:231)
at java.util.ArrayList.add(ArrayList.java:462)
at org.h2.engine.UndoLog.add(UndoLog.java:155)
at org.h2.engine.Session.log(Session.java:721)
at org.h2.table.Table.updateRows(Table.java:476)
at org.h2.command.dml.Update.update(Update.java:146)
at org.h2.command.CommandContainer.update(CommandContainer.java:79)
at org.h2.command.Command.executeUpdate(Command.java:254)
at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:186)
at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:160)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:95)
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:352)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:57)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:125)
at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1229)
at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1211)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:600)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:78)
at liquibase.Liquibase.update(Liquibase.java:202)
at liquibase.Liquibase.update(Liquibase.java:179)
at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:317)
at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:39)
... 1 common frames omittedCause
Upgrading to Bitbucket 7.1 or later includes an upgrade task that updates all rows in the sta_repo_push_ref table in the mirror's H2 database.
The upgrade task executes:
UPDATE PUBLIC.sta_repo_push_ref SET ref_update_type = 0If the sta_repo_push_ref table has a lot of rows, the default max JVM heap for mirrors (1GB) may not be enough to handle the update operation.
Solution
Increase the max memory allocation pool for the JVM to a higher value for the upgrade.
For example, set the JVM_MAXIMUM_MEMORY variable (either in the environment or in the <BITBUCKET_INSTALL_DIR>/bin/_start-webapp.sh script) to "2g" or higher, as required.
Afterwards, restart Bitbucket on the mirror.
After the upgrade completes successfully, reset the JVM_MAXIMUM_MEMORY back to the original value.
Last modified on May 6, 2022
Powered by Confluence and Scroll Viewport.