Bitbucket Server is returning 503 Service Unavailable error after AWS RDS MySQL Server upgraded from 5.6.51 to 5.7.38
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
After AWS RDS MySQL Server upgraded from 5.6.51 to 5.7.38, Bitbucket is not coming up and Browser is showing a 503 Service Unavailable error.
Environment
Bitbucket - 6.10.17
MySQL - 5.7.38
Diagnosis
The following error messages appears in the atlassian-bitbucket.log.
2022-10-18 04:36:41,253 ERROR [spring-startup] com.zaxxer.hikari.pool.HikariPool bitbucket - Exception during pool initialization.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2115)
Caused by: java.sql.SQLException: Unknown system variable 'storage_engine'
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2497)
at com.mysql.jdbc.StatementImpl.executeUpdateInternal(StatementImpl.java:1540)
Cause
MySQL 5.7.x uses InnoDB by default, and has removed the storage_engine system variable. Having this variable in the jdbc connection string for MYSQL database in bitbucket.properties file under <Bitbucket-Home>/shared directory causes Bitbucket to fail while starting.
Solution
To overcome this issue, below steps can be followed:
1. Stop the Bitbucket.
2. Edit the bitbucket.properties file.
3. Replace "storage_engine" string with "default_storage_engine" in the jdbc connection string for database available under bitbucket.properties file.
4. Bring up the Bitbucket.
Note: Before following above steps, please take the Backup of an existing bitbucket.properties file.