Bitbucket Server is returning 503 Service Unavailable error after AWS RDS MySQL Server upgraded from 5.6.51 to 5.7.38
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.
Summary
After AWS RDS MySQL Server upgraded from 5.6.51 to 5.7.38, Bitbucket is not coming up and Browser is showing a 503 Service Unavailable error.
Environment
Bitbucket - 6.10.17
MySQL - 5.7.38
Diagnosis
The following error messages appears in the atlassian-bitbucket.log.
2022-10-18 04:36:41,253 ERROR [spring-startup] com.zaxxer.hikari.pool.HikariPool bitbucket - Exception during pool initialization.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2115)
Caused by: java.sql.SQLException: Unknown system variable 'storage_engine'
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2497)
at com.mysql.jdbc.StatementImpl.executeUpdateInternal(StatementImpl.java:1540)
Cause
MySQL 5.7.x uses InnoDB by default, and has removed the storage_engine system variable. Having this variable in the jdbc connection string for MYSQL database in bitbucket.properties file under <Bitbucket-Home>/shared directory causes Bitbucket to fail while starting.
Solution
To overcome this issue, below steps can be followed:
1. Stop the Bitbucket.
2. Edit the bitbucket.properties file.
3. Replace "storage_engine" string with "default_storage_engine" in the jdbc connection string for database available under bitbucket.properties file.
4. Bring up the Bitbucket.
Note: Before following above steps, please take the Backup of an existing bitbucket.properties file.