500 Error Bitbucket Server and MySQL database
Symptoms
One of the following log line appears in the atlassian-bitbucket.log:
- Table 'table name' doesn't exist
2013-09-23 14:02:06,776 ERROR [http-bio-7990-exec-210] 842x6582x1 1q2z3cg 10.150.55.72,0:0:0:0:0:0:0:1 "GET /mvc/error500 HTTP/1.1" c.a.s.i.web.ErrorPageController There was an unhandled exception loading [/plugins/servlet/branch-permissions/~SUMALI.CARTER/hnau-playpen-test1]
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
Database:
- name:MySQL
- version:5.5.33
- minor version:5
- major version:5
Driver:
- name:MySQL Connector Java
- version:mysql-connector-java-5.1.25 ( Revision: ${bzr.revision-id} )
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'bitbucket_db.AO_6978BB_RESTRICTED_REF' doesn't exist
....
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'bitbucket_db.AO_6978BB_RESTRICTED_REF' doesn't exist
....2. Could not read fields for table 'table name'
2018-09-25 19:04:46,212 ERROR [spring-startup] c.a.s.s.a.l.PatchedLifecycleManager LifecycleAware.onStart() failed for component with class 'com.atlassian.bitbucket.internal.importer.repository.DefaultRepositoryImportService' from plugin 'com.atlassian.bitbucket.server.bitbucket-importer'
com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.bitbucket.server.bitbucket-importer]
...
Caused by: java.lang.RuntimeException: Could not read fields for table AO_D6A508_IMPORT_JOB
...
2018-09-25 19:04:46,804 ERROR [spring-startup] c.a.s.s.a.l.PatchedLifecycleManager LifecycleAware.onStart() failed for component with class 'com.atlassian.bitbucket.internal.mirroring.upstream.DefaultMirrorService' from plugin 'com.atlassian.bitbucket.server.bitbucket-mirroring-upstream'
...
Caused by: java.lang.RuntimeException: Could not read fields for table AO_8E6075_MIRRORING_REQUEST
at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:139)
at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126)Diagnosis
Stacktrace #1 - Table doesn't exist
Check if the table 'AO_6978BB_RESTRICTED_REF' exists in the database with the following command:
show tables;Stacktrace #2 - Could not read fields for table 'table name'
Confirm the MySQL JDBC driver being used.
Cause
Stacktrace #1 - Table doesn't exist
The MySQL database has case sensitive on table name comparison. The table exists in small letters while Bitbucket Server is looking for the table in capital letters.
Stacktrace #2 - Could not read fields for table 'table name'
The error is caused by the driver version according to described in the bug mentioned here: Unable to locate Jira server for this macro. It may be due to Application Link configuration.
NOTE: If the server runs on Fedora and it has been upgraded from 27 to 28, the mysql-connector-java driver might have been upgraded automatically from 5.1.x to 8.0.x.
When switching from Fedora 27 to Fedora 28, the mysql-connector-java gets upgraded from 5.1.38 to an 8.0.13 or above. Hence, the problems listed in this article will show up.
Resolution
Stacktrace #1 - Table doesn't exist
Add 'lower_case_table_names=1' into MySQL my.cnf file and restart MySQL and Bitbucket Server instance.
Stacktrace #2 - Could not read fields for table 'table name'
If you are indeed using the 8.0.11 connector follow the workaround mentioned in the bug report which entails downgrading the JDBC connector version to the previous 5.1.46 connector/j.
If using Fedora 28, you can use the yum/dnf downgrade command to reactivate the 5.1.x version.