Problem

Bitbucket server upgrade fails with database related errors in log

For example  Error: Table '<table>' already exists

2017-07-21 10:55:01,831 ERROR [spring-startup]  liquibase Change Set liquibase/r3_8/upgrade.xml::STASHDEV-8755-1::mheemskerk failed.  Error: Table 'sta_remember_me_token' already exists [Failed SQL: CREATE TABLE <database>.sta_remember_me_token (id BIGINT NOT NULL, series VARCHAR(64) NOT NULL, token VARCHAR(64) NOT NULL, user_id INT NOT NULL COMMENT 'userId', expiry_timestamp DATETIME NOT NULL, claimed BIT(1) NOT NULL, claimed_address VARCHAR(255) NULL, CONSTRAINT pk_remember_me_id PRIMARY KEY (id))]
2017-07-21 10:55:01,837 WARN  [spring-startup]  o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set liquibase/r3_8/upgrade.xml::STASHDEV-8755-1::mheemskerk:

Diagnosis

Diagnostic Steps

• Verify whether any upgrade task is run against the database. Run following query against the databasechangelog.

  select * from databasechangelog order by dateexecuted desc;

  this table has an entry for the every liquibase upgrade task run. As an example result below

  id	author	filename	dateexecuted	orderexecuted	exectype	md5sum	description	comments	tag	liquibase
  BSERV-8242-1a	bturner	liquibase/r4_0/p07.xml	2016-01-15 15:59:56	447	EXECUTED	3:ce04c4936fba5f094ae753a110f8725f	Update Data	Update rows with the stash-bundled-hooks key to use the new bitbucket-bundled-hooks key, unless a row with that key already exists for the repository.		2.0.5

  From the above, we can gather the following information:

  • From the filename we can see that this was an upgrade task for 4.0
  • The ID gives us the JAC ticket which this change is associated with
  • dateexecuted tells us when this upgrade was performed

• Check .version file under Under your $BITBUCKET_HOME. This file contains the Bitbucket Server home directory version. It should match the database version.

Cause

• An upgrade caused the database to be in inconsistent state. 
• Two instances of Bitbucket server connected to the same database and one of them has been upgraded.

Resolution

• Restore from a healthy backup which was taken before the upgrade task. For more details about upgrade and recovery, have a look at Data recovery and backups
• Migrate to another instance of the database: This will create a new schema matching the version and the data will be migrated over to the new database instance. 
  For more details on migrating to another database instance, have a look at Using Database Migration Wizard in Connecting Bitbucket Server to an external database

• Drop the tables which shows as already exists in the error message. Ensure there is no content in the tables . You may have to run this against multiple tables.

  drop table sta_remember_me_token;
  drop table sta_repo_settings;

  Always back up your data before performing any modifications to the database. If possible, test any alter, insert, update, or delete SQL commands on a staging server first.