Bitbucket fails to start in disaster recovery mode.

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

The following issues are observed in Bitbucket when trying to start in disaster recovery:

  • Bitbucket fails to start
The following appears in the log 
2020-10-27 13:56:55,571 WARN  [spring-startup]  o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'fullIntegrityCheckStartupTask'; nested exception is com.atlassian.bitbucket.DataStoreException: A database error has occurred.
2020-10-27 13:56:55,764 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Releasing lock on /nfs/bitbucket
2020-10-27 13:56:55,774 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Releasing lock on /home/atlassian/bitbucket/application_data
2020-10-27 13:56:55,781 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
org.springframework.context.ApplicationContextException: Failed to start bean 'fullIntegrityCheckStartupTask'; nested exception is com.atlassian.bitbucket.DataStoreException: A database error has occurred.
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:174)
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360)
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158)
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122)
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
	at javax.servlet.GenericServlet.init(GenericServlet.java:158)
	at java.lang.Thread.run(Thread.java:748)
	... 8 frames trimmed
Caused by: com.atlassian.bitbucket.DataStoreException: A database error has occurred.
	at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:47)
	at com.atlassian.stash.internal.integrity.FullIntegrityCheckHelper.scheduleOpenPullRequestChecksByRescope(FullIntegrityCheckHelper.java:58)
	at com.atlassian.stash.internal.integrity.FullIntegrityCheckHelper$$FastClassBySpringCGLIB$$aec74624.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at com.atlassian.stash.internal.integrity.FullIntegrityCheckHelper$$EnhancerBySpringCGLIB$$b59d03bf.scheduleOpenPullRequestChecksByRescope(<generated>)
	at com.atlassian.stash.internal.integrity.FullIntegrityCheckStartupTask.lambda$start$0(FullIntegrityCheckStartupTask.java:81)
	at com.atlassian.stash.internal.user.DefaultEscalatedSecurityContext.call(DefaultEscalatedSecurityContext.java:59)
	at com.atlassian.stash.internal.integrity.FullIntegrityCheckStartupTask.start(FullIntegrityCheckStartupTask.java:78)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182)
	... 10 common frames omitted
Caused by: org.hibernate.NonUniqueResultException: query did not return a unique result: 2
	at org.hibernate.query.internal.AbstractProducedQuery.uniqueElement(AbstractProducedQuery.java:1597)
	at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:392)
	at com.atlassian.stash.internal.crowd.HibernateUserDao.internalFindByName(HibernateUserDao.java:452)
	at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:131)
	at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:72)
	at com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(AbstractInternalDirectory.java:167)
	at com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(AbstractInternalDirectory.java:64)
	at com.atlassian.crowd.directory.AbstractForwardingDirectory.findUserByName(AbstractForwardingDirectory.java:74)
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.findUserByName(DirectoryManagerGeneric.java:315)
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.findUserByName(ApplicationServiceGeneric.java:307)
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.getUser(CrowdServiceImpl.java:86)
	at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.getCrowdUser(CrowdAwareLoadListener.java:130)
	at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.setCrowdUserIfNecessary(CrowdAwareLoadListener.java:149)
	at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.onPostLoad(CrowdAwareLoadListener.java:87)
	at org.hibernate.engine.internal.TwoPhaseLoad.postLoad(TwoPhaseLoad.java:517)
	at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.postLoad(AbstractRowReader.java:309)

Cause

This error is caused by a duplicate value in the database. This might be due to:

  • Some corruption in the directory cache.
  • Corruption in the database.

Diagnosis - Find duplicated users in the cwd_user table

Diagnosis Query - 1: Find duplicate users in the cwd_user table that has the same External ID and belong to the same directory:

SELECT directory_id, external_id
              FROM cwd_user
              GROUP BY directory_id, external_id
              HAVING ( COUNT(external_id) > 1)

Diagnosis Query - 2: Find duplicate users in the cwd_user table that has the same lower_user_name and belong to the same directory:

SELECT lower_user_name, directory_id
		from cwd_user 
		GROUP BY lower_user_name, directory_id 
		HAVING (COUNT(lower_user_name) > 1);

Resolution  - Deleting the user from the database

Always backup Bitbucket before removing or modifying any data on the database and ensure Bitbucket is shut down.

Step 1: Find the duplicated user IDs

Run the query below to get the users IDs of the duplicated user(s):

SELECT id, lower_user_name FROM cwd_user WHERE external_id = '<duplicated external_id from diagnostic query>'

or

SELECT id, lower_user_name FROM cwd_user WHERE lower_user_name = '<duplicated username from diagnostic query>'

The result should be as follow:

iduser_name

229377

admin
229378admin

Note the IDs provided in the query above. Determine which entry to delete in the next step (Step 2).

Step 2: Determine which IDs to delete by checking if any of the duplicated users currently belong to any group

Run the following SQL query against the duplicated IDs result. The SQL query will yield the number of groups that each duplicated_user ID is tied to:

SELECT count(*), child_user_id FROM cwd_membership WHERE child_user_id in (<duplicated_id_1>,<duplicated_id_2>) GROUP BY child_user_id

  

Step 3.1 If one of the duplicated ID does not return any group membership result

If the SQL query above return with something as follows,

count(*)child_user_id
2229377

Proceed with safely deleting the other duplicated user id (not the one shown from the SQL query):

DELETE FROM cwd_user_attribute WHERE user_id = <duplicated_id>; 
DELETE FROM cwd_user_credential_record WHERE user_id = <duplicated_id>;
DELETE FROM cwd_user WHERE id = <duplicated_id>;

     

Step 3.2: If both of the duplicated ID returns a number of group memberships

If the SQL query in step 2 return with something as follows:

count(*)child_user_id
2229377
2229378


Run the following SQL query to obtain the duplicated users group membership:

SELECT
u.user_name,
g.id as GroupID,
g.group_name
FROM
cwd_user u
JOIN cwd_membership m
ON
u.id = m.child_user_id
JOIN cwd_group g
ON
g.id = parent_id
WHERE u.user_name = '<duplicated username>'
ORDER BY 2 DESC, 1 ASC;

Take note of the result of the above SQL query. Proceed with safely deleting one of the duplicated user ids. Run the below queries in order:

DELETE FROM cwd_membership WHERE child_user_id = <duplicated_id>;
DELETE FROM cwd_user_attribute WHERE user_id = <duplicated_id>; 
DELETE FROM cwd_user_credential_record WHERE user_id = <duplicated_id>;
DELETE FROM cwd_user WHERE id = <duplicated_id>;

    

Once you have completed the above steps, attempt to start Bitbucket to see if the issue has been resolved. 

If you continue to experience issues, please contact support.atlassian.com/contact for further help.

Last modified on Dec 31, 2020

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.