High CPU Usage caused by Ancestors Check in Confluence

This Knowledge Base article was written specifically for the Atlassian Server platform. Due to the Restricted functions in Atlassian Cloud apps, the contents of this article cannot be applied to Atlassian Cloud applications.

Problem

Corrupted entries in the application Ancestors Table (responsible for maintaining the relationship in between parent and descendant pages on Confluence) can cause View Page threads to run for a long period, triggering symptoms like High CPU Usage and intense GC Activity, which can eventually lead the application to an outage. Thread Dumps will usually show that the offending threads are executing the getInheritedContentPermissionSets (accountable for checking permissions) method being executed:

"http-nio-8090-exec-20 uri:/pages/viewpage.action username:" #274 daemon prio=5 os_prio=0 tid=0x00007f6484011000 nid=0x2e8b runnable [0x00007f61c350f000]
   java.lang.Thread.State: RUNNABLE
	at net.sf.hibernate.impl.AbstractVisitor.processValue(AbstractVisitor.java:67)
	at net.sf.hibernate.impl.AbstractVisitor.processValues(AbstractVisitor.java:36)
	at net.sf.hibernate.impl.SessionImpl.isUpdateNecessary(SessionImpl.java:2666)
	at net.sf.hibernate.impl.SessionImpl.flushEntity(SessionImpl.java:2581)
	at net.sf.hibernate.impl.SessionImpl.flushEntities(SessionImpl.java:2513)
	at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2308)
	at net.sf.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1831)
	at net.sf.hibernate.impl.SessionImpl.getQueries(SessionImpl.java:1594)
	at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1559)
	at net.sf.hibernate.impl.QueryImpl.list(QueryImpl.java:49)
	at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao.lambda$getInheritedContentPermissionSets$0(HibernateContentPermissionSetDao.java:38)
	at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao$$Lambda$766/775792763.doInHibernate(Unknown Source)
	at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
	at org.springframework.orm.hibernate.HibernateTemplate.executeFind(HibernateTemplate.java:341)
	at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao.getInheritedContentPermissionSets(HibernateContentPermissionSetDao.java:25)

Diagnosis

This problem is triggered regardless of environmental variables, but its symptoms are usually the following:

  • High CPU Usage.
  • Garbage Collection Threads being heavily used.

Also, it is important to diagnose if there are corrupted entries in the Ancestors Table or not, since those are known to cause this issue.

Is my Ancestors Table corrupted?

In order to diagnose if there are indeed corrupted entries in the Ancestors Table, an administrator may run the query below at Confluence's database:

Is my Ancestors Table corrupted?
SELECT A.ANCESTORID, A.DESCENDENTID, COUNT(A.DESCENDENTID)
FROM CONFANCESTORS A, CONTENT B
WHERE B.CONTENTID = A.DESCENDENTID
GROUP BY A.ANCESTORID, A.DESCENDENTID HAVING COUNT(A.DESCENDENTID) != 1
ORDER BY 3 DESC

In case it has returned any results, then it is corrupted indeed.

Cause

This is caused by corrupted entries in the application Ancestors Table - Responsible for maintaining the relationship in between parent and descendant pages on Confluence.

Workaround

The workaround consists of re-constructing the application Ancestors Table as described in Rebuilding the Ancestor Table documentation. Steps to do so have been listed below as well:

------------------------------------------------------------------------------------------------------------------

The ancestor table records the parent and descendant (child) relationship between pages. It is also used when determining whether a page will inherit view restrictions from a parent page. 

Occasionally records in the ancestor table can become corrupted. When this happens you will need to rebuild the ancestor table.

To rebuild the ancestor table:

  1. Back up your database.
  2. Use the following URL to access the Page Level Permissions page 
    <your-site>/admin/permissions/pagepermsadmin.action
  3. Choose Rebuild ancestor table

  4. Go to  > General Configuration > Cache Management.
  5. Flush the Inherited Content Permissions cache. 
You may also need to  rebuild the content index so that the permissions take effect in search results.

This process takes about ten minutes for an average size site, and you may not be able to save new content during this time. We recommend you take a full backup of your database and complete this process during a maintenance window.

------------------------------------------------------------------------------------------------------------------

Resolution

There have been some suggestions to properly log and alert administrators when there are problems with the Ancestors Table, being the main one implemented already:

With proper logging, administrators can proactively rebuild the table upon need, however, this doesn't fix the cause behind why the table has been corrupted in first place and this is still pending a fix. Actions can be tracked using below bugs:


Last modified on May 9, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.