Summary

When trying to Export an XML Site Backup, no backup file is created, despite no error is shown in Confluence UI.

Looking through the logs, there are no exceptions related to the backup action.

Environment

Confluence Server or Data Center

Diagnosis

1. Following the Troubleshooting failed XML site backups, add the following content to <confluence_install/confluence/WEB-INF/classes/log4j.properties, aiming to increase the status information logged by these classes:

   log4j.logger.com.atlassian.hibernate.extras.XMLDatabinder=DEBUG, confluencelog
   log4j.additivity.com.atlassian.hibernate.extras.XMLDatabinder=false

2. When performing the backup, we can see the following messages:

   atlassian-confluence.log

   2021-01-14 09:35:57,792 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Backward compatibility for export: Checking...
   2021-01-14 09:35:57,794 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8401: "Upgrade to build number: 8401" - Ok
   2021-01-14 09:35:57,795 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8301: "Add unmanaged unique constraints missing in CONFSERVER-38706 and CONFSERVER-58261" - Ok
   2021-01-14 09:35:57,795 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok
   2021-01-14 09:35:57,796 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok
   2021-01-14 09:35:57,796 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok
   2021-01-14 09:35:57,797 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7901: "Upgrade "synchrony.btf.off" property" - Ok
   2021-01-14 09:35:57,797 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7801: "Disables backup job" - Ok
   2021-01-14 09:35:57,798 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7701: "Fixes wrong spaces for drafts" - Ok
   2021-01-14 09:35:57,808 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7601: "Upgrade custom colour scheme to ADGS" - Ok
   2021-01-14 09:35:57,808 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7400: "Upgrade to build number: 7400" - Ok
   2021-01-14 09:35:57,809 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7202: "Remove the legacy Plugin License Storage plugin from the user installed plugins" - Ok
   2021-01-14 09:35:57,809 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7201: "Upgrade to build number: 7201" - Ok
   2021-01-14 09:35:57,810 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7110: "Reindex all blogs." - Ok
   2021-01-14 09:35:57,811 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7109: "Drop and update some low performance indexes on EVENTS table for Synchrony" - Ok
   2021-01-14 09:35:57,811 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7108: "Update index sp_comp_idx when the definition is not correct" - Ok
   2021-01-14 09:35:57,812 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7106: "Drop the hibernate 2 constraints in preparation for them to be re-created in the hibernate 5 naming style" - Ok
   2021-01-14 09:35:57,812 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7105: "Drop or update some low performance indexes on CONTENT" - Ok
   2021-01-14 09:35:57,813 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7104: "Set the mime type for csv files to text/csv" - Ok
   2021-01-14 09:35:57,813 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7101: "Migrate over Documentation theme settings to Custom Page Settings in the default theme" - Ok
   2021-01-14 09:35:57,823 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6439: "Reindexes all PersonalInformation objects, creating them for any users that do not have one." - Ok
   2021-01-14 09:35:57,824 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6430: "Upgrade to build number: 6430" - Ok
   2021-01-14 09:35:57,833 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6424: "Adds missing multi-column unique constraints to the Crowd database tables" - Ok
   2021-01-14 09:35:57,842 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6423: "Cleaning data that violates constraints we want to add in the next migration task" - Ok
   2021-01-14 09:35:57,842 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6422: "Upgrade to build number: 6422" - Ok
   2021-01-14 09:35:57,852 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6412: "Delete all FileStore IDs" - Ok
   2021-01-14 09:35:57,852 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6406: "Reindex all spaces" - Ok
   2021-01-14 09:35:57,853 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6403: "Upgrade to build number: 6403" - Ok
   2021-01-14 09:35:57,853 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6193: "Rebuild the Confluence favourites index" - Ok
   2021-01-14 09:35:57,854 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6193: "Rebuild the Confluence favourites index" - Ok
   2021-01-14 09:35:57,855 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6189: "Upgrade to build number: 6189" - Ok
   2021-01-14 09:35:57,863 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6180: "Adds a multi-column index to content table that speeds up many common queries." - Ok
   2021-01-14 09:35:57,864 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6175: "Upgrade to build number: 6175" - Ok
   2021-01-14 09:35:57,864 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6169: "Upgrade to build number: 6169" - Not Ok
   2021-01-14 09:35:57,865 INFO [http-nio-8090-exec-53] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Backward compatibility for export: "Upgrade to build number: 6169" prevents compatibility before 6169
   

3. But after that, no additional information is logged for that particular thread (in this example http-nio-8090-exec-53)

4. Searching through Tomcat logs for that thread, at the same time period, we'll see that is was stuck when trying to execute a SQL query in the database:

   14-Jan-2021 09:37:01.867 WARNING [Catalina-utility-1] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-53] (id=[377]) has been active for [66,646] milliseconds (since [1/14/21 9:35 AM]) to serve the same request for [https://confluence.empoweredbenefits.com/admin/dobackup.action] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.
       java.lang.Throwable
           at java.base@11.0.7/java.net.SocketInputStream.socketRead0(Native Method)
           at java.base@11.0.7/java.net.SocketInputStream.socketRead(Unknown Source)
   ...
           at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
           at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1912)
           at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2020)
           at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431)
           at com.atlassian.hibernate.extras.XMLDatabinder.renderCollectionType(XMLDatabinder.java:615)
           at com.atlassian.hibernate.extras.XMLDatabinder.renderProperty(XMLDatabinder.java:475)
           at com.atlassian.hibernate.extras.XMLDatabinder.writeObject(XMLDatabinder.java:277)
           at com.atlassian.hibernate.extras.XMLDatabinder.writeObjects(XMLDatabinder.java:234)
           at com.atlassian.hibernate.extras.XMLDatabinder.toGenericXML(XMLDatabinder.java:212)
           at com.atlassian.confluence.importexport.impl.AtlassianXMLDatabinder.toGenericXML(AtlassianXMLDatabinder.java:50)
           at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.backupEntities(AbstractXmlExporter.java:203)
           at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.backupEverything(AbstractXmlExporter.java:100)
           at com.atlassian.confluence.importexport.impl.FileXmlExporter.backupEverything(FileXmlExporter.java:95)
           at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.doExport(AbstractXmlExporter.java:94)
           at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExportInternal(FileXmlExporter.java:60)
           at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExport(FileXmlExporter.java:54)oFilter(Unknown Source)
   ...
           at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
           at java.base@11.0.7/java.lang.Thread.run(Unknown Source)

Cause

The maxThread parameter in the Tomcat connector was set to a value that is greater than the DB connection pool size.

You can check the maxThreads value in the file <confluence_install>/conf/server.xml (in this example, it was set to 200):

<Connector port="6790" connectionTimeout="20000" redirectPort="8443"
                   maxThreads="200" minSpareThreads="10"
                   enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
                   protocol="org.apache.coyote.http11.Http11NioProtocol"/>

The DB connection pool size can be verified on <confluence_home>/confluence.cfg.xml (in this example, it was set to 60):

<property name="hibernate.c3p0.max_size">60</property>

Solution

1. Stop Confluence
2. Follow the steps documented on the KB Article Startup check: HTTP MaxThreads configuration to either change the size of the database connection pool or change the size of the HTTP thread pool. We need to ensure that the DB connection pool size is greater than the maximum number of HTTP Threads.
3. Start Confluence

After these steps Confluence will not open more threads than the database is able to handle.