APPARENT DEADLOCK creating emergency threads for unassigned pending tasks

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

1) Apparent deadlock shows in log
2) ActiveMQ shuts down

INFO | wrapper | 2008/12/04 11:48:38 | Wrapper Process has not received any CPU time for 12 seconds. Extending timeouts.INFO   | wrapper  | 2008/08/18 11:06:48 | Wrapper Process has not received any CPU time for 71 seconds.  Extending timeouts.
...
INFO   | jvm 1    | 2008/08/18 11:06:48 | Wrapper Manager: The Wrapper code did not ping the JVM for 57 seconds.  Quit and let the Wrapper resynch.
ThreadPoolAsynchronousRunner$DeadlockDetector@7a420e -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
WARN [Timer-1] [ThreadPoolAsynchronousRunner] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7a420e -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2008-12-04 11:10:29,948 WARN [Timer-1] [ThreadPoolAsynchronousRunner] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7a420e -- APPARENT DEADLOCK!!! Complete Status:
	Managed Threads: 3
	Active Threads: 3
	Active Tasks:
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d71301 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@120a798 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@db14bd (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
	Pending Tasks:
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@12eba13
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@ba3914
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f95693
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@a2d82a
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@11dc030
Pool thread stack traces:
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
		java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
		java.io.File.isDirectory(Unknown Source)
		net.sourceforge.jtds.jdbc.ConnectionJDBC2.unpackProperties(ConnectionJDBC2.java:1196)
		net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:269)
		net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
		net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:182)

Cause

These deadlock errors often happen in conjunction with high load or out-of-memory conditions. There are no out of memory problems but the server may have experienced a high load around the time the deadlocks were reported. They are not real deadlocks in this case - they are 'apparent' and the MSSQL driver jumps to conclusions here.

SQL Server 2000 will (particularly in low-memory conditions) escalate locks and abort transactions, and expects the application layer to be able to rerun aborted transactions, which our application is not capable of.

SQL Server 2005 implements multi-version concurrency control which means deadlocks are very much less likely to occur. So practically, if you're running SQL Server 2000, the solution is to upgrade to 2005 or switch to a database that supports MVCC (practically anything except Sybase).

There seem to be some hotfixes for MSSQL 2005 available that deal with the deadlock situation.

Taking 3-4 Thread Dumps during the APPARENT DEADLOCK messages, or at least during the period of slowness, will confirm if the lack of available database connections is the real cause of these messages.

Resolution

To workaround this, you can increase the timeout period during which the wrapper will try to ping the CPU before doing a restart. You can do this by changing the wrapper.conf under the /conf directory. In this file, add the following property:

conf/wrapper.conf
wrapper.ping.timeout=60

This doubles the timeout from the default 30 to 60 seconds. Alternatively you can completely disable wrapper's ability to automatically restart the JVM (in the case of a real hanging) by setting the timeout to 0.

 

Last modified on Jul 30, 2015

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.