Summary

When investigating performance issues, it is common to collect thread dumps. Those are really useful artifacts to check what was being executed on the application during an incident.

However, you may come across unnamed thread pools when reviewing such files and may want to understand where they come from. Normally, they will be presented with a generic stack trace, which means they are just waiting for a task to execute. Here is one example:

"pool-5-thread-4" #9332 prio=5 os_prio=0 tid=0x00007f65cc294800 nid=0x3e34 waiting on condition [0x00007f64a99d7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000003d8447588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

With the stack above and the generic name it's impossible to determine the source of such thread. The only way to capture a meaningful stack trace would be to generate a thread dump exactly while those threads are RUNNABLE. The problem is they usually are quite fast to execute a task, lasting only a few milliseconds. This means thread dumps are not useful for this end unless a long-running task is being executed.

Tool

In order to identify the source of such threads, we need a different tool. Java provides a very useful one called Java Flight Recorder:

• About Java Flight Recorder
  

The main advantage is that it is a recording tool. Just like logging, it will capture all the history of the thread during its lifetime. That's the advantage over thread dumps, which just capture a snapshot of that specific moment in time.

Solution

To generate a recording, consult the appropriate Java vendor documentation:

• Oracle - Running Java Flight Recorder
• AdoptOpenJDK - Java Flight Recorder Available in 8u262

Once the file is generated, open it with the JDK Mission Control application:

• JDK Mission Control (JMC) 7 Downloads

Then, go to Java Application > Threads menu to inspect the thread history recorded. There, you will find all threads listed and can search for the unnamed thread pools.

In the example below, it was possible to identify the Page Properties Report macro as the source of pool-5-thread-x: 

 MasterDetail is the old class path name of the Page Properties Report macro code.

The investigation from the example above led to the creation of the following bug:

• CONFSERVER-61452 - Getting issue details... STATUS

Notes

If you still don't see meaningful stack traces on the recording it means that no task was executed while it was being captured. When that happens, try to capture more recordings for longer periods of time, until you can see any task being executed.

 If you are using Oracle JDK, Java Flight Recorder requires a commercial license for use in production.