How to analyze thread dumps

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Purpose

This document includes both a general overview of what thread dumps are as well as a set of tips for how to analyze them. If you're already familiar with thread dumps, please skip to So, how do I analyze them?, where I'll go through a set of Confluence thread dumps as an example.

Solution

A thread dump is a snapshot of everything that an application is doing right at the instance it's taken. If an application log is a diary of a vacation, a thread dump is a photo. A diary will tell you whatever the author (in this case the application itself) thinks is important, but a photo will give you an unfiltered view of that particular moment. String together a couple of photos and you have a powerful tool for finding out what an application was doing over a certain period.

A thread dump is just a collection of stack traces, one for each thread that's running in the instance. There are different types of threads, and the number of threads will vary depending on the configuration of the instance. For example, if you have changed the maxThreads value of your HTTP connector in <install_directory>/conf/server.xml, then the number of HTTP threads in a thread dump will also change. Let's have a look at one of HTTP threads:

"http-bio-80-exec-77" daemon prio=6 tid=0x0000000026f29000 nid=0xbd0 runnable [0x0000000020c7f000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
    at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
    at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
    - locked <0x00000007b16e3e88> (a org.apache.tomcat.util.net.SocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

This is a pretty boring thread, but there are a few things we should note:

The name of the thread:

In our example: http-bio-80-exec-77. This tells us quite a few things:

  1. The port the instance is running on:http-bio-80-exec-77 tells us we're running on port 80
  2. What sort of connector is being used: http-bio-80-exec-77 tells us we're using the Java BIO connector, the default for Tomcat 7
  3. The number of the thread: http-bio-80-exec-77 tells us this http thread #77, which isn't very useful information by itself, but very useful when we look at the thread across multiple dumps to see if what it's doing has changed

The state of the thread:

In our example: java.lang.Thread.State: RUNNABLE

Threads can be in a few different states:

  • NEW
  • RUNNABLE
  • BLOCKED
  • WAITING
  • TIMED_WAITING
  • TERMINATED

We'll talk more about what these states mean later, but for more information about this, check out the Java docs on thread state.

The stack trace itself

In our example:

    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
    at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
    at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
    - locked <0x00000007b16e3e88> (a org.apache.tomcat.util.net.SocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

This is a breakdown of exactly what that particular thread was doing at the instant this thread dump was taken. In our example, this thread is reading from a socket. This means the thread is connected to a client, and is either in the process of getting data sent to it, or waiting to time out after data has already been sent to it. If we were analysing the thread dumps, this one would be pretty safe to ignore.

When should I use them?

Virtually every performance problem. Thread dumps are second to only application logs in how many problems they will help solve. If an instance is seeing performance problems, thread dumps should always be gathered.

However, it's important they're gathered in a meaningful way. Note that thread dumps are only useful is gathered while the problem is happening. Also, if we take the thread dumps too far apart, we may miss information. Too close together, and we may be swamped with too much information.

My recommendation, as a starting point, is always a total of 6 thread dumps, one taken every 10 seconds. This seems to be a good middle ground that catches threads that long running, but doesn't include too much noise from the standard operation of an instance.

How do I make them?

This varies between operating systems, but Atlassian has made guides that cover all the things that need to be taken into account. For example:

So, how do I analyse them? 

Analysing thread dumps is all about removing the noise and finding something meaningful. It's a process of narrowing down the results until we're only considering a handful of threads. There are a couple tricks that make this process a lot easier.

There are also quite a few tools out there that help with this. I prefer "Thread Dump Analyzer" (TDA), and will be using that in this document.

By default, thread dumps can be found in <install_directory>/logs/catalina.out. If they are split over multiple files (e.g. dump1.txt, dump2.txt, etc), combining them into one will make things easier for you.

Help! catalina.out is too big! TDA crashes when I try to open the dumps!

No problems! The thread dumps we're after are almost always in the last section of this file, so we can split it up and dump the irrelevant logging before the thread dumps were taken. To do so, I use the split command:

split -b 200m catalina.out

This will split the file down into 200 mb chunks that TDA will handle without problem.

In old versions of TDA, there is bug with how thread dumps produced by Java 8 applications are handled. Make sure you're using the latest version available.

Long running threads

The reason I ask for one thread dump every 10 seconds is because it will reveal any long running threads. These are the most interesting part of any set of thread dumps, because they are the threads that are either stuck or are taking a very long time to complete, which is almost universally where a performance problem lies.

It's important to note that when a thread appears as a long running thread, it means that the stack trace matches. That means that, over that 10 second period, the thread had been doing exactly the same method in the code for the entire time. There's definitely some exceptions, but in general it's rare for any application to have a method that legitimately runs for this amount of time when everything is working correctly. Therefore, any long running thread is worth our interest.

To find long running threads, open up TDA (note: ensure you have executable permissions. on the tda.sh script!):

cd /path/to/tda/
cd bin
./tda.sh

Drop your thread dumps into the app via File > Open. Having done so, you'll get something like this:

To find the long running threads, highlight all the thread dumps you want to check, and then click on the binoculars:

In the pop up dialogue, click start detection, and you'll get your long running threads just below the dumps themselves:

In my example, each thread dump has 157 threads. In my set of long running threads, we're down to 91 threads that were doing the same thing over the course of all 5 thread dumps, so we've already been able to discount 66 threads.

The minimum occurrence option defaults to the number of dumps selected, so you can lower the threshold to get a higher hit rate of long running threads here. For example, you can ask for any threads that occur across 2 of the dumps as opposed to seeing only threads that are running across all thread dumps.

What are all these thread types?

Unfortunately, 91 threads is still a huge amount to analyse, so we need to keep narrowing down the results. The next step I take is looking at the type of threads that we have left. Let's have a look at the results of our long running thread detection:

Under Name, we can see quite a few different types of threads.

When looking at Atlassian applications, there are two types that contain the majority of performance problems:

  • HTTP threads
  • scheduler_Worker threads

HTTP threads are what do most of the work in our applications, and scheduler_Worker threads handle scheduled tasks such as sending mail and flushing the index. Other threads include those generated by features, such as the NavLink Plugin, as well as threads generated by Tomcat or Java, which are almost always safe to ignore. That's not to say that these other thread types can't contain useful information, but starting with the HTTP threads and scheduler_Worker threads will be the correct move in the vast majority of cases.

In my example, looking at exclusively the HTTP and scheduler_Worker threads brings us down to a total of just 14 threads to analyse. We're getting close!

Thread states

Even still, not all these threads are going to be interesting. The next thing to check is the state of the threads. There are 6 different possible states (NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED), but the ones of interest to us are:

  • RUNNABLE. This means the thread is actively doing something. These are the most interesting threads, because in the context of our long running threads, it means it's being doing one thing for quite a while, which is almost always unusual.
  • BLOCKED. The thread has something to do, but it can't until a shared resource is freed. An example would be a thread that wants to access a table of the database, but can't do so because another thread has locked that table. These are also interesting to us, as many blocked threads indicates a bottleneck in the application.
  • WAITING and TIMED_WAITING. These threads have been explicitly told (in the code) to do nothing. The difference between WAITING and TIMED_WAITING is that TIMED_WAITING includes a timeout, while a WAITING thread will wait indefinitely.  Waiting threads are usually less helpful to us for identifying performance problems, as it's often reasonable for threads to be in this state for long periods of time. If these are the only interesting threads, I'd recommend extending the interval between thread dumps.

In my example, of the 14 HTTP and scheduler_Worker threads there were 5 RUNNABLE threads, 0 BLOCKED threads, and 9 WAITING threads.

Let's discount the WAITING threads and focus on the 5 remaining RUNNABLE threads.

What's in the stack trace?

So, we have our 5 RUNNABLE threads. Now we need to to dive into what's in the stack traces themselves. Here's one of the RUNNABLE HTTP threads from my example:

"http-bio-80-exec-70" daemon prio=6 tid=0x0000000013ff0000 nid=0xe90 runnable [0x0000000020e76000]
   java.lang.Thread.State: RUNNABLE
	at com.ctc.wstx.sw.SimpleNsStreamWriter.writeStartElement(SimpleNsStreamWriter.java:218)
	at com.ctc.wstx.evt.WstxEventWriter.add(WstxEventWriter.java:137)
	at com.atlassian.confluence.content.render.xhtml.ConfluenceXMLEventWriter.add(ConfluenceXMLEventWriter.java:47)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$XmlEventStreamable.writeTo(DefaultFragmentTransformer.java:285)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV2Marshaller$1.writeTo(StorageMacroV2Marshaller.java:104)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
	at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
	at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
	at com.atlassian.confluence.content.render.xhtml.editor.EditorXhtmlTransformer.transform(EditorXhtmlTransformer.java:50)
	at com.atlassian.confluence.content.render.xhtml.TransformerChain.transform(TransformerChain.java:41)
	at com.atlassian.confluence.content.render.xhtml.PluggableTransformerChain.transform(PluggableTransformerChain.java:51)
	at com.atlassian.confluence.content.render.xhtml.editor.DefaultEditorConverter.convert(DefaultEditorConverter.java:41)
	at com.atlassian.confluence.content.service.DefaultDraftService.saveDraftFromEditor(DefaultDraftService.java:91)
	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy118.saveDraftFromEditor(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:129)
	at com.sun.proxy.$Proxy270.saveDraftFromEditor(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:154)
	at com.sun.proxy.$Proxy270.saveDraftFromEditor(Unknown Source)
	at com.atlassian.confluence.tinymceplugin.rest.DraftsResource.save(DraftsResource.java:105)
	at sun.reflect.GeneratedMethodAccessor1277.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:234)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker._dispatch(DispatchProviderHelper.java:230)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:178)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
	at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:73)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.labs.botkiller.BotKillerFilter.doFilter(BotKillerFilter.java:36)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:26)
	at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:40)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.mywork.client.filter.ServingRequestsFilter.doFilter(ServingRequestsFilter.java:37)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.plugins.cors.CorsFilter.doFilter(CorsFilter.java:65)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter.doFilter(PrettyUrlsSiteMeshFixupFilter.java:36)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at com.atlassian.confluence.util.profiling.ProfilingPageFilter.doFilter(ProfilingPageFilter.java:45)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.core.filters.ServletContextThreadLocalFilter.doFilter(ServletContextThreadLocalFilter.java:21)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.cache.TransactionalCacheFactoryCleanupFilter.doFilter(TransactionalCacheFactoryCleanupFilter.java:68)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.core.filters.ServletContextThreadLocalFilter.doFilter(ServletContextThreadLocalFilter.java:21)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:41)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.util.UserNameHeaderFilter.doFilter(UserNameHeaderFilter.java:26)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.ConfluenceTimeoutFilter.doFilter(ConfluenceTimeoutFilter.java:63)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:240)
	at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.ThreadLocalCacheFilter.doFilter(ThreadLocalCacheFilter.java:30)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100)
	at com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:35)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:44)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.util.ClusterHeaderFilter.doFilter(ClusterHeaderFilter.java:44)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.orm.hibernate.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:170)
	at com.atlassian.spring.filter.FlushingSpringSessionInViewFilter.doFilterInternal(FlushingSpringSessionInViewFilter.java:29)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.util.profiling.filters.ProfilingFilter.doFilter(ProfilingFilter.java:99)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.core.filters.cache.AbstractCachingFilter.doFilter(AbstractCachingFilter.java:33)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:33)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.gzipfilter.GzipFilter.doFilterInternal(GzipFilter.java:121)
	at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:92)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.confluence.web.filter.ConfluenceTimingFilter.doFilter(ConfluenceTimingFilter.java:79)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.confluence.extra.webdav.servlet.filter.ReverseProxyFilter.doFilter(ReverseProxyFilter.java:427)
	at com.atlassian.confluence.extra.webdav.servlet.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:34)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:58)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:44)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:73)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:53)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:32)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:71)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.DropIfNotSetupFilter.doFilter(DropIfNotSetupFilter.java:37)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:50)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
	- locked <0x00000007af39d048> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)


Once again, we're faced with information overload. What we need to do is identify if this thread is going to help us solve the problem. The biggest giveaway here is looking for com.atlassian package. This example has a heap of com.atlassian methods, so it's safe to say that it's going to help us pinpoint the problem.

Clean it up

However, there's still about 350 lines in this stack trace. Let's trim it down a bit by getting rid of some of the wiring that goes into a Java application. I do this by dumping the stack trace into an editor and using pattern matching to find and remove this wiring. Here's what I use:

(.*(servlet|catalina|filters).*filter.*|.*doFilter.*|.*sitemesh.*|.*invoke.*|.*intercept.*|.*dispatch.*|.*com\.sun\.jersey.*|.*org\.apache\.catalina.*|.*proceed\(.*)

I replace this with an empty string, and then run a second find and replace that turns "\n\n" into "\n", getting rid of all the empty strings. Here's what we're left with:

"http-bio-80-exec-70" daemon prio=6 tid=0x0000000013ff0000 nid=0xe90 runnable [0x0000000020e76000]
   java.lang.Thread.State: RUNNABLE
    at com.ctc.wstx.sw.SimpleNsStreamWriter.writeStartElement(SimpleNsStreamWriter.java:218)
    at com.ctc.wstx.evt.WstxEventWriter.add(WstxEventWriter.java:137)
    at com.atlassian.confluence.content.render.xhtml.ConfluenceXMLEventWriter.add(ConfluenceXMLEventWriter.java:47)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$XmlEventStreamable.writeTo(DefaultFragmentTransformer.java:285)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.writeRawXML(StorageMacroV1Marshaller.java:118)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller.access$400(StorageMacroV1Marshaller.java:20)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:76)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.xhtml.api.MacroDefinition.getBodyText(MacroDefinition.java:113)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV1Marshaller$1.writeTo(StorageMacroV1Marshaller.java:64)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.content.render.xhtml.storage.macro.StorageMacroV2Marshaller$1.writeTo(StorageMacroV2Marshaller.java:104)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$NonXmlSubstreamable.writeTo(DefaultFragmentTransformer.java:302)
    at com.atlassian.confluence.content.render.xhtml.transformers.DefaultFragmentTransformer$AggregatedXmlStreamable.writeTo(DefaultFragmentTransformer.java:325)
    at com.atlassian.confluence.content.render.xhtml.Streamables.writeToString(Streamables.java:214)
    at com.atlassian.confluence.content.render.xhtml.editor.EditorXhtmlTransformer.transform(EditorXhtmlTransformer.java:50)
    at com.atlassian.confluence.content.render.xhtml.TransformerChain.transform(TransformerChain.java:41)
    at com.atlassian.confluence.content.render.xhtml.PluggableTransformerChain.transform(PluggableTransformerChain.java:51)
    at com.atlassian.confluence.content.render.xhtml.editor.DefaultEditorConverter.convert(DefaultEditorConverter.java:41)
    at com.atlassian.confluence.content.service.DefaultDraftService.saveDraftFromEditor(DefaultDraftService.java:91)
    at com.sun.proxy.$Proxy118.saveDraftFromEditor(Unknown Source)
    at com.sun.proxy.$Proxy270.saveDraftFromEditor(Unknown Source)
    at com.sun.proxy.$Proxy270.saveDraftFromEditor(Unknown Source)
    at com.atlassian.confluence.tinymceplugin.rest.DraftsResource.save(DraftsResource.java:105)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
    - locked <0x00000007af39d048> (a org.apache.tomcat.util.net.SocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

We're down to 100 lines, but that's still a lot to look through.

Where to look

Thankfully, we don't need to check every single line. What we can do instead is start at the top of the stack trace and scan down until we get to the last couple com.atlassian entries. In our example, that takes us down to line 97-101:

    at com.atlassian.confluence.content.service.DefaultDraftService.saveDraftFromEditor(DefaultDraftService.java:91)
    at com.sun.proxy.$Proxy118.saveDraftFromEditor(Unknown Source)
    at com.sun.proxy.$Proxy270.saveDraftFromEditor(Unknown Source)
    at com.sun.proxy.$Proxy270.saveDraftFromEditor(Unknown Source)
    at com.atlassian.confluence.tinymceplugin.rest.DraftsResource.save(DraftsResource.java:105)

All of a sudden, we know a lot more about what is going on. Confluence is trying to save a draft! All the method calls above this one are in someway related to this high-level operation. From here, we can look back up through the rest of the method calls, remembering that they're all there as part of trying to save a draft. 

Red Herrings

In this exercise we had a strong focus on RUNNABLE threads. However, a thread being in the RUNNABLE state doesn't necessarily indicate a problem. For example, there is always one Garbage Collection thread per core, and these will always be RUNNABLE, whether they're currently performing a collection or not. That they are RUNNABLE doesn't mean that GC is a problem however, even though it looks that way at first glance, as they're likely not collecting.

One of the biggest contributors to success when analysing thread dumps is being able to separate the wheat from the chaff, and the best way to gain this ability is to look at a lot of thread dumps.

Blocked Threads

Outside of RUNNABLE threads, the other major problem that thread dumps can reveal is BLOCKED threads. Thankfully, TDA provides the 'Monitors' feature which makes identifying blocking problems much easier. To use this, go to Monitors,  right click and sort by thread count. TDA will highlight any monitors that are blocking many threads in red, and these should always be investigated. 


CPU Usage

Often, it's useful to know which threads are chewing up the CPU of the machine the application is running on. Thankfully, you can use the top command to generate a list of processes running on a machine that contains how much CPU each of the processes is using. e.g.:

top -n1 -b -H

This command prints off all the individual threads used by processes on a server. You can then map the process IDs (PID) here against the NativeID column in TDA. For example, if you have 3 threads at 100 %CPU in top then you can search for them quickly in TDA, and find out what exactly the application is doing that is consuming so much CPU.

Check the source

In some cases, we'll need to go one step further and check the source of the application itself. Thankfully, IDEA has a great feature for this. Copy your stack trace to the clipboard, and then head into IDEA, find Analyze along the top, and select Analyze Stacktrace... from the dropdown. This will give you a set of links to the source of the methods in the stacktrace. We allow the source for our applications to be downloaded by anyone that has a commercial license. Please see our licensing FAQs for more information.

DescriptionThis document includes both a general overview of what thread dumps are as well as a set of tips for how to analyze them.
ProductJira, Confluence, Bitbucket, Crucible, Fisheye, Bamboo
PlatformServer
Last modified on Feb 13, 2023

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.