JIRA threads grow uncontrollably with lots of Analytics threads in Timed_Waiting state
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
JIRA threads grow uncontrollably (a few thousands) exceeding the maxThreads specified in the Connector used by JIRA in server.xml. Due to this, JIRA may run out of resources and throw this error:
21-Feb-2017 13:51:46.985 SEVERE [ajp-nio-8009-exec-148] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at com.sun.jndi.ldap.Connection.<init>(Connection.java:234)
at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137)Diagnosis
A lot of these threads are found in thread dumps: Analytics in Timed_Waiting state
"Analytics" prio=1 tid=0x0000000000004645 nid=0 sleeping
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.segment.backo.Backo.sleep(Backo.java:72)
at com.segment.analytics.internal.AnalyticsClient$BatchUploadTask.run(AnalyticsClient.java:158)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.segment.analytics.Platform$1$1.run(Platform.java:44)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- java.util.concurrent.ThreadPoolExecutor$Worker@@8a291d
Cause
The thread above comes from Tempo Analytics, which is bundled in one of the Tempo plugins. The keyword that helps identify this, in this case, is com.segment.
The way to find which plugin is causing this is to unpack the installed plugins' .jar files and then grep for the class name in all the unpacked folders. E.g. for this case:
grep -r "com.segment" ./Be aware that ScriptRunner plugin also uses *com.segment.analytics* classes for analytics purposes.
Workaround
Go to JIRA Administration -> Add-ons -> Tempo Analytics -> disable Analytics. This will prevent the Analytics threads from accumulating.
Resolution
According to Tempo for JIRA, the issue is fixed in Tempo Timesheets 8.1.3. Release notes can be found here.