High CPU usage caused by Stash

Symptoms

Stash is taking a lot of CPU usage on the server - we had examples of 80%+ CPU usage (can be viewed with a top command - example below).

top - 15:54:19 up 158 days, 1:02, 2 users, load average: 4.04, 5.00, 4.58
Tasks: 243 total, 1 running, 240 sleeping, 0 stopped, 2 zombie
Cpu(s): 81.1%us, 3.7%sy, 0.0%ni, 15.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 3924792k total, 3812744k used, 112048k free, 68068k buffers
Swap: 2097144k total, 110204k used, 1986940k free, 1917588k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9569 git 20 0 3632m 1.3g 10m S 301.8 34.1 1277:32 java
7210 git 20 0 212m 22m 19m S 5.9 0.6 0:00.81 git
7249 git 20 0 185m 21m 20m S 5.6 0.6 0:00.46 git
7166 git 20 0 214m 24m 20m S 5.3 0.7 0:01.12 git
7238 git 20 0 185m 21m 20m S 4.9 0.6 0:00.48 git
7155 git 20 0 215m 26m 21m S 4.6 0.7 0:01.39 git
7239 git 20 0 185m 22m 20m S 4.6 0.6 0:00.49 git
7267 git 20 0 161m 18m 17m S 3.9 0.5 0:00.24 git
27888 git 20 0 878m 363m 174m S 1.0 9.5 1:03.53 git
7163 82052508 20 0 33060 2464 1700 R 0.3 0.1 0:00.07 top

 

Cause

There could be different root causes for this issue.

We have seen "Awesome Graphs" plugin come up as the root cause of many of them. This plugin appears to cope well with smaller repositories but the performance can drop with larger ones.

To identify this is the case, refer to atlassian-stash-profile.log and if you found the entry below (calling "HistoryService" as you can see on the second line), it is most probably the Awesome Graphs plugin:

2013-08-20 15:19:40,779 | pool-5-thread-252 | 549x73988x2 | stashwallmonitor | 6w76oc
[68152ms] - Page com.atlassian.stash.history.HistoryService.getChangesets(Repository,String,String,PageRequest)
  [0ms] - ScmCommandFactory com.atlassian.stash.scm.ScmService.getCommandFactory(Repository)
  [68101ms] - /opt/git/bin/git rev-list --format=%H%x02%h%x02%P%x02%p%x02%aN%x02%aE%x02%at%n%B%n%x03 HEAD --
  [0ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)
  [0ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)
  [0ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)
  [44ms] - Map com.atlassian.stash.internal.content.ChangesetDao.getAttributesForChangesets(Collection,Collection)

 

Thread dumps (thread-dump directory after you generate a zip log) might reinforce the "Awesome Graphs"

 

[62670] pool-5-thread-409: TIMED_WAITING (waiting on java.util.concurrent.CountDownLatch$Sync@7144afda)
   ...
   com.stiltsoft.stash.graphs.manager.GraphsManager.getChangesets(GraphsManager.java:45)
   com.stiltsoft.stash.graphs.manager.CommitActivityManager.getCommitActivity(CommitActivityManager.java:46)
   com.stiltsoft.stash.graphs.manager.CommitActivityManager.access$100(CommitActivityManager.java:22)
   com.stiltsoft.stash.graphs.manager.CommitActivityManager$EntityBuilder.getEntity(CommitActivityManager.java:112)
   com.stiltsoft.stash.graphs.manager.CommitActivityManager$EntityBuilder.getEntity(CommitActivityManager.java:99)
   com.stiltsoft.stash.graphs.cache.CacheUpdateCallable.call(CacheUpdateCallable.java:25)
   java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   java.util.concurrent.FutureTask.run(FutureTask.java:166)
   java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   java.util.concurrent.FutureTask.run(FutureTask.java:166)
   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   java.lang.Thread.run(Thread.java:722)

 

Resolution

  • Uninstall the "Awesome Graphs" plugin.

Last modified on Mar 30, 2016

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.