Bitbucket Server stops responding due to a Java-level deadlock

Related content

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Symptoms

Bitbucket Server becomes unresponsive.

The sequence of events that lead to this situation are not yet known and it is not easily reproducible.

Environment

The issue affects instances using Java 8.

Thread dump

Thread dumps taken during this time would show Java-level deadlocks.

The deadlock is encountered between a ClientPoller, BlockPoller and a few other http/s threads on the following methods:

  • java.nio.channels.spi.AbstractSelectableChannel.removeKey
  • java.nio.channels.spi.AbstractSelector.cancel

Click here for a sample thread dump 
Found one Java-level deadlock:
=============================
"https-jsse-nio-8201-exec-222":
  waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-ClientPoller"
"https-jsse-nio-8201-ClientPoller":
  waiting to lock monitor 0x00007f8bb41791b8 (object 0x00000006c4342f48, a java.lang.Object),
  which is held by "https-jsse-nio-8201-exec-52"
"https-jsse-nio-8201-exec-52":
  waiting to lock monitor 0x00007f8c38306148 (object 0x00000006c5f6dcb0, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-BlockPoller"
"https-jsse-nio-8201-BlockPoller":
  waiting to lock monitor 0x00007f8b441b6568 (object 0x00000006c4343de0, a java.lang.Object),
  which is held by "https-jsse-nio-8201-exec-15"
"https-jsse-nio-8201-exec-15":
  waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-ClientPoller"

Java stack information for the threads listed above:
===================================================
"https-jsse-nio-8201-exec-222":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c43842d0> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c43842c0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c43842f8> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c43843a8> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4384308> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-ClientPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c6278be8> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c31a3fc0> (a sun.nio.ch.Util$3)
- locked <0x00000006c31a3fb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c31a3f68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:706)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-exec-52":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c5f6dcb0> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4342f58> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4342f80> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4343030> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4342f90> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-BlockPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c5f6dcb0> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c3da4290> (a sun.nio.ch.Util$3)
- locked <0x00000006c3da4280> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c3da4238> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:310)
"https-jsse-nio-8201-exec-15":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4394e78> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4394ea0> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4394f50> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4394eb0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

Cause

The deadlock is caused by the following bug in Tomcat versions 9.0.20 to 9.0.31:

64007 - A Deadlock with Poller run and cancelledKey

Affected Versions

The Bitbucket Server versions affected by this bug are:

Bitbucket VersionEmbedded Tomcat Version
6.5.x9.0.21
6.6.x9.0.21
6.7.x9.0.21
6.8.x9.0.21
6.9.x9.0.21
6.10.0 - 6.10.29.0.27
7.0.0 - 7.0.19.0.30

Workarounds

Perform a restart of Bitbucket Server.
For multi-node Bitbucket Data Center instances, only the affected node needs to be restarted.

Fix Versions

This issue is fixed in Bitbucket Server versions 6.10.3+, 7.0.2+, 7.1.0+ or later.

These versions use Tomcat version 9.0.31+ or later.

Java 11

If an upgrade to a fix version of Bitbucket is not possible, an alternative is to use Java 11 where the issue with deadlocks is addressed (reference: Tomcat repository commit). Java 11 is installed by default with the Bitbucket 8.6.0 installer.


Description

Bitbucket Server stops responding due to a Java-level deadlock.

The issue is due to a bug in certain embedded Tomcat versions

ProductBitbucket Server

Last modified on Nov 23, 2022

Was this helpful?

Yes
No
Provide feedback about this article

Related content

Powered by Confluence and Scroll Viewport.