Summary

Unable to start Confluence Data Center on a Linux machine without the AVX instruction set flag

Environment

• Confluence Data Center 8.8.x to latest.
• Linux Server.

Diagnosis

• During the startup, the following JVM crash stack can be seen in the Catalina.out file.

  #
  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGILL (0x4) at pc=0x00007f2af2e49fb4, pid=118018, tid=118041
  #
  # JRE version: OpenJDK Runtime Environment Temurin-17.0.8.1+1 (17.0.8.1+1) (build 17.0.8.1+1)
  # Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.8.1+1 (17.0.8.1+1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
  # Problematic frame:
  # C  [libbc-lts-avx.so+0x5fb4]  init_bytearray_ctx+0x4
  #
  # Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /data/jira/Confluence/confluence_5_4_4/bin/core.118018)
  #
  # An error report file with more information is saved as:
  # /data/jira/Confluence/confluence_5_4_4/bin/hs_err_pid118018.log
  #
  # If you would like to submit a bug report, please visit:
  #   https://github.com/adoptium/adoptium-support/issues
  # The crash happened outside the Java Virtual Machine in native code.
  # See problematic frame for where to report the bug.
  #

• hs_err_pid logs has the following entries related to the crash.

  #
  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGILL (0x4) at pc=0x00007f007c43b5b4, pid=1594195, tid=1594218
  #
  # JRE version: OpenJDK Runtime Environment Temurin-17.0.8.1+1 (17.0.8.1+1) (build 17.0.8.1+1)
  # Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.8.1+1 (17.0.8.1+1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
  # Problematic frame:
  # C  [libbc-lts-avx.so+0x85b4]  init_bytearray_ctx+0x4
  #
  # Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /data/jira/Confluence/confluence_5_4_4/bin/core.1594195)
  #
  # If you would like to submit a bug report, please visit:
  #   https://github.com/adoptium/adoptium-support/issues
  # The crash happened outside the Java Virtual Machine in native code.
  # See problematic frame for where to report the bug.
  #
  ---------------  S U M M A R Y ------------
  Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=/data/jira/Confluence/confluence_5_4_4/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Datlassian.plugins.startup.options= -Dindex.queue.thread.count=10 -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xlog:gc*:file=/data/jira/Confluence/confluence_5_4_4/logs/gc-%t.log:tags,time,uptime,level:filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-opens=java.base/sun.util.locale=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports=java.base/sun.security.action=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xml.internal.utils=ALL-UNNAMED --add-exports=java.desktop/sun.font=ALL-UNNAMED --add-exports=java.base/sun.security.util=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-modules=java.se -XX:ReservedCodeCacheSize=256m -Xms8192m -Xmx8192m -Dignore.endorsed.dirs= -Dcatalina.base=/data/jira/Confluence/confluence_5_4_4 -Dcatalina.home=/data/jira/Confluence/confluence_5_4_4 -Djava.io.tmpdir=/data/jira/Confluence/confluence_5_4_4/temp org.apache.catalina.startup.Bootstrap start
  Host: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, 4 cores, 15G, Oracle Linux Server release 8.10
  Time: Tue Jul 16 17:22:59 2024 IST elapsed time: 69.342502 seconds (0d 0h 1m 9s)
  ---------------  T H R E A D  ---------------
  Current thread (0x00007f0104f33850):  JavaThread "Catalina-utility-1" [_thread_in_native, id=1594218, stack(0x00007f00d8fb4000,0x00007f00d90b5000)]
  Stack: [0x00007f00d8fb4000,0x00007f00d90b5000],  sp=0x00007f00d90b0da8,  free space=1011k
  Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
  C  [libbc-lts-avx.so+0x85b4]  init_bytearray_ctx+0x4
  Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
  j  org.bouncycastle.crypto.engines.AESNativeCBC.init(J[B[B)V+0
  j  org.bouncycastle.crypto.engines.AESNativeCBC.init(Lorg/bouncycastle/crypto/params/KeyParameter;)V+117
  j  org.bouncycastle.crypto.engines.AESNativeCBC.init(ZLorg/bouncycastle/crypto/CipherParameters;)V+85
  j  org.bouncycastle.crypto.paddings.PaddedBufferedMultiBlockCipher.init(ZLorg/bouncycastle/crypto/CipherParameters;)V+43
  j  org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.init(ZLorg/bouncycastle/crypto/CipherParameters;)V+6
  j  org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineInit(ILjava/security/Key;Ljava/security/spec/AlgorithmParameterSpec;Ljava/security/SecureRandom;)V+1999
  j  org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineInit(ILjava/security/Key;Ljava/security/AlgorithmParameters;Ljava/security/SecureRandom;)V+63
  j  javax.crypto.Cipher.init(ILjava/security/Key;Ljava/security/AlgorithmParameters;Ljava/security/SecureRandom;)V+35 java.base@17.0.8.1
  j  javax.crypto.Cipher.init(ILjava/security/Key;Ljava/security/AlgorithmParameters;)V+7 java.base@17.0.8.1
  j  com.atlassian.db.config.password.ciphers.algorithm.AlgorithmCipher.encrypt(Lcom/atlassian/db/config/password/ciphers/algorithm/paramters/EncryptionParameters;)Lcom/atlassian/db/config/password/ciphers/algorithm/paramters/DecryptionParameters;+49
  j  com.atlassian.db.config.password.ciphers.algorithm.AlgorithmCipher.encrypt(Ljava/lang/String;)Ljava/lang/String;+28
  j  com.atlassian.crowd.crypto.ClusterAwareCipherWrapper.encrypt(Lcom/atlassian/db/config/password/ciphers/algorithm/paramters/EncryptionParameters;)Lcom/atlassian/db/config/password/ciphers/algorithm/paramters/DecryptionParameters;+115
  j  com.atlassian.crowd.crypto.DbConfigPasswordCipherEncryptor.doEncrypt(Ljava/lang/String;Ljava/lang/String;)Lcom/atlassian/db/config/password/ciphers/algorithm/paramters/DecryptionParameters;+83
  j  com.atlassian.crowd.crypto.DbConfigPasswordCipherEncryptor.changeEncryptionKey()Z+4
  j  com.atlassian.crowd.crypto.CachedEncryptor.changeEncryptionKey()Z+4
  j  com.atlassian.crowd.crypto.SaltingEncryptor.changeEncryptionKey()Z+4
  j  com.atlassian.crowd.crypto.MissingKeyHandlingEncryptor.encrypt(Ljava/lang/String;)Ljava/lang/String;+30
  j  com.atlassian.crowd.crypto.PrefixBasedSwitchableEncryptor.encrypt(Ljava/lang/String;)Ljava/lang/String;+75
  j  com.atlassian.crowd.crypto.DirectoryPasswordsEncryptor$$Lambda$2585+0x00007f0086062e38.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
  j  com.atlassian.crowd.directory.LazyAttributesEvaluationDirectory.lambda$extractFromDelegateAndTransform$1(Ljava/lang/String;)Ljava/util/Optional;+41
  j  com.atlassian.crowd.directory.LazyAttributesEvaluationDirectory$$Lambda$2591+0x00007f0086063dd0.call()Ljava/lang/Object;+8

• When installing Confluence Data Centre 8.8.x with the default internal user directory, there are no startup issues on the machine.
• The issue starts once we add an external user directory.
• However, startup issue has been noticed in the 9.x version with internal user directory.

Cause

• Confluence is crashing while encrypting the directory password.
• The UserDirectoryPasswordEncryptionUpgradeTask is a newly introduced feature for encrypting LDAP directory passwords, available starting from version 8.8.0.
• The issue is related to the VMware/hardware configuration. The AVX instruction set flag, used by Bouncy Castle for encryption, is not available on the system, which is causing the problem.
• The lscpu command will display the CPU flags, allowing us to check whether the AVX flag is available on the machine. AVX flag is not available in the below example.

  [admin@xx.xx.xx.xx ConfluenceHost]$ lscpu
  Architecture:        x86_64
  CPU op-mode (s):     32-bit, 64-bit
  Byte Order:          Little Endian
  CPU(s):              4
  On-line CPU(s) list: 0-3
  Thread(s) per core:  1
  Core (s) per socket: 1
  Socket(s):           4
  NUMA node (s):       1
  Vendor ID:           GenuineIntel
  CPU family:          6
  Model:               85
  Model name:          Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
  Stepping:            4
  CPU MHz:             2294.609
  BogoMIPS:            4589.21
  Hypervisor vendor:   VMware
  Virtualization type: full
  Lld cache:           32K
  Lli cache:           32K
  L2 cache:            1024K
  L3 cache:            25344K
  NUMA node0 CPU(s):   0-3
  Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpelgb rdtscp lm consta nt_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline _timer aes fl6c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmil smep bmi2 invpcid r seed adx smap clflushopt clwb arat ospke md clear flush lld arch capabilities

Solution

To fix this problem, follow the below steps:

• Please inform the VMware team to add AVX flag.
• Below is example of lscpu output with AVX flag (You will be able to see avx and avx2 in the flags options)

  admin@xx.xx.xx.xx :~$ lscpu
  Architecture:                         x86_64
  CPU op-mode(s):                       32-bit, 64-bit
  Byte Order:                           Little Endian
  Address sizes:                        48 bits physical, 48 bits virtual
  CPU(s):                               2
  On-line CPU(s) list:                  0,1
  Thread(s) per core:                   2
  Core(s) per socket:                   1
  Socket(s):                            1
  NUMA node(s):                         1
  Vendor ID:                            AuthenticAMD
  CPU family:                           25
  Model:                                1
  Model name:                           AMD EPYC 7R13 Processor
  Stepping:                             1
  CPU MHz:                              2649.998
  BogoMIPS:                             5299.99
  Hypervisor vendor:                    KVM
  Virtualization type:                  full
  L1d cache:                            32 KiB
  L1i cache:                            32 KiB
  L2 cache:                             512 KiB
  L3 cache:                             4 MiB
  NUMA node0 CPU(s):                    0,1
  Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid

• You should now be able to start Confluence.

In case of further assistance please reach out to the Atlassian Support team and share the information that's been gathered so far along with a fresh Support zip file.