Authentication Failed Due to Broken Crowd Cache File

Still need help?

The Atlassian Community is here for you.

Ask the community

This article only applies to Atlassian's server products. Learn more about the differences between cloud and server.

This is for an outdated version of Hipchat Server

 This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.

When will my version be deprecated?

The following versions have been deprecated:

  • Hipchat Server 1.3 (EOL Date: Aug 17, 2017)
  • Hipchat Server 2.0 (EOL Date: Jun 17, 2018)
  • Hipchat Server 2.1 (EOL Date: Dec 8, 2018)

The following version will be deprecated soon:

  • Hipchat Server 2.2 (EOL Date: May 30, 2019)

You can read more about Atlassian's End of Life policy here.

You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.

Problem

The problem symptoms may include —but are not limited to— one or many of the followings: 

  • No users (local or directory) able to login to Hipchat Server. Users get this message in the UI when attempting a login:

    "An unknown error occurred. Please try again in a few minutes or reset your password."
  • HTTP Status 404 - Could not find servlet for: /crowd/plugins/servlet/embedded-crowd/directories/list is shown when you go to Server admin > Directory
     
  • The following appears in the /var/log/hipchat/atlassian-crowd.log

    2015-10-15 06:05:05,263 localhost-startStop-1 ERROR [atlassian.plugin.loaders.ScanningPluginLoader] Because of the following exception:
    java.lang.IllegalArgumentException: File must be a jar: /etc/crowd/caches/transformed-plugins/hipchat-server-crowd-plugin-0.10_1441096490000.jar
    	at com.atlassian.plugin.osgi.factory.transform.JarUtils.withJar(JarUtils.java:103)
    	at com.atlassian.plugin.osgi.factory.transform.JarUtils.getManifest(JarUtils.java:38)
    	at com.atlassian.plugin.osgi.factory.OsgiPluginUninstalledHelper.isRemotePlugin(OsgiPluginUninstalledHelper.java:67)
    	at com.atlassian.plugin.osgi.factory.OsgiPlugin.getInstallationMode(OsgiPlugin.java:154)
    	at com.atlassian.plugin.parsers.XmlDescriptorParser.configurePlugin(XmlDescriptorParser.java:113)
    	at com.atlassian.plugin.osgi.factory.OsgiPluginFactory.create(OsgiPluginFactory.java:213)
    	at com.atlassian.plugin.loaders.ScanningPluginLoader.deployPluginFromUnit(ScanningPluginLoader.java:155)
    	at com.atlassian.plugin.loaders.ScanningPluginLoader.loadAllPlugins(ScanningPluginLoader.java:89)
    	at com.atlassian.plugin.loaders.PermissionCheckingPluginLoader.loadAllPlugins(PermissionCheckingPluginLoader.java:24)
    	at com.atlassian.plugin.manager.DefaultPluginManager.earlyStartup(DefaultPluginManager.java:454)
    	at com.atlassian.plugin.manager.DefaultPluginManager.init(DefaultPluginManager.java:431)
    	at com.atlassian.crowd.plugin.CrowdPluginManager.init(CrowdPluginManager.java:69)
    	at com.atlassian.crowd.plugin.CrowdPluginManager.onApplicationStartedEvent(CrowdPluginManager.java:82)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:606)
    	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
    	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48)
    	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
    	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107)
    	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
    	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
    	at com.atlassian.crowd.util.TransactionalEventPublisher.publish(TransactionalEventPublisher.java:25)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:606)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
    	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
    	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    	at com.sun.proxy.$Proxy16.publish(Unknown Source)
    	at com.atlassian.crowd.console.listener.StartupListener.contextInitialized(StartupListener.java:105)
    	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
    	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
    	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
    	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
    	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
    	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:672)
    	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1862)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    	at java.lang.Thread.run(Thread.java:745)
    Caused by: java.util.zip.ZipException: error in opening zip file
    	at java.util.zip.ZipFile.open(Native Method)
    	at java.util.zip.ZipFile.<init>(ZipFile.java:215)
    	at java.util.zip.ZipFile.<init>(ZipFile.java:145)
    	at java.util.jar.JarFile.<init>(JarFile.java:154)
    	at java.util.jar.JarFile.<init>(JarFile.java:118)
    	at com.atlassian.plugin.osgi.factory.transform.JarUtils.withJar(JarUtils.java:98)
    	... 50 more

Diagnosis

  • Password reset for local accounts are not working
  • Email address reset for owner/first user's account is not working
  • Embedded Crowd (Hipchat Server utilises Crowd to handle directory integration) process in the server is running as normal. You can check via this command:

    ps aux | grep crowd
    

    Sample output of running Crowd process:

    admin@server:~$ ps aux | grep crowd
    root      7427  0.4  8.9 2620548 681116 ?      Sl   01:48   2:06 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/opt/atlassian/crowd/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms128m -Xmx1024m -XX:MaxPermSize=256m -Dfile.encoding=UTF-8 -Datlassian.org.osgi.framework.bootdelegation=weblogic,weblogic.*,META-INF.services,com.yourkit,com.yourkit.*,com.chronon,com.chronon.*,org.jboss.byteman,org.jboss.byteman.*,com.jprofiler,com.jprofiler.*,org.apache.xerces,org.apache.xerces.*,org.apache.xalan,org.apache.xalan.*,org.apache.xml.serializer,sun.*,com.sun.xml.bind.v2,com.sun.xml.internal.bind.v2,com.icl.saxon,com_cenqua_clover,com.cenqua.clover,com.cenqua.clover.*,com.atlassian.clover,com.atlassian.clover.*,com.sun.jndi.ldap -Djava.endorsed.dirs=/opt/atlassian/crowd/apache-tomcat/endorsed -classpath /opt/atlassian/crowd/apache-tomcat/bin/bootstrap.jar:/opt/atlassian/crowd/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/crowd/apache-tomcat -Dcatalina.home=/opt/atlassian/crowd/apache-tomcat -Djava.io.tmpdir=/opt/atlassian/crowd/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
    admin    16691  0.0  0.0   8080   916 pts/1    S+   09:43   0:00 grep --color=auto crowd

Cause

At one point, Crowd cache in Hipchat Server got corrupted. This could happen after an upgrade that include a Crowd update (potentially prior to 2.2.3 where we introduced Crowd 2.11.2)

Resolution

(warning) Snapshot the server before making any changes as precautionary measure

SSH into the server and run the followings:

sudo dont-blame-hipchat
/etc/init.d/crowd stop
mv /etc/crowd/caches /etc/crowd/caches.bak 
/etc/init.d/crowd start

The commands above will stop the Crowd service, rename Crowd cache file to something else and start up the service again. User can then attempt a login. If problem persists, send in /var/log/hipchat/atlassian-crowd.log and /var/log/hipchat/coral.log to Hipchat Support.

Last modified on Nov 2, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.