Authentication Failed Due to Broken Crowd Cache File
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products will end after February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
This is for an outdated version of Hipchat Server
This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.
Problem
The problem symptoms may include —but are not limited to— one or many of the followings:
No users (local or directory) able to login to Hipchat Server. Users get this message in the UI when attempting a login:
"An unknown error occurred. Please try again in a few minutes or reset your password."
- HTTP Status 404 - Could not find servlet for: /crowd/plugins/servlet/embedded-crowd/directories/list is shown when you go to Server admin > Directory
The following appears in the
/var/log/hipchat/atlassian-crowd.log
2015-10-15 06:05:05,263 localhost-startStop-1 ERROR [atlassian.plugin.loaders.ScanningPluginLoader] Because of the following exception: java.lang.IllegalArgumentException: File must be a jar: /etc/crowd/caches/transformed-plugins/hipchat-server-crowd-plugin-0.10_1441096490000.jar at com.atlassian.plugin.osgi.factory.transform.JarUtils.withJar(JarUtils.java:103) at com.atlassian.plugin.osgi.factory.transform.JarUtils.getManifest(JarUtils.java:38) at com.atlassian.plugin.osgi.factory.OsgiPluginUninstalledHelper.isRemotePlugin(OsgiPluginUninstalledHelper.java:67) at com.atlassian.plugin.osgi.factory.OsgiPlugin.getInstallationMode(OsgiPlugin.java:154) at com.atlassian.plugin.parsers.XmlDescriptorParser.configurePlugin(XmlDescriptorParser.java:113) at com.atlassian.plugin.osgi.factory.OsgiPluginFactory.create(OsgiPluginFactory.java:213) at com.atlassian.plugin.loaders.ScanningPluginLoader.deployPluginFromUnit(ScanningPluginLoader.java:155) at com.atlassian.plugin.loaders.ScanningPluginLoader.loadAllPlugins(ScanningPluginLoader.java:89) at com.atlassian.plugin.loaders.PermissionCheckingPluginLoader.loadAllPlugins(PermissionCheckingPluginLoader.java:24) at com.atlassian.plugin.manager.DefaultPluginManager.earlyStartup(DefaultPluginManager.java:454) at com.atlassian.plugin.manager.DefaultPluginManager.init(DefaultPluginManager.java:431) at com.atlassian.crowd.plugin.CrowdPluginManager.init(CrowdPluginManager.java:69) at com.atlassian.crowd.plugin.CrowdPluginManager.onApplicationStartedEvent(CrowdPluginManager.java:82) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36) at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107) at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160) at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79) at com.atlassian.crowd.util.TransactionalEventPublisher.publish(TransactionalEventPublisher.java:25) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy16.publish(Unknown Source) at com.atlassian.crowd.console.listener.StartupListener.contextInitialized(StartupListener.java:105) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:672) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1862) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:215) at java.util.zip.ZipFile.<init>(ZipFile.java:145) at java.util.jar.JarFile.<init>(JarFile.java:154) at java.util.jar.JarFile.<init>(JarFile.java:118) at com.atlassian.plugin.osgi.factory.transform.JarUtils.withJar(JarUtils.java:98) ... 50 more
Diagnosis
- Password reset for local accounts are not working
- Email address reset for owner/first user's account is not working
Embedded Crowd (Hipchat Server utilises Crowd to handle directory integration) process in the server is running as normal. You can check via this command:
ps aux | grep crowd
Sample output of running Crowd process:
admin@server:~$ ps aux | grep crowd root 7427 0.4 8.9 2620548 681116 ? Sl 01:48 2:06 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/opt/atlassian/crowd/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms128m -Xmx1024m -XX:MaxPermSize=256m -Dfile.encoding=UTF-8 -Datlassian.org.osgi.framework.bootdelegation=weblogic,weblogic.*,META-INF.services,com.yourkit,com.yourkit.*,com.chronon,com.chronon.*,org.jboss.byteman,org.jboss.byteman.*,com.jprofiler,com.jprofiler.*,org.apache.xerces,org.apache.xerces.*,org.apache.xalan,org.apache.xalan.*,org.apache.xml.serializer,sun.*,com.sun.xml.bind.v2,com.sun.xml.internal.bind.v2,com.icl.saxon,com_cenqua_clover,com.cenqua.clover,com.cenqua.clover.*,com.atlassian.clover,com.atlassian.clover.*,com.sun.jndi.ldap -Djava.endorsed.dirs=/opt/atlassian/crowd/apache-tomcat/endorsed -classpath /opt/atlassian/crowd/apache-tomcat/bin/bootstrap.jar:/opt/atlassian/crowd/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/crowd/apache-tomcat -Dcatalina.home=/opt/atlassian/crowd/apache-tomcat -Djava.io.tmpdir=/opt/atlassian/crowd/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start admin 16691 0.0 0.0 8080 916 pts/1 S+ 09:43 0:00 grep --color=auto crowd
Cause
At one point, Crowd cache in Hipchat Server got corrupted. This could happen after an upgrade that include a Crowd update (potentially prior to 2.2.3 where we introduced Crowd 2.11.2)
Resolution
Snapshot the server before making any changes as precautionary measure
SSH into the server and run the followings:
sudo dont-blame-hipchat
/etc/init.d/crowd stop
mv /etc/crowd/caches /etc/crowd/caches.bak
/etc/init.d/crowd start
The commands above will stop the Crowd service, rename Crowd cache file to something else and start up the service again. User can then attempt a login. If problem persists, send in /var/log/hipchat/atlassian-crowd.log and /var/log/hipchat/coral.log to Hipchat Support.