Problem

The problem symptoms may include —but are not limited to— one or many of the followings: 

• No users (local or directory) able to login to Hipchat Server. Users get this message in the UI when attempting a login:

  "An unknown error occurred. Please try again in a few minutes or reset your password."

• HTTP Status 404 - Could not find servlet for: /crowd/plugins/servlet/embedded-crowd/directories/list is shown when you go to Server admin > Directory
   

• The following appears in the /var/log/hipchat/atlassian-crowd.log

  2015-10-15 06:05:05,263 localhost-startStop-1 ERROR [atlassian.plugin.loaders.ScanningPluginLoader] Because of the following exception:
  java.lang.IllegalArgumentException: File must be a jar: /etc/crowd/caches/transformed-plugins/hipchat-server-crowd-plugin-0.10_1441096490000.jar
  	at com.atlassian.plugin.osgi.factory.transform.JarUtils.withJar(JarUtils.java:103)
  	at com.atlassian.plugin.osgi.factory.transform.JarUtils.getManifest(JarUtils.java:38)
  	at com.atlassian.plugin.osgi.factory.OsgiPluginUninstalledHelper.isRemotePlugin(OsgiPluginUninstalledHelper.java:67)
  	at com.atlassian.plugin.osgi.factory.OsgiPlugin.getInstallationMode(OsgiPlugin.java:154)
  	at com.atlassian.plugin.parsers.XmlDescriptorParser.configurePlugin(XmlDescriptorParser.java:113)
  	at com.atlassian.plugin.osgi.factory.OsgiPluginFactory.create(OsgiPluginFactory.java:213)
  	at com.atlassian.plugin.loaders.ScanningPluginLoader.deployPluginFromUnit(ScanningPluginLoader.java:155)
  	at com.atlassian.plugin.loaders.ScanningPluginLoader.loadAllPlugins(ScanningPluginLoader.java:89)
  	at com.atlassian.plugin.loaders.PermissionCheckingPluginLoader.loadAllPlugins(PermissionCheckingPluginLoader.java:24)
  	at com.atlassian.plugin.manager.DefaultPluginManager.earlyStartup(DefaultPluginManager.java:454)
  	at com.atlassian.plugin.manager.DefaultPluginManager.init(DefaultPluginManager.java:431)
  	at com.atlassian.crowd.plugin.CrowdPluginManager.init(CrowdPluginManager.java:69)
  	at com.atlassian.crowd.plugin.CrowdPluginManager.onApplicationStartedEvent(CrowdPluginManager.java:82)
  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:606)
  	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
  	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48)
  	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
  	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107)
  	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
  	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
  	at com.atlassian.crowd.util.TransactionalEventPublisher.publish(TransactionalEventPublisher.java:25)
  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:606)
  	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
  	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
  	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
  	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
  	at com.sun.proxy.$Proxy16.publish(Unknown Source)
  	at com.atlassian.crowd.console.listener.StartupListener.contextInitialized(StartupListener.java:105)
  	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
  	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
  	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
  	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
  	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
  	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
  	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:672)
  	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1862)
  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  	at java.lang.Thread.run(Thread.java:745)
  Caused by: java.util.zip.ZipException: error in opening zip file
  	at java.util.zip.ZipFile.open(Native Method)
  	at java.util.zip.ZipFile.<init>(ZipFile.java:215)
  	at java.util.zip.ZipFile.<init>(ZipFile.java:145)
  	at java.util.jar.JarFile.<init>(JarFile.java:154)
  	at java.util.jar.JarFile.<init>(JarFile.java:118)
  	at com.atlassian.plugin.osgi.factory.transform.JarUtils.withJar(JarUtils.java:98)
  	... 50 more

Diagnosis

• Password reset for local accounts are not working
• Email address reset for owner/first user's account is not working

• Embedded Crowd (Hipchat Server utilises Crowd to handle directory integration) process in the server is running as normal. You can check via this command:

  ps aux | grep crowd
  

  Sample output of running Crowd process:

  admin@server:~$ ps aux | grep crowd
  root      7427  0.4  8.9 2620548 681116 ?      Sl   01:48   2:06 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/opt/atlassian/crowd/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms128m -Xmx1024m -XX:MaxPermSize=256m -Dfile.encoding=UTF-8 -Datlassian.org.osgi.framework.bootdelegation=weblogic,weblogic.*,META-INF.services,com.yourkit,com.yourkit.*,com.chronon,com.chronon.*,org.jboss.byteman,org.jboss.byteman.*,com.jprofiler,com.jprofiler.*,org.apache.xerces,org.apache.xerces.*,org.apache.xalan,org.apache.xalan.*,org.apache.xml.serializer,sun.*,com.sun.xml.bind.v2,com.sun.xml.internal.bind.v2,com.icl.saxon,com_cenqua_clover,com.cenqua.clover,com.cenqua.clover.*,com.atlassian.clover,com.atlassian.clover.*,com.sun.jndi.ldap -Djava.endorsed.dirs=/opt/atlassian/crowd/apache-tomcat/endorsed -classpath /opt/atlassian/crowd/apache-tomcat/bin/bootstrap.jar:/opt/atlassian/crowd/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/crowd/apache-tomcat -Dcatalina.home=/opt/atlassian/crowd/apache-tomcat -Djava.io.tmpdir=/opt/atlassian/crowd/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
  admin    16691  0.0  0.0   8080   916 pts/1    S+   09:43   0:00 grep --color=auto crowd

Cause

At one point, Crowd cache in Hipchat Server got corrupted. This could happen after an upgrade that include a Crowd update (potentially prior to 2.2.3 where we introduced Crowd 2.11.2)

Resolution

Snapshot the server before making any changes as precautionary measure

SSH into the server and run the followings:

sudo dont-blame-hipchat
/etc/init.d/crowd stop
mv /etc/crowd/caches /etc/crowd/caches.bak 
/etc/init.d/crowd start

The commands above will stop the Crowd service, rename Crowd cache file to something else and start up the service again. User can then attempt a login. If problem persists, send in /var/log/hipchat/atlassian-crowd.log and /var/log/hipchat/coral.log to Hipchat Support.