Problem

The Fisheye/Crucible User Interface does not load or the users are unable to log in.

One of the following error is shown on the browser page and/or in the atlassian-fisheye-YYYY-MM-DD.log file.

Stack trace - option #1

javax.servlet.ServletException: javax.servlet.ServletException: javax.servlet.jsp.JspException: javax.el.ELException:
  com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: 
  Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

Stack trace - option #2

ERROR [FelixDispatchQueue ] com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager FelixOsgiContainerManager$BundleRegistration-frameworkEvent - Framework error in bundle com.atlassian.activeobjects.activeobjects-plugin
org.osgi.framework.ServiceException: Service factory exception: org.springframework.osgi.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.sal.api.transaction.TransactionTemplate)] unavailable
  at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:311)
  at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:221)
...
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: org.springframework.osgi.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.sal.api.transaction.TransactionTemplate)] unavailable
  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
... 
Caused by: org.springframework.osgi.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.sal.api.transaction.TransactionTemplate)] unavailable
  at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:419)

Cause

There are four possible causes to this behaviour:

1. Multiple copies of Applinks and/or SAL plugin JARs are present in the Fisheye installation. These JARs can be checked under the following directories:
   
   • <FISHEYE_INST>/var/plugins/bundled/
   • <FISHEYE_INST>/var/plugins/user/
   • <FishEye Installation Directory>/lib/
2. The application initialises slowly and the plugin start up times-out.
3. A corrupt plugin JAR (or more than one) and/or corrupt cache status in the installation.
4. An unsupported version of Java has recently been installed.

Resolution

According to the possible causes the following resolutions are suggested:

1. Remove duplicate plugin JAR files
   1. Remove the multiple copies of plugin JAR files from the following directories ensuring that there is only a single copy of each JAR file inside the <FISHEYE_INST>/var/plugins/bundled/ folder:
      1. <FISHEYE_INST>/var/plugins/user/
      2. <FishEye Installation Directory>/lib/

      The list of JARs present inside <FISHEYE_INST>/var/plugins/bundled/ should be:

      applinks-plugin-x.x.jar
      sal-fisheye-plugin-x.x.jar (should ideally be the same version bundled with the FishEye release)

      The list of JARs present under lib/ should be:

      applinks-api-x.x.jar 
      applinks-host-x.x.jar
      applinks-spi-x.x.jar
      sal-spring-x.x.jar
      sal-spi-x.x.jar
      sal-api-x.x.jar

   2. Make sure no other directories that contain JARs were accidentally copied to incorrect locations, e.g. the following directory locations are not valid:
      1. <FISHEYE_INST>/lib
      2. <FISHEYE_INST>/var/lib
         
         
2. Increase the plugin timeout setting
   

   Increase the timeout by providing the following additional JVM parameter, the following will increase the timeout to 300 seconds:

   -Datlassian.plugins.enable.wait=300

   The default plugin timeout is 60 seconds. 

    

3. Remove Corrupt JAR files/JAR cache
   

   Check the content of the <FISHEYE_INST>/var/plugins/config/pluginstate.properties file to see whether there are any plugins reported to be in the "false" (disabled) state. If such entries are present as shown in the following snippet:

   com.atlassian.plugins.rest.atlassian-rest-module=false
   com.atlassian.sal.fisheye=false

    

   1. Shutdown Fisheye

   2. Backup the file and remove all lines that are not commented

   3. Start Fisheye

   
   

   Otherwise, remove all plugins and the plugin cache:

    
   1. Shutdown Fisheye.
   2. Backup and delete the contents of the following folders:
      1. <FISHEYE_INST>/var/plugins 
      2. <FISHEYE_INST>/var/osgi-cache
   3. Start Fisheye (this will recreate the above folders minus any individually downloaded plugins).
   
   
4. Ensure Correct Java Version Is Referenced By Fisheye/Crucible
   
   1. Ensure that an appropriate Java JVM is installed. See Supported Platforms for more information.
   2. Ensure that java is in the PATH, or that the JAVA_HOME environment variable is set and check that the correct Java is referenced in wrapper.conf if running Fisheye as a service on Windows.
      
      
5. Delete OSGI cache by removing:
   1. FISHEYE_INST/cache/plugins/osgi-cache
   2. <FishEye Installation Directory>/cache/plugins/osgi-cache

Further Troubleshooting

If the problem persists, please contact Atlassian Support and attach the following information to the Support ticket:

1. A list of the JARs in the following locations:
   1. <FISHEYE_INST>/var/plugins/bundled/ 
   2. <FISHEYE_INST>/var/plugins/user/
   3. <FishEye Installation Directory>/lib/
2. A Zip file of the <FISHEYE_INST>/var/log directory after:

   1. Uncommenting the following block of lines from your /log4j.xml file

      <!--<logger name="com.atlassian.plugin">-->
              <!--<level value="debug"/>-->
              <!--<appender-ref ref="debuglog" />-->
              <!--<appender-ref ref="console" />-->
      <!--</logger>-->
      

   2. Restarting Fisheye.
   3. Reproducing the error.
      
      This will log a lot of plugin related debug messages in the log file which will hopefully provide us with enough information to deduce the cause of the problem.