"Internal Server Error" after Bitbucket server upgrade

Related content

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

After upgrading Bitbucket on the UI you might see Internal Server error, something like this:

Environment

Bitbucket Server/DC

Diagnosis

In the atlassian-bitbucket.log, you would see the below stack trace:

2022-10-11 11:16:08,021 ERROR [http-nio-7990-exec-4] *JRBB5Dx676x89787x0 3q2bbo xx.xx.xx.xxx,xxx.x.x.x "GET /mvc/error500 HTTP/1.1" c.a.s.i.web.ErrorPageController There was an unhandled exception loading [/dashboard]
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is com.google.template.soy.tofu.SoyTofuException: When evaluating "to_json($ij.principal)": While computing function "to_json($ij.principal)": null
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)

	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
	at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:84)
	at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
	at java.base/java.lang.Thread.run(Thread.java:829)
	... 291 frames trimmed
Caused by: com.google.template.soy.tofu.SoyTofuException: When evaluating "to_json($ij.principal)": While computing function "to_json($ij.principal)": null
	at bitbucket.internal.app.dashboard.page(dashboard.soy:26)
	at com.google.template.soy.tofu.internal.BaseTofu.renderMainHelper(BaseTofu.java:369)
	at com.google.template.soy.tofu.internal.BaseTofu.renderMain(BaseTofu.java:322)

	at com.atlassian.soy.springmvc.SoyView.render(SoyView.java:50)
	at com.atlassian.stash.internal.web.soy.StashSoyViewResolver$1.render(StashSoyViewResolver.java:34)
	... 36 common frames omitted
Caused by: com.google.template.soy.sharedpasses.render.RenderException: When evaluating "to_json($ij.principal)": While computing function "to_json($ij.principal)": null
	at bitbucket.internal.app.dashboard.page(dashboard.soy:26)
	at com.google.template.soy.sharedpasses.render.RenderVisitor.eval(RenderVisitor.java:742)
	at com.google.template.soy.sharedpasses.render.RenderVisitor.visitPrintNode(RenderVisitor.java:248)
	at com.google.template.soy.soytree.AbstractSoyNodeVisitor.visit(AbstractSoyNodeVisitor.java:87)

	at com.google.template.soy.soytree.AbstractSoyNodeVisitor.visit(AbstractSoyNodeVisitor.java:56)
	at com.google.template.soy.basetree.AbstractNodeVisitor.exec(AbstractNodeVisitor.java:40)
	at com.google.template.soy.tofu.internal.BaseTofu.renderMainHelper(BaseTofu.java:366)
	... 46 common frames omitted
Caused by: com.google.template.soy.sharedpasses.render.RenderException: While computing function "to_json($ij.principal)": null
	at com.google.template.soy.sharedpasses.render.EvalVisitor.computeFunctionHelper(EvalVisitor.java:673)
	at com.google.template.soy.sharedpasses.render.EvalVisitor.visitFunctionNode(EvalVisitor.java:653)

	at com.google.template.soy.sharedpasses.render.RenderVisitor.eval(RenderVisitor.java:739)
	... 66 common frames omitted
Caused by: java.lang.NullPointerException: null
	at com.google.common.base.CharMatcher.replaceFrom(CharMatcher.java:719)
	at com.atlassian.stash.internal.web.soy.functions.ToJsonFunction.safeJson(ToJsonFunction.java:56)

	at com.atlassian.soy.impl.modules.CompositeFunctionAdaptor.computeForJava(CompositeFunctionAdaptor.java:38)
	at com.google.template.soy.sharedpasses.render.EvalVisitor.computeFunctionHelper(EvalVisitor.java:670)
	... 75 common frames omitted

The Bitbucket access logs will also show 500 response code:

atlassian-bitbucket-access.log 
xx.xx.xx.xxx,xxx.x.x.x | https | o@PYL7AZx753x42x2 | xxxx | 2022-10-04 12:33:13,760 | "GET /rest/jira-integration/latest/issues/assignedToMe HTTP/1.1" | "https://bitbucket-test.ads.vhv.de/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" | 500 | 0 | 314 | - | 69 | 8gudl2 |

Cause

It is possible that during the upgrade, some of the user-installed plugins might not remain compatible or were not upgraded.

Solution

Disable plugins during Bitbucket start up, for which you'll need to remove the plugin folder from the shared file system to another temporary location.

Steps

  1.  Shut down Bitbucket on all nodes
  2. On the filesystem, go to the $BITBUCKET_HOME/shared/plugins directory

You'll see something like this:

ubuntu@ip-xx-x-xx-xxx:/var/atlassian/application-data/bitbucket/shared/plugins$ cd installed-plugins/
ubuntu@ip-xx-x-xx-xxx:/var/atlassian/application-data/bitbucket/shared/plugins/installed-plugins$ ls
plugin.12570867300720346.advanced-logging-1.0.8-server.jar

    3. Move the installed-plugins}} folder to another location outside the $BITBUCKET_HOME/shared/plugins directory 
    4. Restart Bitbucket

Please ensure that you do this on Bitbucket test instance first to avoid causing any issues in the Production Bitbucket instance.

Java issue

In case you see the below issue while starting Bitbucket:

atlbitbucket@lvt-bitbct-t01:~> /opt/webapp.dist.lin/app/bitbucket/current/bin/start-bitbucket.sh --no-search
JRE_HOME '/usr/lib64/jvm/jre-1.8.0' is Java
Atlassian Bitbucket requires Java 8 to run
Set JRE_HOME to a Java 8 JRE and try again

Startup has been aborted

ensure that the Java path is defined correctly:

When using the embedded JRE that ships with Bitbucket, the only line that would need to be modified in the set-jre-home.sh is the last line of the first section, as shown below. 

# Uncomment and edit the line below to define JRE_HOME.
# If you use the Bitbucket installer, this value will be automatically set to point to the bundled JRE.
# Once a value is set here, existing JRE_HOME and JAVA_HOME values set in the en
vironment will be overridden and ignored.
JRE_HOME="/opt/atlassian/bitbucket/7.21.0/jre"


Here's an example from a working environment for comparison:

# Uncomment and edit the line below to define JRE_HOME.
# If you use the Bitbucket installer, this value will be automatically set to point to the bundled JRE.
# Once a value is set here, existing JRE_HOME and JAVA_HOME values set in the environment will be overridden and ignored.
JRE_HOME="/opt/atlassian/bitbucket/7.21.0/jre"

# Otherwise, use an existing installed JDK defined by JAVA_HOME
if [ -z "$JRE_HOME" ]; then
    if [ -n "$JAVA_HOME" ] && [ -x "$JAVA_HOME/jre/bin/java" ]; then
        # If JAVA_HOME points to a valid JDK, use its JRE
        JRE_HOME="$JAVA_HOME/jre"
    elif [ -n "$JAVA_HOME" ] && [ -x "$JAVA_HOME/bin/java" ]; then
        # If JAVA_HOME appears to point to a JRE and not a JDK use this anyway
        JRE_HOME="$JAVA_HOME"
    fi
fi

# By this line, JRE_HOME should be defined
if [ -z "$JRE_HOME" ]; then
    echo "Neither the JAVA_HOME nor the JRE_HOME environment variable is defined"
    echo "Edit set-jre-home.sh and define JRE_HOME"
    return 1
fi

if [ -n "$JRE_HOME" ] && [ -x "$JRE_HOME/bin/java" ]; then
    # Found java executable in JRE_HOME
    JAVA_BINARY="$JRE_HOME/bin/java"
    JAVA_VERSION=$("$JAVA_BINARY" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    if [ "$JAVA_VERSION" \< "1.8" ]; then
        echo "JRE_HOME '$JRE_HOME' is Java $JAVA_VERSION"
        echo "Atlassian Bitbucket requires Java 8 to run"
        echo "Set JRE_HOME to a Java 8 JRE and try again"
        return 1
    fi
else
    # If JRE_HOME/bin/java is not an executable, JRE_HOME is not valid
    echo "The JRE_HOME environment variable is not defined correctly"
    echo "This environment variable is needed to run this program"
    echo "Edit set-jre-home.sh and define JRE_HOME"
    return 1
fi

# This ensures that JAVA_HOME and JRE_HOME are consistent so that
# both Bitbucket and the bundled search server use the same JVM.
JAVA_HOME=$JRE_HOME

# If we make it here, the Java environment looks good
export JRE_HOME
export JAVA_HOME
export JAVA_BINARY
export JAVA_VERSION


If you/your team have a system installed/managed Java that you use, this path should be used in the set-jre-home.sh file to point to your installed version.  This would also explain why there is no Java at that location as the installer will not include the embedded JRE if Java is found via the JAVA_HOME or JRE_HOME environment variables if defined.

If started as a service, then the Java environment variables could be defined in the unit file, possibly other locations depending on how the admins manage the system.

In case the above is done correctly and you're still facing the same issue, reinstalling Java in your environment should work.



Last modified on Mar 9, 2023

Was this helpful?

Yes
No
Provide feedback about this article

Related content

Powered by Confluence and Scroll Viewport.