Fix gadget titles showing as __MSG_gadget in Jira server

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Problem

After upgrading to Jira 7.1.x+, gadget titles appear as __MSG_gadget.xxxxxx as per the below screenshot:

Diagnosis

Environment

  • Affects Jira 7.1.x and higher

One or more of the following conditions may be true:

  • iptables or firewalld has been configured to forward packets from 80 to 8080 and/or 443 to 8443
  • From the Jira server itself, Jira is unable to communicate with itself via its Base URL (Run curl -v <base_url> to verify)
  • Run SSLPoke from the Jira server itself and see if it returns successfully
  • Additionally run the httpclienttest from the Jira server itself to confirm if the SSL configuration is okay, as this will verify if you're affected by  JRA-47568 - Getting issue details... STATUS
  • There are application links configured in Jira but they seem to be having some issue
  • Outbound proxy is configured and Base URL is not whitelisted
  • Using a plugin to prevent anonymous access
  • Reverse proxy information is missing from the HTTP connector when using a HTTP proxy
  • When using an AJP connector, there's no retry element in the ProxyPass directive at the proxy server

Cause

In Jira 7.1.x+ there's a change in gadgets. Now a number of charts are generated using JavaScript in the browser rather than server-side with Java. This has brought significant performance improvements as the server offloads processing to the browsers and results in less CPU usage. Part of the change has modified the way Jira previously operated, and now it is required that Jira is able to connect to itself.

When accessing a dashboard, the gadgets serve a response which is processed by the client (i.e. the browser) to generate gadgets. Whilst this happens, the server also connects to itself to provide the gadgets' translated strings. Jira uses the URL the instance is connected to, so the Jira instance needs to be able to access itself.

Without Usage of Proxy

  1. User sends request towards Jira on loading the Dashboard
  2. Jira will perform a call towards itself, through the gadget URL
  3. Jira replies with the completed page + gadgets' translated strings
With Usage of Proxy

1. User sends request towards Jira on loading the Dashboard
2. Proxy forwards this request to Jira
3. Jira will perform a call back to itself through the gadget URL to get the gadgets' title, performing request to Proxy
4. Proxy forwards this request back to Jira itself to provide the gadgets' translated strings
5. Jira replies gadget's translated strings to proxy
6. Proxy forwards this accordingly to Jira
7. Jira replies with the completed page to proxy
8. Proxy forwards this accordingly to user

If it's not able to do so this error will occur. Some known causes are:

  • Basic or certificate-based auth on a proxy layer that Jira is behind
  • The domain name is not resolvable
  • Jira is served on an IP that's in a separate restricted subnet to the server Jira is running on
  • Jira is served with a self-signed certificate that doesn't exist in its own trust store

The recommended best practice is to have Jira served on one domain name only. This can have different IPs (for example internal and external) configured by different DNS, however will only resolve to one domain. For example https://Jira.atlassian.com has an external IP of 131.103.28.11 (mapped to by external DNS) and an internal IP of 192.168.1.2 (mapped with an internal DNS). It has one certificate for that FQDN, and all users access it on that domain only.

(info) Additionally, there are some environmental-specific causes to this problem. Please see below for further information.

Resolution

Jira Applications with SSL Configured

The following appears in the atlassian-Jira.log file:

2016-02-26 14:22:28,843 http-bio-8443-exec-11 ERROR admin 862x395x1 m32d6k xx.xx.xx.xx /secure/MyJiraHome.jspa [c.a.g.r.internal.http.HttpClientFetcher] Unable to retrieve response
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

Cause

This is a known problem issue the certificate is not trusted by the Java trust store.

If using SNI on the certificate, we have an open bug for this being tracked under JRA-47568 - Getting issue details... STATUS . A fix for this bug is scheduled for a future release of Jira. Currently, the most simple workaround for this issue is to use a non-SNI SSL certificate for your Jira instance. You can confirm if you are affected by this bug by using httpclienttest in conjunction with the SSLPoke tool linked below. If the SSLPoke test works without error and the httpclienttest reports an exception, you are experiencing the bug as above.


Resolution

  1. Jira's SSL certificate needs to be imported to its own trust store in order for Jira able to trust itself here. Follow our resolution in our Unable to connect to SSL services due to "PKIX Path Building Failed" error knowledge base article to resolve the problem.
  2. Additionally verify the configuration with httpclienttest as described in the readme of that repository.

Jira applications with port forwarding configured

Applies to environments that are using port forwarding, such as iptables or firewalld:

Cause

If iptables or firewalld has been configured to forward packets from 80 to 8080 and/or 443 to 8443, you may see gadget titles be broken and show as __MSG_gadget.xxxxxxx.  This can happen if the rules are configured only to forward packets from external sources.  

Resolution

You need to create a rule both for packets coming from an external IP sources and for local packets:

Using iptables

iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080
 -A OUTPUT -o lo -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8443 
-A OUTPUT -d XX.XXX.XXX.XX/32 -o lo -p tcp -m tcp --dport 80 -j DNAT --to-destination XX.XXX.XXX.XX:8080
-A OUTPUT -d XX.XXX.XXX.XX/32 -o lo -p tcp -m tcp --dport 443 -j DNAT --to-destination XX.XXX.XXX.XX:8443

Replace XX.XXX.XXX.XX with the destination's IP Address

Using Firewalld

Configure external port forwarding for the Firewall Zone

</description>
  <service name="dhcpv6-client"/>
  <service name="ssh"/>
  <service name="http"/>
  <service name="https"/>
  <port protocol="tcp" port="80"/>
  <port protocol="tcp" port="443"/>
  <port protocol="tcp" port="8080"/>
  <port protocol="tcp" port="8443"/>
  <forward-port to-port="8443" protocol="tcp" port="443"/>
  <forward-port to-port="8080" protocol="tcp" port="80"/>
</zone>

Create additional direct rule for local traffic

firewall-cmd --permanent --direct --add-rule ipv4 nat OUTPUT 0 -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080
firewall-cmd --permanent --direct --add-rule ipv4 nat OUTPUT 0 -p tcp -o lo --dport 443 -j REDIRECT --to-ports 8443

The following appears in the atlassian-Jira.log file:

2016-02-26 13:48:12,564 http-nio-2186-exec-23 ERROR      [o.a.c.c.C.[.[localhost].[/JSP-262186].[servlet-module-container-servlet]] Servlet.service() for servlet [servlet-module-container-servlet] in context with path [/JSP-262186] threw exception
com.atlassian.templaterenderer.RenderingException: org.apache.velocity.exception.MethodInvocationException: 
Invocation of method 'getText' in  class com.sun.proxy.$Proxy12830 threw exception java.lang.NullPointerException at com/atlassian/applinks/oauth/auth/outbound_nonapplinks.vm[line 52, column 44]
	at com.atlassian.templaterenderer.velocity.one.six.internal.VelocityTemplateRendererImpl.render(VelocityTemplateRendererImpl.java:109)
	at sun.reflect.GeneratedMethodAccessor1986.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	...
Caused by: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getText' in  class com.sun.proxy.$Proxy12830 threw exception java.lang.NullPointerException at com/atlassian/applinks/oauth/auth/outbound_nonapplinks.vm[line 52, column 44]
	at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:337)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:284)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:262)
	at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:342)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
	at org.apache.velocity.Template.merge(Template.java:235)
	at com.atlassian.templaterenderer.velocity.one.six.internal.VelocityTemplateRendererImpl.render(VelocityTemplateRendererImpl.java:100)
	... 234 more
Caused by: java.lang.NullPointerException
	at com.atlassian.Jira.i18n.AbstractI18nResolver.getText(AbstractI18nResolver.java:24)
	at sun.reflect.GeneratedMethodAccessor1918.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	...

Cause

There appears to be a broken application link configured in Jira. A broken application link is defined as an application linked to another application, e.g. Confluence that is non-functional and inaccessible. Likely causes include network issues, the other application being offline or a corrupted application link configuration. Or the Application Link may have been established when a trusted certificate did not exist, so it failed to create properly.

To check for broken application links, go to Administration > Applications > Application Links and check if any of the links are flagged as Offline, for example:

Alternatively, you can also type the letter G twice while logged in as an admin and type in 'Application links' in the admin search box.

You may want to refer to the Application Links Troubleshooting Guide for more information on how to fix broken application links.

Resolution

Delete the broken application link and recreate it, or follow the Application Links Troubleshooting Guide to fix the broken application link.

 Jira applications with a broken Proxy configuration

Cause

Jira is unable to communicate with itself due to a Web Proxy not able to correctly resolve the Jira Server.

Resolution

Add the Jira Server's hostname to the -Dhttp.nonProxyHosts and -Dhttps.nonProxyHosts arguments as in How to Configure an Outbound HTTP and HTTPS Proxy for Jira applications.

Jira applications using an AJP Connector with Apache Web Server

Diagnosis

You may see messages in the proxy logs like:

[Sat Feb 23 15:08:42.168550 2019] [proxy:error] [pid 22708:tid 140608035321600] AH00940: AJP: disabled connection for (x.x.x.x)
[Sat Feb 23 15:08:46.079259 2019] [proxy:error] [pid 22710:tid 140608060499712] (111) Connection refused: AH00957: AJP: attempt to connect to x.x.x.x:8009 (x.x.x.x) failed
[Sat Feb 23 15:08:46.079298 2019] [proxy:error] [pid 22710:tid 140608060499712] AH00959: ap_proxy_connect_backend disabling worker for (x.x.x.x) for 60s

You may also find that other tests outlined on this page (such as SSLPoke) are successful. AJP provides proxy information across the connector, so the proxy parameters should not be required.

Cause

When the reverse proxy encounters an error communicating with JIRA (particularly if it's on a remote machine) the backend worker may be disabled (which in turn disrupts subsequent connections

Resolution

You may be able to reduce the occurence of this by adding retry=0 to your proxyPass directive:

<Location />
  ProxyPass / ajp://localhost:8009/ retry=0
  ProxyPassReverse / ajp://localhost:8009/ retry=0
</Location>

Restart Apache for the changes to take effect


Jira configured with Web Proxy in front with BasicAuth password restriction

Cause

Jira is unable to communicate with itself due to a Web Proxy requiring authentication.

Bug ticket raise JRA-61273 - Getting issue details... STATUS

Resolution for Apache

Add the Server IP itself for access without password:

Order Allow,Deny
Allow from <hostname>
Satisfy Any

Resolution for IIS

Use  'Anonymous access' instead of selecting 'Basic authentication' or 'Integrated Windows authentication'. For more details steps, please refer Integrating Jira applications with IIS documentation under Troubleshooting section.

Resolution for Shibboleth

If using Shibboleth authentication,

  1. leave the /rest URLs outside the shibboleth authentication.
  2. /rest URL must not be shibboleth protected.
  3. Example :
<LocationMatch "^(/rest)/[^/]+">
AuthType None
Require all granted
</LocationMatch>

<LocationMatch "/">
AuthType Shibboleth
ShibRequireSession On
require valid-user
ShibUseEnvironment On
Order allow,deny
Allow from all
</LocationMatch>


Anonymous requests are blocked by Jira

Cause

A known cause is Jira uses Prevent Anonymous Access plugin that blocks all anonymous requests. This blocks the request to /rest/gadgets/1.0/g/messagebundle..., hence Jira can't display the gadget titles.

Resolution

The plugin above comes with a whitelist page. Whitelisting the following endpoint will allow Jira to get the gadget titles correctly:


/rest/gadgets/1.0/g/messagebundle.+

Jira restart is required


Web Fragment Finder App

Cause

The logs show LittleProxy trying to talk back to com.wittified.webfragment-finder which from the Web Fragment Finder add-on:

2017-04-14 06:42:16,602 LittleProxy-ClientToProxyWorker-0 WARN b757 758x82771x1 dh8gtn 62.153.237.114 /rest/plugins/1.0/com.wittified.webfragment-finder-key [i.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x3e6311c8, /127.0.0.1:33660 => /127.0.0.1:8888]
2017-04-14 06:42:16,602 LittleProxy-ClientToProxyWorker-1 WARN b757 758x82771x1 dh8gtn 62.153.237.114 /rest/plugins/1.0/com.wittified.webfragment-finder-key [i.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x9b85bea9, /127.0.0.1:33662 => /127.0.0.1:8888]

Resolution

  1. Disable Web Fragment Finder Add-On
  2. Restart Jira

SSL certificate using SAN

Applies to Jira 7.4 and above where Jira is running over SSL with SAN.

Cause

Jira fails to communicate with itself due to an Apache HttpClient bug. The following error is seen in the logs:

2017-10-10 17:04:06,623 HealthCheck:thread-5 ERROR ServiceRunner     [c.a.t.j.healthcheck.support.BaseUrlHealthCheck] An error occurred when performing the Base URL healthcheck:
java.lang.ClassCastException: [B cannot be cast to java.lang.String
    at org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309)

Workaround

  1. Temporarily use an SSL certificate that doesn't use SAN
  2. Temporarily run Jira over HTTP

Please watch  JRASERVER-65595 - Getting issue details... STATUS  for a permanent fix.



DescriptionAfter upgrading to Jira 7.1.x+, gadget titles appear as __MSG_gadget.xxxxxx
ProductJira
PlatformServer
Last modified on Jan 8, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.