Fix gadget titles showing as __MSG_gadget in Jira server
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.
After upgrading to Jira 7.1.x+, gadget titles appear as
__MSG_gadget.xxxxxx as per the below screenshot:
- Affects Jira 7.1.x and higher
One or more of the following conditions may be true:
- iptables or firewalld has been configured to forward packets from 80 to 8080 and/or 443 to 8443
- From the Jira server itself, Jira is unable to communicate with itself via its Base URL (Run curl -v <base_url> to verify)
- Run SSLPoke from the Jira server itself and see if it returns successfully
- Additionally run the httpclienttest from the Jira server itself to confirm if the SSL configuration is okay, as this will verify if you're affected by - JRA-47568Getting issue details... STATUS
- There are application links configured in Jira but they seem to be having some issue
- Outbound proxy is configured and Base URL is not whitelisted
- Using a plugin to prevent anonymous access
- Reverse proxy information is missing from the HTTP connector when using a HTTP proxy
- When using an AJP connector, there's no
retryelement in the
ProxyPassdirective at the proxy server
When accessing a dashboard, the gadgets serve a response which is processed by the client (i.e. the browser) to generate gadgets. Whilst this happens, the server also connects to itself to provide the gadgets' translated strings. Jira uses the URL the instance is connected to, so the Jira instance needs to be able to access itself.
- User sends request towards Jira on loading the Dashboard
- Jira will perform a call towards itself, through the gadget URL
- Jira replies with the completed page + gadgets' translated strings
1. User sends request towards Jira on loading the Dashboard
2. Proxy forwards this request to Jira
3. Jira will perform a call back to itself through the gadget URL to get the gadgets' title, performing request to Proxy
4. Proxy forwards this request back to Jira itself to provide the gadgets' translated strings
5. Jira replies gadget's translated strings to proxy
6. Proxy forwards this accordingly to Jira
7. Jira replies with the completed page to proxy
8. Proxy forwards this accordingly to user
If it's not able to do so this error will occur. Some known causes are:
- Basic or certificate-based auth on a proxy layer that Jira is behind
- The domain name is not resolvable
- Jira is served on an IP that's in a separate restricted subnet to the server Jira is running on
- Jira is served with a self-signed certificate that doesn't exist in its own trust store
The recommended best practice is to have Jira served on one domain name only. This can have different IPs (for example internal and external) configured by different DNS, however will only resolve to one domain. For example https://Jira.atlassian.com has an external IP of
126.96.36.199 (mapped to by external DNS) and an internal IP of
192.168.1.2 (mapped with an internal DNS). It has one certificate for that FQDN, and all users access it on that domain only.
Additionally, there are some environmental-specific causes to this problem. Please see below for further information.
Jira Applications with SSL Configured
The following appears in the
2016-02-26 14:22:28,843 http-bio-8443-exec-11 ERROR admin 862x395x1 m32d6k xx.xx.xx.xx /secure/MyJiraHome.jspa [c.a.g.r.internal.http.HttpClientFetcher] Unable to retrieve response javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
This is a known problem issue the certificate is not trusted by the Java trust store.
If using SNI on the certificate, we have an open bug for this being tracked under - JRA-47568Getting issue details... STATUS . A fix for this bug is scheduled for a future release of Jira. Currently, the most simple workaround for this issue is to use a non-SNI SSL certificate for your Jira instance. You can confirm if you are affected by this bug by using httpclienttest in conjunction with the SSLPoke tool linked below. If the SSLPoke test works without error and the httpclienttest reports an exception, you are experiencing the bug as above.
- Jira's SSL certificate needs to be imported to its own trust store in order for Jira able to trust itself here. Follow our resolution in our Unable to connect to SSL services due to "PKIX Path Building Failed" error knowledge base article to resolve the problem.
Additionally verify the configuration with httpclienttest as described in the readme of that repository.
Jira applications with port forwarding configured
Applies to environments that are using port forwarding, such as iptables or firewalld:
If iptables or firewalld has been configured to forward packets from 80 to 8080 and/or 443 to 8443, you may see gadget titles be broken and show as
__MSG_gadget.xxxxxxx. This can happen if the rules are configured only to forward packets from external sources.
You need to create a rule both for packets coming from an external IP sources and for local packets:
iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080
iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8443
iptables -t nat -A OUTPUT -d XX.XXX.XXX.XX/32 -o lo -p tcp -m tcp --dport 80 -j DNAT --to-destination XX.XXX.XXX.XX:8080 iptables -t nat -A OUTPUT -d XX.XXX.XXX.XX/32 -o lo -p tcp -m tcp --dport 443 -j DNAT --to-destination XX.XXX.XXX.XX:8443
Replace XX.XXX.XXX.XX with the destination's IP Address
Configure external port forwarding for the Firewall Zone
</description> <service name="dhcpv6-client"/> <service name="ssh"/> <service name="http"/> <service name="https"/> <port protocol="tcp" port="80"/> <port protocol="tcp" port="443"/> <port protocol="tcp" port="8080"/> <port protocol="tcp" port="8443"/> <forward-port to-port="8443" protocol="tcp" port="443"/> <forward-port to-port="8080" protocol="tcp" port="80"/> </zone>
Create additional direct rule for local traffic
firewall-cmd --permanent --direct --add-rule ipv4 nat OUTPUT 0 -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080 firewall-cmd --permanent --direct --add-rule ipv4 nat OUTPUT 0 -p tcp -o lo --dport 443 -j REDIRECT --to-ports 8443
Jira applications with a broken Application Link
The following appears in the
2016-02-26 13:48:12,564 http-nio-2186-exec-23 ERROR [o.a.c.c.C.[.[localhost].[/JSP-262186].[servlet-module-container-servlet]] Servlet.service() for servlet [servlet-module-container-servlet] in context with path [/JSP-262186] threw exception com.atlassian.templaterenderer.RenderingException: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getText' in class com.sun.proxy.$Proxy12830 threw exception java.lang.NullPointerException at com/atlassian/applinks/oauth/auth/outbound_nonapplinks.vm[line 52, column 44] at com.atlassian.templaterenderer.velocity.one.six.internal.VelocityTemplateRendererImpl.render(VelocityTemplateRendererImpl.java:109) at sun.reflect.GeneratedMethodAccessor1986.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ... Caused by: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getText' in class com.sun.proxy.$Proxy12830 threw exception java.lang.NullPointerException at com/atlassian/applinks/oauth/auth/outbound_nonapplinks.vm[line 52, column 44] at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:337) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:284) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:262) at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:342) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336) at org.apache.velocity.Template.merge(Template.java:235) at com.atlassian.templaterenderer.velocity.one.six.internal.VelocityTemplateRendererImpl.render(VelocityTemplateRendererImpl.java:100) ... 234 more Caused by: java.lang.NullPointerException at com.atlassian.Jira.i18n.AbstractI18nResolver.getText(AbstractI18nResolver.java:24) at sun.reflect.GeneratedMethodAccessor1918.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) ...
There appears to be a broken application link configured in Jira. A broken application link is defined as an application linked to another application, e.g. Confluence that is non-functional and inaccessible. Likely causes include network issues, the other application being offline or a corrupted application link configuration. Or the Application Link may have been established when a trusted certificate did not exist, so it failed to create properly.
To check for broken application links, go to Administration > Applications > Application Links and check if any of the links are flagged as Offline, for example:
Alternatively, you can also type the letter G twice while logged in as an admin and type in 'Application links' in the admin search box.
You may want to refer to the Application Links Troubleshooting Guide for more information on how to fix broken application links.
Delete the broken application link and recreate it, or follow the Application Links Troubleshooting Guide to fix the broken application link.
Jira applications with a broken Proxy configuration
Jira is unable to communicate with itself due to a Web Proxy not able to correctly resolve the Jira Server.
Add the Jira Server's hostname to the
-Dhttps.nonProxyHosts arguments as in How to Configure an Outbound HTTP and HTTPS Proxy for Jira applications.
Jira applications using an AJP Connector with Apache Web Server
You may see messages in the proxy logs like:
[Sat Feb 23 15:08:42.168550 2019] [proxy:error] [pid 22708:tid 140608035321600] AH00940: AJP: disabled connection for (x.x.x.x) [Sat Feb 23 15:08:46.079259 2019] [proxy:error] [pid 22710:tid 140608060499712] (111) Connection refused: AH00957: AJP: attempt to connect to x.x.x.x:8009 (x.x.x.x) failed [Sat Feb 23 15:08:46.079298 2019] [proxy:error] [pid 22710:tid 140608060499712] AH00959: ap_proxy_connect_backend disabling worker for (x.x.x.x) for 60s
You may also find that other tests outlined on this page (such as SSLPoke) are successful. AJP provides proxy information across the connector, so the proxy parameters should not be required.
When the reverse proxy encounters an error communicating with JIRA (particularly if it's on a remote machine) the backend worker may be disabled (which in turn disrupts subsequent connections
You may be able to reduce the occurence of this by adding
retry=0 to your
<Location /> ProxyPass / ajp://localhost:8009/ retry=0 ProxyPassReverse / ajp://localhost:8009/ retry=0 </Location>
Restart Apache for the changes to take effect
Jira configured with Web Proxy in front with BasicAuth password restriction
Jira is unable to communicate with itself due to a Web Proxy requiring authentication.
Resolution for Apache
Add the Server IP itself for access without password:
Order Allow,Deny Allow from <hostname> Satisfy Any
Resolution for IIS
Use 'Anonymous access' instead of selecting 'Basic authentication' or 'Integrated Windows authentication'. For more details steps, please refer Integrating Jira applications with IIS documentation under Troubleshooting section.
Resolution for Shibboleth
If using Shibboleth authentication,
- leave the /rest URLs outside the shibboleth authentication.
- /rest URL must not be shibboleth protected.
- Example :
<LocationMatch "^(/rest)/[^/]+"> AuthType None Require all granted </LocationMatch> <LocationMatch "/"> AuthType Shibboleth ShibRequireSession On require valid-user ShibUseEnvironment On Order allow,deny Allow from all </LocationMatch>
Anonymous requests are blocked by Jira
A known cause is Jira uses Prevent Anonymous Access plugin that blocks all anonymous requests. This blocks the request to
/rest/gadgets/1.0/g/messagebundle..., hence Jira can't display the gadget titles.
The plugin above comes with a whitelist page. Whitelisting the following endpoint will allow Jira to get the gadget titles correctly:
Jira restart is required
Web Fragment Finder App
The logs show LittleProxy trying to talk back to com.wittified.webfragment-finder which from the Web Fragment Finder add-on:
2017-04-14 06:42:16,602 LittleProxy-ClientToProxyWorker-0 WARN b757 758x82771x1 dh8gtn 188.8.131.52 /rest/plugins/1.0/com.wittified.webfragment-finder-key [i.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x3e6311c8, /127.0.0.1:33660 => /127.0.0.1:8888] 2017-04-14 06:42:16,602 LittleProxy-ClientToProxyWorker-1 WARN b757 758x82771x1 dh8gtn 184.108.40.206 /rest/plugins/1.0/com.wittified.webfragment-finder-key [i.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x9b85bea9, /127.0.0.1:33662 => /127.0.0.1:8888]
- Disable Web Fragment Finder Add-On
- Restart Jira
SSL certificate using SAN
Applies to Jira 7.4 and above where Jira is running over SSL with SAN.
Jira fails to communicate with itself due to an Apache HttpClient bug. The following error is seen in the logs:
2017-10-10 17:04:06,623 HealthCheck:thread-5 ERROR ServiceRunner [c.a.t.j.healthcheck.support.BaseUrlHealthCheck] An error occurred when performing the Base URL healthcheck: java.lang.ClassCastException: [B cannot be cast to java.lang.String at org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309)
- Temporarily use an SSL certificate that doesn't use SAN
- Temporarily run Jira over HTTP