Problem

After upgrading to Jira 7.1.x+, gadget titles appear as __MSG_gadget.xxxxxx as per the below screenshot:

Diagnosis

Environment

• Affects Jira 7.1.x and higher

One or more of the following conditions may be true:

• iptables or firewalld has been configured to forward packets from 80 to 8080 and/or 443 to 8443
• From the Jira server itself, Jira is unable to communicate with itself via its Base URL (Run curl -v <base_url> to verify)
• Run SSLPoke from the Jira server itself and see if it returns successfully
• Additionally run the httpclienttest from the Jira server itself to confirm if the SSL configuration is okay, as this will verify if you're affected by  JRA-47568 - Getting issue details... STATUS
• There are application links configured in Jira but they seem to be having some issue
• Outbound proxy is configured and Base URL is not whitelisted
• Using a plugin to prevent anonymous access
• Reverse proxy information is missing from the HTTP connector when using a HTTP proxy
• When using an AJP connector, there's no retry element in the ProxyPass directive at the proxy server
• Depending on the setup and environment, a CORS error might be happening when Jira is connecting to itself 

Cause

In Jira 7.1.x+ there's a change in gadgets. Now a number of charts are generated using JavaScript in the browser rather than server-side with Java. This has brought significant performance improvements as the server offloads processing to the browsers and results in less CPU usage. Part of the change has modified the way Jira previously operated, and now it is required that Jira is able to connect to itself.

When accessing a dashboard, the gadgets serve a response which is processed by the client (i.e. the browser) to generate gadgets. Whilst this happens, the server also connects to itself to provide the gadgets' translated strings. Jira uses the URL the instance is connected to, so the Jira instance needs to be able to access itself.

If it's not able to do so this error will occur. Some known causes are:

• Basic or certificate-based auth on a proxy layer that Jira is behind
• The domain name is not resolvable
• Jira is served on an IP that's in a separate restricted subnet to the server Jira is running on
• Jira is served with a self-signed certificate that doesn't exist in its own trust store

The recommended best practice is to have Jira served on one domain name only. This can have different IPs (for example internal and external) configured by different DNS, however will only resolve to one domain. For example https://Jira.atlassian.com has an external IP of 131.103.28.11 (mapped to by external DNS) and an internal IP of 192.168.1.2 (mapped with an internal DNS). It has one certificate for that FQDN, and all users access it on that domain only.

 Additionally, there are some environmental-specific causes to this problem. Please see below for further information.

Resolution

Jira Applications with SSL Configured

The following appears in the atlassian-Jira.log file:

2016-02-26 14:22:28,843 http-bio-8443-exec-11 ERROR admin 862x395x1 m32d6k xx.xx.xx.xx /secure/MyJiraHome.jspa [c.a.g.r.internal.http.HttpClientFetcher] Unable to retrieve response
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

Cause

This is a known problem issue the certificate is not trusted by the Java trust store.

Resolution

1. Jira's SSL certificate needs to be imported to its own trust store in order for Jira able to trust itself here. Follow our resolution in our Unable to Connect to SSL Services Due to 'PKIX Path Building Failed' Error in Jira Datacenter and Server knowledge base article to resolve the problem.

2. Additionally verify the configuration with httpclienttest as described in the readme of that repository.

Jira applications with port forwarding configured

Applies to environments that are using port forwarding, such as iptables or firewalld:

Cause

If iptables or firewalld has been configured to forward packets from 80 to 8080 and/or 443 to 8443, you may see gadget titles be broken and show as __MSG_gadget.xxxxxxx.  This can happen if the rules are configured only to forward packets from external sources.  

Resolution

You need to create a rule both for packets coming from an external IP sources and for local packets:

Using iptables

iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080

iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8443 

iptables -t nat -A OUTPUT -d XX.XXX.XXX.XX/32 -o lo -p tcp -m tcp --dport 80 -j DNAT --to-destination XX.XXX.XXX.XX:8080
iptables -t nat -A OUTPUT -d XX.XXX.XXX.XX/32 -o lo -p tcp -m tcp --dport 443 -j DNAT --to-destination XX.XXX.XXX.XX:8443

Using Firewalld

Configure external port forwarding for the Firewall Zone

</description>
  <service name="dhcpv6-client"/>
  <service name="ssh"/>
  <service name="http"/>
  <service name="https"/>
  <port protocol="tcp" port="80"/>
  <port protocol="tcp" port="443"/>
  <port protocol="tcp" port="8080"/>
  <port protocol="tcp" port="8443"/>
  <forward-port to-port="8443" protocol="tcp" port="443"/>
  <forward-port to-port="8080" protocol="tcp" port="80"/>
</zone>

Create additional direct rule for local traffic

firewall-cmd --permanent --direct --add-rule ipv4 nat OUTPUT 0 -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080
firewall-cmd --permanent --direct --add-rule ipv4 nat OUTPUT 0 -p tcp -o lo --dport 443 -j REDIRECT --to-ports 8443

Jira applications with a broken Application Link

The following appears in the atlassian-Jira.log file:

2016-02-26 13:48:12,564 http-nio-2186-exec-23 ERROR      [o.a.c.c.C.[.[localhost].[/JSP-262186].[servlet-module-container-servlet]] Servlet.service() for servlet [servlet-module-container-servlet] in context with path [/JSP-262186] threw exception
com.atlassian.templaterenderer.RenderingException: org.apache.velocity.exception.MethodInvocationException: 
Invocation of method 'getText' in  class com.sun.proxy.$Proxy12830 threw exception java.lang.NullPointerException at com/atlassian/applinks/oauth/auth/outbound_nonapplinks.vm[line 52, column 44]
	at com.atlassian.templaterenderer.velocity.one.six.internal.VelocityTemplateRendererImpl.render(VelocityTemplateRendererImpl.java:109)
	at sun.reflect.GeneratedMethodAccessor1986.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	...
Caused by: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getText' in  class com.sun.proxy.$Proxy12830 threw exception java.lang.NullPointerException at com/atlassian/applinks/oauth/auth/outbound_nonapplinks.vm[line 52, column 44]
	at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:337)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:284)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:262)
	at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:342)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
	at org.apache.velocity.Template.merge(Template.java:235)
	at com.atlassian.templaterenderer.velocity.one.six.internal.VelocityTemplateRendererImpl.render(VelocityTemplateRendererImpl.java:100)
	... 234 more
Caused by: java.lang.NullPointerException
	at com.atlassian.Jira.i18n.AbstractI18nResolver.getText(AbstractI18nResolver.java:24)
	at sun.reflect.GeneratedMethodAccessor1918.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	...

Cause

There appears to be a broken application link configured in Jira. A broken application link is defined as an application linked to another application, e.g. Confluence that is non-functional and inaccessible. Likely causes include network issues, the other application being offline or a corrupted application link configuration. Or the Application Link may have been established when a trusted certificate did not exist, so it failed to create properly.

To check for broken application links, go to Administration > Applications > Application Links and check if any of the links are flagged as Offline, for example:

Alternatively, you can also type the letter G twice while logged in as an admin and type in 'Application links' in the admin search box.

You may want to refer to the Application Links Troubleshooting Guide for more information on how to fix broken application links.

Resolution

Delete the broken application link and recreate it, or follow the Application Links Troubleshooting Guide to fix the broken application link.

 Jira applications with a broken Proxy configuration

Cause

Jira is unable to communicate with itself due to a Web Proxy not able to correctly resolve the Jira Server.

Resolution

Add the Jira Server's hostname to the -Dhttp.nonProxyHosts and -Dhttps.nonProxyHosts arguments as in How to Configure an Outbound HTTP and HTTPS Proxy for Jira applications.

Jira applications using an AJP Connector with Apache Web Server

Diagnosis

You may see messages in the proxy logs like:

[Sat Feb 23 15:08:42.168550 2019] [proxy:error] [pid 22708:tid 140608035321600] AH00940: AJP: disabled connection for (x.x.x.x)
[Sat Feb 23 15:08:46.079259 2019] [proxy:error] [pid 22710:tid 140608060499712] (111) Connection refused: AH00957: AJP: attempt to connect to x.x.x.x:8009 (x.x.x.x) failed
[Sat Feb 23 15:08:46.079298 2019] [proxy:error] [pid 22710:tid 140608060499712] AH00959: ap_proxy_connect_backend disabling worker for (x.x.x.x) for 60s

Cause

When the reverse proxy encounters an error communicating with JIRA (particularly if it's on a remote machine) the backend worker may be disabled (which in turn disrupts subsequent connections

Resolution

You may be able to reduce the occurence of this by adding retry=0 to your proxyPass directive:

<Location />
  ProxyPass / ajp://localhost:8009/ retry=0
  ProxyPassReverse / ajp://localhost:8009/ retry=0
</Location>

Restart Apache for the changes to take effect

Jira configured with Web Proxy in front with BasicAuth password restriction

Cause

Jira is unable to communicate with itself due to a Web Proxy requiring authentication.

Bug ticket raise JRA-61273 - Getting issue details... STATUS

Resolution for Apache

Add the Server IP itself for access without password:

Order Allow,Deny
Allow from <hostname>
Satisfy Any

Resolution for IIS

Use  'Anonymous access' instead of selecting 'Basic authentication' or 'Integrated Windows authentication'. For more details steps, please refer Integrating Jira applications with IIS documentation under Troubleshooting section.

Resolution for Shibboleth

If using Shibboleth authentication,

1. leave the /rest URLs outside the shibboleth authentication.
2. /rest URL must not be shibboleth protected.
3. Example :

<LocationMatch "^(/rest)/[^/]+">
AuthType None
Require all granted
</LocationMatch>

<LocationMatch "/">
AuthType Shibboleth
ShibRequireSession On
require valid-user
ShibUseEnvironment On
Order allow,deny
Allow from all
</LocationMatch>

Anonymous requests are blocked by Jira

Cause

A known cause is Jira uses Prevent Anonymous Access plugin that blocks all anonymous requests. This blocks the request to /rest/gadgets/1.0/g/messagebundle..., hence Jira can't display the gadget titles.

Resolution

The plugin above comes with a whitelist page. Whitelisting the following endpoint will allow Jira to get the gadget titles correctly:

/rest/gadgets/1.0/g/messagebundle.+

Web Fragment Finder App

Cause

The logs show LittleProxy trying to talk back to com.wittified.webfragment-finder which from the Web Fragment Finder add-on:

2017-04-14 06:42:16,602 LittleProxy-ClientToProxyWorker-0 WARN b757 758x82771x1 dh8gtn 62.153.237.114 /rest/plugins/1.0/com.wittified.webfragment-finder-key [i.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x3e6311c8, /127.0.0.1:33660 => /127.0.0.1:8888]
2017-04-14 06:42:16,602 LittleProxy-ClientToProxyWorker-1 WARN b757 758x82771x1 dh8gtn 62.153.237.114 /rest/plugins/1.0/com.wittified.webfragment-finder-key [i.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x9b85bea9, /127.0.0.1:33662 => /127.0.0.1:8888]

Resolution

1. Disable Web Fragment Finder Add-On
2. Restart Jira

SSL certificate using SAN

Applies to Jira 7.4 and above where Jira is running over SSL with SAN.

Cause

Jira fails to communicate with itself due to an Apache HttpClient bug. The following error is seen in the logs:

2017-10-10 17:04:06,623 HealthCheck:thread-5 ERROR ServiceRunner     [c.a.t.j.healthcheck.support.BaseUrlHealthCheck] An error occurred when performing the Base URL healthcheck:
java.lang.ClassCastException: [B cannot be cast to java.lang.String
    at org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309)

Workaround

1. Temporarily use an SSL certificate that doesn't use SAN
2. Temporarily run Jira over HTTP

CORS errors when connecting 

Depending on how your environment is set up, a CORS error could be happening.

Cause

Jira fails to communicate with itself due to CORS verifications failing. 

1.  Enable HTTP debugging to identify what's failing during the HTTP request Jira is doing when trying to load the gadget's name;
   1. Go to *Cog > System > Logging and Profiling*
   2. Click on "Configure logging level for another package"
   3. Enter {{org.apache.http}} for the Package name → Set Logging Level to *DEBUG* → *Add*
   4. Start monitoring the {{$jira-home/log/atlassian-jira.log}} file;

   5. Now, reload Jira dashboard and go to add a gadget. Keep an eye on the log file for a line starting with /rest/config/1.0/directoryitems/local.json, as in the example below:

      2023-11-01 11:20:12,042-0300 http-nio-8941-exec-2 DEBUG admin 680x402x2 aoiost 0:0:0:0:0:0:0:1 /rest/config/1.0/directoryitems/local.json [o.apache.http.headers] http-outgoing-169 << HTTP/1.1 200 
      2023-11-01 11:20:12,042-0300 http-nio-8941-exec-2 DEBUG admin 680x402x2 aoiost 0:0:0:0:0:0:0:1 /rest/config/1.0/directoryitems/local.json [o.apache.http.headers] http-outgoing-169 << X-AREQUESTID: 680x426x2

If CORS errors are happening, you should see some lines similar to:

2023-11-16 10:32:48,232+0000 http-nio-8080-exec-7 WARN MYUSERID 632x11278x1 67wlxl [JIRAIP] /rest/config/1.0/directoryitems/local.json [c.a.g.r.internal.http.HttpClientFetcher] A request to [THIS-IS-THE-URL]/rest/gadgets/1.0/g/messagebundle/und/gadget.common%2Cgadget.resolutiontime has been denied. To allow requests to this URL add the application URL to your whitelist (http://confluence.atlassian.com/x/KQfCDQ).

Workaround

1. In Jira, go to Cog > System > Allowlist
2. On "Type", select "Wildcard expression", and on expression try the following expression, making sure to replace with the same exact URL that you are seeing on the example above, in the "THIS-IS-THE-URL" section:
   • THIS-IS-THE-URL/*
     • Example: http://myJiraInstance.com:8080/*
   • Check "Allow Incoming"
   • Check "Allow anonymous users"
3. Try refreshing the dashboard and see if the gadget's names are rendered now