Jira Data Center Troubleshooting

Related content

Still need help?

The Atlassian Community is here for you.

Ask the community


Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.

Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

We've gathered some resources and troubleshooting tips that will help you solve your problems with Jira Data Center.

Health Checks


The best first step to troubleshoot Jira Data Center is from the Health Check tools. Jira Data Center provides a set of tools that you can use to monitor the health of your instance, as well as to identify the root cause when the instance is not performing as expected.
To access the health check tools, go to  > System > Support Tools. All health checks are listed in the Instance health tab. Expand each of them to see the details and links to knowledge base articles that describe the results.

Installation & Administration Checklist


Starting a Jira Data Center can be a smooth experience following the guide we have available in Installing Jira Data Center documentation. There can be times when a problem rises and this will be the best place to start troubleshooting.

Is the Shared Home directory configured?

jira.shared.home is to be configured in cluster.properties file found in each nodes’ Jira Home directory. Shared Home directory must be accessible by all the nodes. The user who runs Jira on each node requires permission to read and write to the Shared Home directory path. 

Can all servers in the cluster see each other?

Each server that is configured as nodes in the same cluster must be able to see each other. This should be possible via IP address or Hostnames. You can choose how the nodes see each other by configuring ehcache.listener.hostName value in cluster.properties of the said node.

How to test: Ensure ping <hostname> or ping <ip address> execute properly on each of the node

Are all the ports open and accessible?

Jira nodes will require to communicate with each other for the purpose of replicating the Caches data. Jira Data Center utilise the following ports by default:

  • Tomcat uses 8005 and 8080
  • Ehcache uses 40001 and 40011
  • Prior to Jira 7.13.1, ehcache.object.port must be declared to 40011 otherwise ephemeral port is used

How to test: Ensure telnet <hostname> <portnumber> execute properly on each of the node. 
Setup secondary HTTP connector bypassing Load Balancer

For troubleshooting purpose, it is important to add secondary Tomcat connector on each node bypassing the Load Balancer so that we can login to individual node. 
To do this, open $Jira_INSTALL/conf/server.xml on a text editor and add the following section and restart the application node. Repeat the same on each application node. You can access individual node from https://hostname:8081

Are your servers synchronised?

Unsynchronised server times can break the cache synchronisation. Each node in the cluster must periodically send a heartbeat message to the cluster and there is an x minutes threshold before node deemed as OFFLINE, therefore, server time synchronisation is important. 

  • Is the time and date significantly different between each machine? Consider using a Network Time Server to ensure all machines are in synchronization.
  • Is each machine aware of the time zone that it's in?

See more: Health Check Cluster Time Synchronization
Are Sticky Sessions configured on the Load Balancer?

Jira is a stateful app (ie. saves client data generated in one session for use in the next session with that client) therefore it requires session affinity so multistep event can work. This means when a user logs in to one node, the same user must maintain a session on the same node otherwise re-authentication is required. 

Make sure JVM settings are identical across nodes

Each server node comes with its own JVM, therefore, the setting must be made on individual nodes. It is recommended that the JVM startup parameters (e.g. memory allocation, GC algorithm etc) are identical between nodes to avoid discrepancy. 

Ensure single cluster have single database

Jira Data Center can have multiple nodes in the same cluster and they are all connected to the same database. The database is the common entity and is used as the medium for Index replications in addition to storing all Jira application’s data.

Have you checked the logs for error messages?

Atlassian server products are Java applications that bundle the Tomcat application server. Both the application and Tomcat generate logs that can help to diagnose a problem with application links. Jira Home is where the application logs can be found.

Are all the Health Checks green?

Ultimately, we want to see the Data Center Health Checks all green on successful installation of the Jira Data Center. 
If there’s a warning or an error, please have a look at our Health Checks documents.

Startup Troubleshooting

Starting a node in Jira Data Center is similar to starting up a Jira Server. The first step is to check our Troubleshooting Jira Startup Failed Error. Next is to check on the cluster.properties file stored in the Jira Local Home.

cluster.properties file

The cluster.properties file is located in Jira Local Home of each node and its content is unique to each node. Jira determines if the node is part of a Jira Data Center cluster by this particular file. The two must have parameters are as follows:

Example of cluster.properties

# This ID must be unique across the cluster
jira.node.id = node1
# The location of the shared home directory for all Jira nodes
jira.shared.home = /data/jira/sharedhome

Known issues

See also our list of known issues with starting up a second (or more) nodes in the cluster:

Internodes communication Troubleshooting


Jira nodes actively communicate with each other to perform to main tasks

  • Index replications
  • Cache replications

Index Replication


Jira keeps all the copies of the indexes up to date automatically. The synchronization aims for eventual consistency and is not synchronous, which means that there can be some delay before the index changes are seen on other nodes in the cluster. For instance, during busy hour or during issue/project import we expect to see a temporary gap in the index count between nodes; we expect the gap closing over time. It is communicated through the database.

How to know there is a problem:

  • Health Check Cluster Index Replication indicates a warning or an error

  • Inconsistency of search results between the nodes in the same cluster

    • The results remain different for a long period
    • The results do not correct itself during off-peak hours
    • The results show an increase in differences over time
  • Examples of search results inconsistency
    • Issue "A" appears on Node-1 but not on Node-3
    • Searching "Project Taco" returns 100 issues on Node-1 but only 89 issues on Node-2
    • Gadget returning different results between nodes

Believe there is a problem? Go to Index Replication Jira Data Center Troubleshooting

Cache Replication

Jira keeps some data in memory local to the node especially data that are used often such as permissions. The cache synchronisation is asynchronous but expected to be fast and consistent across nodes. It is communicated and being replicated over the network.

How to know there is a problem:

  • Health Check Cluster Cache Replication indicates a warning or an error

  • Inconsistency of Filter permission between nodes in the same cluster

    • Example: filter A is private in Node-1 while shared to group jira-users in Node-2

  • Inconsistency of Project permission between nodes in the same cluster

Believe there is a problem? Go to Cache Replication Jira Data Center Troubleshooting

Performance troubleshooting

Troubleshooting a performance issue in Jira Data Center takes a similar approach to troubleshooting performance on Jira Server. As a start, have a look at the links below


Description
ProductJira

Last modified on Mar 1, 2022

Was this helpful?

Yes
No
Provide feedback about this article

In this section

Related content

Powered by Confluence and Scroll Viewport.