Jira Data Center Troubleshooting
Platform Notice: Data Center Only - This article only applies to Atlassian products on the data center platform.
We've gathered some resources and troubleshooting tips that will help you solve your problems with Jira Data Center.
The best first step to troubleshoot Jira Data Center is from the Health Check tools. Jira Data Center provides a set of tools that you can use to monitor the health of your instance, as well as to identify the root cause when the instance is not performing as expected.
To access the health check tools, go to > System > Support Tools. All health checks are listed in the Instance health tab. Expand each of them to see the details and links to knowledge base articles that describe the results.
- HealthCheck: Attachments
- HealthCheck: Cluster Cache Replication
- HealthCheck: Cluster Index Replication
- HealthCheck: Cluster Locks
- HealthCheck: Cluster Time Synchronization
- HealthCheck: Free Space
- HealthCheck: Lucene Search Indexes
- HealthCheck: Scheduler
- HealthCheck: Secondary Storage
- HealthCheck: Shared Home
- HealthCheck: Storage Index Snapshot
Installation & Administration Checklist
Starting a Jira Data Center can be a smooth experience following the guide we have available in Installing Jira Data Center documentation. There can be times when a problem rises and this will be the best place to start troubleshooting.
jira.shared.home is to be configured in cluster.properties file found in each nodes’ Jira Home directory. Shared Home directory must be accessible by all the nodes. The user who runs Jira on each node requires permission to read and write to the Shared Home directory path.
Each server that is configured as nodes in the same cluster must be able to see each other. This should be possible via IP address or Hostnames. You can choose how the nodes see each other by configuring ehcache.listener.hostName value in cluster.properties of the said node.
How to test: Ensure ping <hostname> or ping <ip address> execute properly on each of the node
Jira nodes will require to communicate with each other for the purpose of replicating the Caches data. Jira Data Center utilise the following ports by default:
- Tomcat uses 8005 and 8080
- Ehcache uses 40001 and 40011
- RMI uses 1099
- Prior to Jira 7.13.1, ehcache.object.port must be declared to 40011 otherwise ephemeral port is used
How to test: Ensure telnet <hostname> <portnumber> execute properly on each of the node.
For troubleshooting purpose, it is important to add secondary Tomcat connector on each node bypassing the Load Balancer so that we can login to individual node.
To do this, open $Jira_INSTALL/conf/server.xml on a text editor and add the following section and restart the application node. Repeat the same on each application node. You can access individual node from https://hostname:8081
Unsynchronised server times can break the cache synchronisation. Each node in the cluster must periodically send a heartbeat message to the cluster and there is an x minutes threshold before node deemed as OFFLINE, therefore, server time synchronisation is important.
- Is the time and date significantly different between each machine? Consider using a Network Time Server to ensure all machines are in synchronization.
- Is each machine aware of the time zone that it's in?
See more: Health Check Cluster Time Synchronization
Jira is a stateful app (ie. saves client data generated in one session for use in the next session with that client) therefore it requires session affinity so multistep event can work. This means when a user logs in to one node, the same user must maintain a session on the same node otherwise re-authentication is required.
Each server node comes with its own JVM, therefore, the setting must be made on individual nodes. It is recommended that the JVM startup parameters (e.g. memory allocation, GC algorithm etc) are identical between nodes to avoid discrepancy.
Jira Data Center can have multiple nodes in the same cluster and they are all connected to the same database. The database is the common entity and is used as the medium for Index replications in addition to storing all Jira application’s data.
Atlassian server products are Java applications that bundle the Tomcat application server. Both the application and Tomcat generate logs that can help to diagnose a problem with application links. Jira Home is where the application logs can be found.
Ultimately, we want to see the Data Center Health Checks all green on successful installation of the Jira Data Center.
If there’s a warning or an error, please have a look at our Health Checks documents.
Starting a node in Jira Data Center is similar to starting up a Jira Server. The first step is to check our Troubleshooting Jira Startup Failed Error. Next is to check on the
cluster.properties file stored in the Jira Local Home.
The cluster.properties file is located in Jira Local Home of each node and its content is unique to each node. Jira determines if the node is part of a Jira Data Center cluster by this particular file. The two must have parameters are as follows:
# This ID must be unique across the cluster
jira.node.id = node1
# The location of the shared home directory
all Jira nodes
jira.shared.home = /data/jira/sharedhome
See also our list of known issues with starting up a second (or more) nodes in the cluster:
- Adding a node to Jira Data Center results in setup screen
- ClusteringChecklistLauncher startup check failed
- Jira 7.6.x only one node will start in cluster
- Jira Data Center functionalities loss due to cluster wide lock
Internodes communication Troubleshooting
Jira nodes actively communicate with each other to perform to main tasks
- Index replications
- Cache replications
Jira keeps all the copies of the indexes up to date automatically. The synchronization aims for eventual consistency and is not synchronous, which means that there can be some delay before the index changes are seen on other nodes in the cluster. For instance, during busy hour or during issue/project import we expect to see a temporary gap in the index count between nodes; we expect the gap closing over time. It is communicated through the database.
How to know there is a problem:
- Health Check Cluster Index Replication indicates a warning or an error
Inconsistency of search results between the nodes in the same cluster
- The results remain different for a long period
- The results do not correct itself during off-peak hours
- The results show an increase in differences over time
- Examples of search results inconsistency
- Issue "A" appears on Node-1 but not on Node-3
- Searching "Project Taco" returns 100 issues on Node-1 but only 89 issues on Node-2
- Gadget returning different results between nodes
Believe there is a problem? Go to Index Replication Jira Data Center Troubleshooting
Jira keeps some data in memory local to the node especially data that are used often such as permissions. The cache synchronisation is asynchronous but expected to be fast and consistent across nodes. It is communicated and being replicated over the network.
How to know there is a problem:
- Health Check Cluster Cache Replication indicates a warning or an error
Inconsistency of Filter permission between nodes in the same cluster
Example: filter A is private in Node-1 while shared to group jira-users in Node-2
- Inconsistency of Project permission between nodes in the same cluster
Believe there is a problem? Go to Cache Replication Jira Data Center Troubleshooting
Troubleshooting a performance issue in Jira Data Center takes a similar approach to troubleshooting performance on Jira Server. As a start, have a look at the links below