Document-based replication in Jira Data Center
The Document-Based Replication feature introduced for Jira 8.12.0 and later mitigates apps' impact on indexing time and prevents index inconsistencies in Jira Data Center even if an app takes time indexing data. With the DBR feature on, Jira Data Center is much more horizontally scalable. As a result, the more nodes there are in the cluster, the better is the overall throughput while maintaining search consistency.
Benefits of DBR
We've introduced DBR to lower index update distribution time in Jira DC cluster. The immediate results are the following:
From index perspective: Improved, more stable and faster index consistency between nodes in Jira DC.
From user perspective: Changes in the index are distributed faster between nodes. That creates less friction to your end users because data is more consistent across the nodes. Users can collaborate effectively.
DBR considerations
The introduced architectural changes require each node to communicate directly with other nodes which can double the network traffic. When testing this on a 8 node cluster running a sustained stress load of 400 requests per second, the total network traffic volume amounted to 25% of total capacity of a 1Gbps link. Traffic will be lower if the load or cluster size is lower.
Metrics without DBR (versions below 8.12.0)
The chart shows the number of operations that the replication process needs to perform to catch up with issue changes and propagate those to all the nodes. The job runs every 5 seconds and makes sure each issue gets indexed in the local Lucene index.
The blue dots show a healthy index and data being consistently replicated between the nodes. The scattered colored dots show the index getting delayed which results in data inconsistency. Ideally there is no or minimal delay in data indexing.
A 0ms index time delay means Jira DC runs without apps impacting issue indexing. It looks good and shows a healthy Jira DC. The dots are frequent, which means that the replication job consumes the queue every 5 seconds, and nothing is piling up.
The bad state of Jira DC shows up when the index delay is 100, 200 or 500 ms. In this state, the Jira DC with 4 nodes we've used for tests can't handle the installed apps that delay the issue indexing time by more than 100ms. The dots get further apart, which means that each run takes more than 5 seconds to catch up. Also, the queue eventually grows, and the job finds more and more items with each run.
Metrics with DBR (8.12.0+)
With DBR on, the “bad state” is nearly completely gone. Jira DC can stand even a 20 sec index time delay. In conclusion, DBR can mitigate consistency problems even if apps take time indexing data.
The test was carries out with the following set-up:
4 Jira DC nodes, running on c5.9xlarge instances.
Jira version (8.12.0-SNAPSHOT)
Test duration: 40 minutes
Load: 400 active concurrent users (ramp-up time of 10 minutes, then flat load for 20 minutes).
2M issues
Stress tests
Our stress test on an 8-node cluster using Jira DC 8.5 shows that the index became highly inconsistent after the load of 200 requests per second.
Jira DC 8.13 was able to handle higher flat load of 330 requests per second with the index remaining consistent throughout. It proves that Jira DC can now handle the index consistency issues caused by high load.
Technical background
Jira DC uses the replicated index operations (RIO) database table to replicate index changes across the cluster. When a user undertakes an action that changes the index (create issue, add comment, …) on a node that node creates an entry in the RIO as a sender node. Then, other nodes execute the replay process (RP) every 5 seconds to process the RIO table, load the issue from the DB, and create Lucene entries for the issue. RP is trying to catch up with all changes in a single thread. As long as each run takes less than 5 seconds and is able to process all created RIOs since the last execution, Jira DC is healthy and the index is consistent across the cluster.
The Document-Based Replication (DBR) uses the cache replication mechanism to share the generated Lucene documents created on the sender node to other nodes directly. The other nodes don’t have to read the full issue data from the DB or prepare the Lucene document (which is the expensive part). They apply what is being sent directly to their indexes (which is the cheap part).
The RIO/RP method is still there, but it uses entity versioning (see below) to figure out if DBR has done all the work, and corrects anything DBR missed. It’s also used as a fallback method in case there are network issues.
DBR ensures that the indexing delay is between 10-100 miliseconds on each node irrespective of the number of issues being changed.
Entity versioning
To keep the index consistent we introduced entity versioning. With entity versioning each update of an issue is also reflected by bumping the version - a number that goes up with every change. We have associated an issue with that number in the DB and in the index. It is assumed that the latest version (the one with the highest version number) is the one that should exist in the index. This prevents a version mismatch in the cluster resulting, for example, from issue status changes that happen at the same time, and ensures consistency between the database and the nodes.
Actions using DBR
The DBR mechanism is generally applied for user actions that trigger reindexing.
Actions using DBR | Actions not using DBR |
|---|---|
Background reindexing | Full foreground reindexing |
Project archiving | Issue archiving |
Reindexing when a node starts |
DBR in the logs
To find more data about DBR actions and monitor the cluster behavior, look up the logs in the atlassian-jira.log file.
In the standard atlassian-jira.log you can find DBR-related aggregated logs showing DBR properties on a given node. All DBR logs are prefixed by:[DBR] .
DBR Sender Stats Logs
The DBR sender logs describe the part responsible for preparing the DBR message with the Lucene document(s) and have the following format:
[DBR] [SENDER] snpapshot stats...
[DBR] [SENDER] total stats...
The snapshot stats cover the last ~5min and the total stats cover the time from the start of a node.
The following are the most important stats:
createDBRMessageUpdateInMillis / createDBRMessageUpdateWithRelatedInMillis- the time to create a DBR message; note that this is done asynchronously and not in the request thread generating the change in the local index;createDBRMessageUpdateBytes / createDBRMessageUpdateWithRelatedBytes- the size of the DBR message;sendDBRMessage- the number of DBR messages successfully delivered to the transport layer (not the destination nodes); this should be equal to the number of index updates; note that a single DBR message may contain a single issue and some or all related entities (comments, worklogs, change history);sendDBRMessageErrors- the number of DBR messages which weren't delivered to the transport layer.
DBR Receiver Stats Logs
The DBR receiver logs describe the part responsible for receiving DBR messages with the Lucene document(s) from other nodes and have the following format:
[DBR] [RECEIVER] snapshot stats...
[DBR] [RECEIVER] total stats...
The snapshot stats cover the last ~5min and the total stats cover the time from the start of a node.
The following are the most important stats:
receiveDBRMessage- the number of received DBR messages;receiveDBRMessageDelayedInMillis- the delay between creating a DBR message on the source node and accepting the DBR message on the destination node; note that this is based on comparing local times from two different nodes;processDBRMessageUpdateSerializeInMillis / processDBRMessageUpdateWithRelatedSerializeInMillis- the partial time to process the DBR message locally - the time to deserialise Lucene document from the DBR message;processDBRMessageUpdateIndexInMillis / processDBRMessageUpdateWithRelatedIndexInMillis- the partial time to process the DBR message locally - the time to perform a conditional update of the local index with the documents from the DBR message.
Cache Replication Stats Logs
These are not specific to the DBR feature but contain stats for all remote caches replicated by value and have the following format:
[LOCALQ] [VIA-INVALIDATION] Cache replication queue stats per node: ... snapshot stats ...
[LOCALQ] [VIA-INVALIDATION] Cache replication queue stats per node: ... total stats ...
The snapshot stats cover the last ~5min and the total stats cover the time from the start of a node.
The following are the most important stats:
timeToAddMillis- the time to store a cache replication message in local store before sending it to the other node;sendCounter- the number of cache replication messages sent from the current node to the destination node;timeToSendMillis- the time to deliver the cache replication message from current node to the destination node.
Node Index Replay Stats Logs
Node Index Replay is the process responsible for replaying index operation from other nodes. As of Jira 8.12 the replay will be done conditionally. If an index operation to replay has already been delivered by DBR the reindexing process is skipped. This can be used to measure DBR effectiveness. When all index replication operations are skipped, it means all indexing updates are propagated through the cluster via DBR (100% DBR effectiveness).
The logs have the following format:
[INDEX-REPLAY] [STATS] Node replay index operations stats (total)...
[INDEX-REPLAY] [STATS] Node replay index operations stats (snapshot)...
The snapshot stats cover the last ~5min and the total stats cover the time from the start of a node.
The following are the most important stats:
numberOfRemoteOperations- the number of processed index operations from other nodes;numberOfLocalOperations- the number of processed index operations from the current node;timeInMillis- the time of processing a batch of index operations; the replay process is run every 5secs andtimeInMillisshould be much lower than 5sec;filterOutAlreadyIndexedBeforeCounter- the number of index replay operations that may be processed locally;filterOutAlreadyIndexedAfterCounter- the number of index replay operations that are processed locally; iffilterOutAlreadyIndexedAfterCounterequals 0 we have 100% DBR effectiveness;