Considerations for Jira index snapshots
The information in this page relates to customizations in JIRA. Consequently, Atlassian Support cannot guarantee to provide any support for the steps described on this page as customizations are not covered under Atlassian Support Offerings. Please be aware that this material is provided for your information only and that you use it at your own risk.
Also, please be aware that customizations done by directly modifying files are not included in the upgrade process. These modifications will need to be reapplied manually on the upgraded instance.
Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.
The questions about performance are highly dependent on their hardware and also how much load the server is experiencing while concurrently taking the snapshot.
JIRA Uses indexes to provide quick results to search queries that are made by users and internal functions of JIRA. The Index snapshots provide copies of the recent state of the Index. These are useful in the event of a fail over scenario so that an offline node can quickly recover.
In the Index Recovery Page, Under Setting , If we enable Index recovery, For the option below, what time/day, it will trigger the snapshot creation:
a. Once per hour: (time?)
b. Once per day: (time?)
c: Once per week: (day? time?)
It will trigger after a minute or two and then at the interval specified, All JIRA services work this way up until JIRA 6.4 where a change is made to allow them to be scheduled more flexibly including at a particular time. In 6.4 we have changed all the services to support proper on scheduling and then you can specify what days and times fully.
Can we create the snapshot OnDemand?
If we have the index size, 18G, could you estimate how the size for one snapshot?
This is difficult to say accurately, but the process is that an optimized copy is taken of the index, this will typically be somewhat smaller than the original but depends upon many factors like hardware or how much load the server is experiencing at that moment. This compressed copy is then compressed into a zip archive. We typically see a compression ration of about 4 or 5 to 1. So for 18G, I would estimate about 4G. Your Mileage May Vary.
If we have the index size, 18G, Could you estimate how much additional free disk size required for index snapshots?
During taking snapshot, the process is that an optimized copy is taken of the index - so this up to 18Gb. This compressed copy is then compressed into a zip archive - 4Gb. JIRA will keep 3 archives. Total: 36Gb of additional free space (2x index size).
Please note that copy of index will be copied to java.io.tmpdir location, by default it's set to
-Djava.io.tmpdir="$CATALINA_TMPDIR", which is set to
CATALINA_TMPDIR="$CATALINA_BASE"/temp which is part of JIRA_INSTALL. If you split JIRA_INSTALL from JIRA_HOME, please make sure that you have enough space there.
How can we change the number of index snapshots retained
You can edit the number of backups to retain for the JIRA Index Snapshot Service by following these steps:
- Navigate to System > Indexing (in the Advanced section)
- Make sure that Enable Index Recovery is ON.
- Navigate to System > Services (also in the Advanced section)
- Locate the JIRA Index Snapshot Service and click Edit.
- The first field will be Number of backups to retain, edit this field and click Update.
Can I change the compression method from .tar.sz to zip or tar?
Yes. Starting with Jira 7.12.0, you can choose the compression algorithm that Jira will use to compress the index snapshot file. Please see the following page: How to change the compression method of index snapshot files in Jira Data Center
During the snapshot creation, (size=18G), how much time require to generate the snapshot? and will it affect system performance, like slowdown the system, create high load, etc?
At this size the snapshot will take many minutes maybe as long as an hour. How this affects the general system throughput seems to vary. In our own testing we see little impact, but some customers have reported a significant performance hit. This is an I/O intensive operation and the compression requires both CPU and memory so the impact will depend to a large extent on the capacity of the hardware. Another impact is, that a reindex at the same time as the snapshot creation on the same machine will fail due to a write lock on the index files.
Can we change the snapshot directory to another path?
No. You can use a Symbolic Link in a *nix environment. If you map this to a network drive you would need to be careful it did not introduce performance issues.
During the restoration snapshot, it mentions the system will be unavailable, but during a test it was found, that some parts of the system are accessible. Can users work on issue while snapshot restoration?
There is a very short window when the old index is swapped out for the one being restored when the index is not available and access is not possible. During the major operation of unzipping the snapshot the system will remain available as well as during the final reconciliation when any issues modified after the snapshot was taken are re-indexed.
And for an instance with 18G index size, how much time needed if perform snapshot restoration?
This is a three-step process:
- The snapshot is unzipped, this will depend on your hardware.
- The current index is swapped out and replaced by the snapshot, this is just a directory rename (move) but should be almost instantaneous.
- Any issues that have been updated since the snapshot was taken will need to be reindexed manually. Typically this would take less time than recovering the whole index. For example, if you have a one week snapshot and your system has 3 years worth of issues, it will take 2% of the time to re-index instead of reindexing a whole 3 years indexes.