How do I Avoid Long Reindex Times When I Upgrade?

Mitigating lengthy reindex times

If reindexing your repository takes longer than you can allow, you can use a temporary copy of your repository and FishEye instance to reduce downtime during the reindexing process.

Most upgrades (even major ones) do not require a reindex. If a reindex is required, this will always be explicitly mentioned in the Upgrade Guide for that release.

Reindexing with a temporary copy of your FishEye instance

This section describes how to perform a full reindex of a particular repository. Note that, depending on the repository size, the reindex could take up to several days.

To reindex a temporary copy of your FishEye instance:

Make a copy of your FishEye instance to another server. See 'How to make a temporary copy of your FishEye instance' below for instructions.
Upgrade the temporary FishEye, then start it up, connected to your repository. It will automatically begin the scanning process.
If you are concerned about the repository being overloaded by the scanning process, you can make a copy of that as well. See 'How to make a temporary copy of your repository' below for instructions.
If you do that, you must edit the config.xml of your temporary FishEye instance to point to your temporary repository.
The copied instance will run its course without affecting your production instance.
1. Shutdown both your servers completely.
2. Make a backup of your FISHEYE_INST directory.
3. Replace the FISHEYE_INST/var/cache directory on live FishEye with the FISHEYE_INST/var/cache from your test server.
4. Download the latest FishEye/Crucible from Atlassian downloads.
5. Follow the instructions in the Upgrade Guide to upgrade to the new version.
The scan of the temporary FishEye instance (and repository, if you copied that also) is complete. You're now free to delete the temporary copy(s).

How to make a temporary copy of your FishEye instance

To make a copy of your FishEye instance, follow the instructions for Migrating FishEye Between Servers.

How to make a temporary copy of your repository

To make a copy of your repository use rsync (for CVS repositories in the Linux environment) or svnsync documentation (for Subversion only).

How to reindex a single repository on a test server

If you need to reindex your repository on your production system but don't want to burden your production server, carry out the following steps:

1. Install another instance of FishEye on a test server (the same FishEye version as the one you are using).
2. Add a repository to FishEye with the exact same name and details as that referenced by the production server.
3. Let it finish indexing. Go to Administration > View Repository List > Stop (shown next to the name of your repository) and disable on both production and test.
4. Copy over the FISHEYE_INST/var/cache/REPO directory on the production FishEye with the FISHEYE_INST/var/cache/REPO directory from the test server.

5. Trigger a review revision data re-index: Administration > Repository > Maintenance > Review-Revision Data Index.

For this procedure, neither server needs to be shut down.

Upgrading your cross-repository index using a temporary staging server

This section describes how to upgrade the cross-repository index for selected repositories. Note that, depending on the repository size, the reindex will typically finish in a few hours, but should never take longer than a few days.

In this procedure it is assumed that you have a production server (referred to as PROD in these instructions) that is running a FishEye version earlier than 3.1, and a separate staging server (STAGING) that will be used to perform the cross-repository index upgrade offline.

Make a live backup of the PROD server with the following options:
1. Repository and application caches
2. SQL database
You can do this either from the FishEye Admin area (go to Administration > System Settings > Backup), or from a command line, for example:
```
$ ./bin/fisheyectl.sh backup -f ~/Documents/backup.zip --no-uploads --no-templates --no-plugins --cache --no-ao
```
Install FishEye 3.1, or a later version, on the STAGING server.
Restore the backup of PROD to the STAGING server.
Start FishEye on the STAGING server. Note that:
- The cross-repository index upgrade will start automatically on the STAGING server. If you want to perform the cross-repository index upgrade for selected repositories only, it is safe to remove unwanted repositories from the STAGING server now, either by going to Administration > Repositories, or by using REST endpoints (see below).
- The STAGING server doesn't need to have access to configured SCM's as the cross-repository upgrade task does not interact with them.
- You may want to disable polling on the STAGING server. You can either go to Administration > Repository Settings > Defaults > Updater to disable polling for all repositories (although this will not affect particular repositories that have been configured to ignore default settings), or go to Administration > Repository Settings > Repositories > Repository X > Updates to disable polling for just Repository X. Disabling polling is not required, but will avoid logging errors to the FishEye log file if the SCMs are not accessible from the STAGING server.
Wait for the cross-repository index upgrade to finish on the STAGING server. Check by going to Administration > Repositories.
Stop the STAGING server.
Make a full backup of the PROD server and then stop it.
Install the same version of FishEye on the PROD server as used on the STAGING server (as in step 2 above).

Delete the following FishEye indexes on the PROD server and replace them with the equivalent caches from the STAGING server. You can choose your preferred option to copy files between machines using ssh/scp/rsync, possibly combined with tar/zip tools. The example below shows how the scp command could be used:

ssh PROD
cd FISHEYE_INST  # replace FISHEYE_INST with the location of your FISHEYE_INST folder
rm -rf cache/globalfe
rm -rf var/cache/repoX  # repeat for each repository repoX that was upgraded on STAGING server
scp -r STAGING:STAGING_FISHEYE_INST/cache/globalfe cache/
scp -r STAGING:STAGING_FISHEYE_INST/var/cache/repoX var/cache/ # repeat for each repository repoX that was upgraded on STAGING server

Start the PROD server.
All the changesets that were added to SCMs after backing up the PROD server in step 1 will now be indexed on the PROD server.

The only drawback with this procedure is that changeset comments added for changesets in PROD after step 1 will not get indexed, so they will not appear in the activity stream. There is no easy way to reindex them, apart from fully reindexing each affected repository, which is what this procedure is intended to avoid. A new REST endpoint could be implemented to address this (see FECRU-3764 - Getting issue details... STATUS ).

Note: the following REST endpoint could be used to force a cross-repository index upgrade for a selected repository: /rest-service-fecru/admin/repositories-v1/repoX/reindex-search. There should be no need to use this, but it may be useful if something goes wrong.

Child pages