Recommended Action plan if a Repository Becomes Corrupted on a Stash Server
Symptoms
This is a general recommendation/real life example for situations where the repository on your Stash server becomes corrupted for some reason.
Resolution
As per our Support reports, no customer has yet encountered a situation where one of Stash's repositories became corrupted. Git's internals are very robust, so, as long as you are using Stash normally, it's very unlikely, short of some form of disk failure, for one of the repositories to become damaged.
But what if one does? Unfortunately, there's no "one size fits all" approach here. The best and recommended approach is to contact Atlassian Support at https://support.atlassian.com by lodging a Support request with us.
If you are still curious to know more, here are some of the steps carried out internally by our Stash development team, when a repository corruption was detected while developing new Stash features. Development was able to recover (without losing any commits) using a mix of two strategies:
- They shut down Stash periodically and zip up their repositories. To reduce the downtime, they just copy the data to a separate location while Stash is down (which is very fast), and then they compress it after Stash has been brought back online. This allows them to backup their repositories (some of which are quite large, for test data) in just a few minutes of downtime (generally 5 minutes or less)
- Every developers' local clone is, in effect, a backup of the Stash repository. Git repositories include the full history of every reachable ref when cloning, making them very useful for "restoring" repositories.
The most complicated restore development had to do internally (which, again, was accomplished without losing any data), was started by unpacking the zipped repository to serve as a base. They then used scp to copy the .pack and .idx files from some developers' machines into that unpacked repository. A simple git gc eliminated all the duplicate and corrupt objects and produced a single clean, fully functional repository. They replaced the repository directory in their STASH_HOME with the rebuilt repository and development continued without missing a beat.