Recommended Action plan if a Repository Becomes Corrupted on a Stash Server

'How Do I...' and 'How to...' Guide to Stash

On this page

Still need help?

The Atlassian Community is here for you.

Ask the community

Symptoms

This is a general recommendation/real life example for situations where the repository on your Stash server becomes corrupted for some reason.

Resolution

As per our Support reports, no customer has yet encountered a situation where one of Stash's repositories became corrupted. Git's internals are very robust, so, as long as you are using Stash normally, it's very unlikely, short of some form of disk failure, for one of the repositories to become damaged.

But what if one does? Unfortunately, there's no "one size fits all" approach here. The best and recommended approach is to contact Atlassian Support at https://support.atlassian.com by lodging a Support request with us.

If you are still curious to know more, here are some of the steps carried out internally by our Stash development team, when a repository corruption was detected while developing new Stash features. Development was able to recover (without losing any commits) using a mix of two strategies:

  1. They shut down Stash periodically and zip up their repositories. To reduce the downtime, they just copy the data to a separate location while Stash is down (which is very fast), and then they compress it after Stash has been brought back online. This allows them to backup their repositories (some of which are quite large, for test data) in just a few minutes of downtime (generally 5 minutes or less)
  2. Every developers' local clone is, in effect, a backup of the Stash repository. Git repositories include the full history of every reachable ref when cloning, making them very useful for "restoring" repositories.

The most complicated restore development had to do internally (which, again, was accomplished without losing any data), was started by unpacking the zipped repository to serve as a base. They then used scp to copy the .pack and .idx files from some developers' machines into that unpacked repository. A simple git gc eliminated all the duplicate and corrupt objects and produced a single clean, fully functional repository. They replaced the repository directory in their STASH_HOME with the rebuilt repository and development continued without missing a beat.

 

Last modified on Feb 23, 2016

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.