Symptoms

When Bitbucket Server Backup Client is performing a backup, it hits to OOME and from the logs, the last table that gets backed up before the memory issues starts is sta_shared_lob.

2015-03-03 05:07:34,702 INFO  [threadpool:thread-124] admin @851LERx306x483x2 10.4.0.121,127.0.0.1 "POST /mvc/admin/backups HTTP/1.0" c.a.s.i.b.l.DefaultLiquibaseMigrationDao Backing up sta_shared_lob table

 sta_shared_lob is a table that persists of large objects that's used by repository hook settings and user settings

• So the 'getting started' data is part of the user settings (coloration with sta_user_settings table)
• The table is used for repo (hook) configurations (coloration with sta_repo_hook table)

In this scenario, it may look like this table may contain a huge amount of data that causes OOME when it is loaded into memory.

Diagnosis

At this stage, we would be interested  to see how many rows are in this table and the total size

• select pg_size_pretty(pg_relation_size('sta_shared_lob'));

• select count(1) from sta_shared_lob;

If the size is unreasonably huge, perhaps this would be the bug that you're hitting BSERV-7137 - Getting issue details... STATUS  

Workaround

For the workaround, you would need to:

1. Identify the rows which do have not have reference from other tables.

select id
from sta_shared_lob 
where not exists (select sta_repo_hook.lob_id from sta_repo_hook where sta_repo_hook.lob_id = sta_shared_lob.id)
and not exists (select sta_user_settings.lob_id from sta_user_settings where sta_user_settings.lob_id = sta_shared_lob.id)

2. Remove those rows

delete from sta_shared_lob
where id in
(select id
from sta_shared_lob 
where not exists (select sta_repo_hook.lob_id from sta_repo_hook where sta_repo_hook.lob_id = sta_shared_lob.id)
and not exists (select sta_user_settings.lob_id from sta_user_settings where sta_user_settings.lob_id = sta_shared_lob.id));

 The table sta_shared_lob has constraints that will stop deletion if data are present on another table

Please backup your $BITBUCKET_HOME and the database first before performing any altering it

Resolution

The resolution is to watch this JIRA ticket until the fix has been applied BSERV-7137 - Getting issue details... STATUS