PostgreSQL segmentation fault after upgrading to Bitbucket 4.11 and up
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
After upgrading to Bitbucket Server 4.11+, the PostgreSQL database periodically restarts with errors similar to the following
2016-12-11 06:40:06 EST LOG: server process (PID 8370) was terminated by signal 11: Segmentation fault
2016-12-11 06:40:06 EST LOG: terminating any other active server processes
2016-12-11 06:40:06 EST WARNING: terminating connection because of crash of another server process
2016-12-11 06:40:06 EST DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2016-12-11 06:40:06 EST HINT: In a moment you should be able to reconnect to the database and repeat your command.
2016-12-11 06:40:06 EST WARNING: terminating connection because of crash of another server processDiagnosis
Environment
- PostgreSQL version 9.3.0 to 9.3.5 is installed
Core dump
If configured appropriately, the PostgreSQL server process can create a core file upon receipt of the SIGSEGV. The failure is the result of a null pointer defererence and the failing thread will have a stack trace like the following:
$ gdb /usr/lib/postgresql/9.3/bin/postgres core
..
(gdb) where
#0 0x00000000004ea5ca in transformTopLevelStmt ()
#1 0x00000000004ea68d in parse_analyze ()
#2 0x00000000006451c3 in pg_analyze_and_rewrite ()
#3 0x00000000006fa723 in RevalidateCachedQuery ()
#4 0x00000000006fa980 in GetCachedPlan ()
#5 0x0000000000646e4d in PostgresMain ()
#6 0x000000000045a5ad in ServerLoop ()
#7 0x00000000006076c3 in PostmasterMain ()
#8 0x000000000045af44 in main ()Cause
This issue is caused by a bug in early releases of PostgreSQL 9.3. The bug is specifically triggered by load from Bitbucket Server 4.11 (and later) due to an upgrade to the PostgreSQL JDBC driver from version 9.4.1209 to 9.4.1211.
Resolution
Upgrade PostgreSQL 9.3 to 9.3.6 or later, or to a newer release from 9.4 - 9.6.