Bamboo doesn't start up with PostgreSQL errors: remaining connection slots are reserved for non-replication superuser connections
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Bamboo fails to start up and the following errors appear inside the <Bamboo server home directory>/logs/atlassian-bamboo.log file:
2021-09-17 09:15:18,961 INFO [localhost-startStop-1] [BootstrapUpgradeManagerImpl] ---------------------------------------------------------------------------
2021-09-17 09:15:18,961 INFO [localhost-startStop-1] [BootstrapUpgradeManagerImpl] 3 : Validate if build number in home directory matches database (bootstrap)
2021-09-17 09:15:18,961 INFO [localhost-startStop-1] [BootstrapUpgradeManagerImpl] ---------------------------------------------------------------------------
2021-09-17 09:15:19,326 INFO [localhost-startStop-1] [HikariDataSource] HikariPool-1 - Started.
2021-09-17 09:15:19,384 ERROR [localhost-startStop-1] [BootstrapUpgradeHelper] Error getting database connection
org.hibernate.HibernateException: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: remaining connection slots are reserved for non-replication superuser connections
at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:63)
at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:107)
at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:246)
at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
at com.atlassian.bamboo.upgrade.BootstrapTaskConnectionProvider.getConnectionProvider(BootstrapTaskConnectionProvider.java:30)
at com.atlassian.bamboo.upgrade.BootstrapTaskConnectionProvider.getConnectionProvider(BootstrapTaskConnectionProvider.java:25)
at com.atlassian.bamboo.upgrade.BootstrapTaskConnectionProvider.getConnection(BootstrapTaskConnectionProvider.java:68)
at com.atlassian.bamboo.upgrade.utils.impl.BootstrapUpgradeHelper.getDatabaseConnection(BootstrapUpgradeHelper.java:45)
at com.atlassian.bamboo.upgrade.tasks.validation.BuildNumberInHomeMatchesDatabase.getBuildNumberFromDatabase(BuildNumberInHomeMatchesDatabase.java:76)
at com.atlassian.bamboo.upgrade.tasks.validation.BuildNumberInHomeMatchesDatabase.doUpgrade(BuildNumberInHomeMatchesDatabase.java:50)
at com.atlassian.bamboo.upgrade.BootstrapUpgradeManagerImpl.runValidationTask(BootstrapUpgradeManagerImpl.java:134)
at com.atlassian.bamboo.upgrade.BootstrapUpgradeManagerImpl.lambda$runValidationTasks$0(BootstrapUpgradeManagerImpl.java:93)
at com.atlassian.bamboo.upgrade.AbstractUpgradeManager.forEachTask(AbstractUpgradeManager.java:147)
at com.atlassian.bamboo.upgrade.BootstrapUpgradeManagerImpl.runValidationTasks(BootstrapUpgradeManagerImpl.java:93)
at com.atlassian.bamboo.setup.DefaultBootstrapManager.runValidationTasks(DefaultBootstrapManager.java:392)
at com.atlassian.bamboo.setup.DefaultBootstrapManager.performPersistenceUpgrade(DefaultBootstrapManager.java:296)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:81)
at com.atlassian.bamboo.setup.BootstrapLoaderListener.contextInitialized(BootstrapLoaderListener.java:143)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4689)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5155)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1412)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1402)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: remaining connection slots are reserved for non-replication superuser connections
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:512)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:59)
... 26 more
Caused by: org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2644)
at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:137)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:255)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217)
at org.postgresql.Driver.makeConnection(Driver.java:458)
at org.postgresql.Driver.connect(Driver.java:260)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
... 29 more
2021-09-17 09:15:19,575 FATAL [localhost-startStop-1] [DefaultBootstrapManager] Validation tests failed: [Task 3 failed with exception: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: remaining connection slots are reserved for non-replication superuser connections, Task 2902 failed:, Cannot open DB connection, Task 4300 failed:, Cannot open DB connection, Task 4410 failed:, Cannot open DB connection, Task 51111 failed with exception: null, Task 51302 failed:, Cannot open DB connection, Task 60601 failed:, Cannot open DB connection, Task 60602 failed:, Cannot open DB connection]The following message shows up when trying to access Bamboo from a web browser:
Bamboo bootstrap failed: Your Bamboo instance could not start because health check failed. Contact Atlassian Support at https://support.atlassian.comDiagnosis
Environment
Bamboo connected to PostgreSQL
Diagnostic Steps
By default PostgreSQL has max_connections set to 100. This is the setting that defines the maximum number of client connections allowed – but keep in mind the superuser_reserved_connections. You can check your current
max_connectionsby running the following statement inside your PostgreSQL database:SHOW max_connections;- Review the number of applications connected to your PostgreSQL server and how many connections they can open to the database.
Cause
When you have multiple applications connected to the same PostgreSQL server you need to account for the maximum number of connections they can all open together when setting up your PostgreSQL max_connections. For example, if you have Bamboo (db max connection pool: 100) running together with Jira (db max connection pool: 150) and Confluence (db max connection pool: 120) simultaneously connected to the same PostgreSQL server max_connections must be able to accommodate the maximum number of open connections from all applications simultaneously otherwise one (or more of the applications) might have trouble communicating to the database. In this example, if Jira and Confluence together reach 100 open connections there will be none left for Bamboo so you needed to have at least 373 connections to be able to accommodate the max of each application + 3 connections ("slots") reserved for the PostgreSQL superuser (superuser_reserved_connections). The default number of slots reserved for the PostgreSQL superuser is 3 but if you have customized that and changed this number you will have to take into account when updating your PostgreSQL max_connections.
Resolution
Review the max database connection pool of every application connected to your database server and adjust PostgreSQL max_connections to make sure it's capable of handling all of the connections coming from your applications (and don't forget about taking the superuser_reserved_connections into account as well).