Connection Timeout with MySQL or SQL Server Database
Problem
When installing Bitbucket Server or during the application usage, the following appears in the atlassian-bitbucket.log
depending on the database used.
MySQL Database
ERROR [main] com.jolbox.bonecp.ConnectionHandle Database access problem. Killing off all remaining connections in the connection pool. SQL State = 08S01
ERROR [main] o.h.transaction.JDBCTransaction JDBC begin failed
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 18,058 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
SQL Server Database
ERROR [BoneCP-pool-watch-thread] com.jolbox.bonecp.ConnectionHandle Database access problem. Killing off all remaining connections in the connection pool. SQL State = 08S01
ERROR [BoneCP-pool-watch-thread] com.jolbox.bonecp.PoolWatchThread Error in trying to obtain a connection. Retrying in 7000ms
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host <host_name>, port <port> has failed. Error: "connect timed out. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:190) ~[sqljdbc-4.0.2206.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:241) ~[sqljdbc-4.0.2206.jar:na]
Cause
MySQL Database
This problem is caused due to MySQL server timing-out database connections faster than the Bitbucket Server connection pool can notice. This exhausts the Bitbucket Server connection pool as it will keep its connections open while the MySQL side of the connections is already closed.
SQL Server Database
This problem is caused due to Bitbucket Server not being able to connect to SQL Server.
There may be various reasons for this message to occur, the following has been reported in the past:
- not enough space available on the server hosting the database
Resolution
MySQL Database
Bitbucket Server is shipped with a connection test interval default of 10 minutes.
MySQL has its wait_timeout variable default value set to 28800 seconds (8 hours).
Therefore, if both sides of the connection still keep the defaults, the problem will never happen, as MySQL will never timeout a connection before Bitbucket Server does it.
If the MySQL wait_timeout variable had its value reduced:
In this case, in order to avoid having MySQL close the connections and Bitbucket Server not notice, define the Bitbucket Server test interval value to one that is smaller than the MySQL value.
In theBITBUCKET_HOME/shared/bitbucket.properties
file, add the following line and define the number of minutes for the connection timeout:db.pool.idle.testInterval=1
If the MySQL wait_timeout is less than a minute, add the
interactiveClient=true
into the JDBC URL inbitbucket.properties
file to use MySQL interactive_timeout default value 28800 seconds, eg:jdbc.url=jdbc:mysql://localhost:3306/bitbucket212?autoReconnect=true&characterEncoding=utf8&useUnicode=true&sessionVariables=storage_engine%3DInnoDB&interactiveClient=true
Note that MySQL retains several scopes for the wait_timeout
and interactive_timeout
variable. Double check the same setting of the variable by running the following queries on the database:
- SHOW VARIABLES
- SHOW SESSION VARIABLES
- SHOW GLOBAL VARIABLES
SQL Server Database
The Microsoft support website provides a guide to troubleshoot these errors at https://support.microsoft.com/en-us/kb/942861.