Confluence is blocked because of many connection errors
Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.
Problem
Confluence connects to MySQL database. All of a sudden Confluence becomes inaccessible and the following message appears in the browser:
Error displayed:
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.NullPointerException
com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getGlobalSettings(ConfluenceEncodingFilter.java:45)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getEncodingInternal(ConfluenceEncodingFilter.java:35)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getEncoding(ConfluenceEncodingFilter.java:27)
com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:38)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:65)
root cause
java.lang.NullPointerException
The following appears in the atlassian-confluence.log
2015-02-24 18:25:05,509 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] [mchange.v2.resourcepool.BasicResourcePool] run com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@171f6647 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
java.sql.SQLException: null, message from server: "Host 'confluence' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1072)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:986)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:981)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1104)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2183)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:784)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.GeneratedConstructorAccessor30.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:354)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:284)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Cause
MySQL Server blocked the connection of the database to Confluence.
Resolution
This issue originates with the database server so can be resolved by following steps:
- Shut down Confluence.
- Run mysqladmin flush-hosts on the server running MySQL server.
To avoid this from showing up again:
Increase the max_connect_errors value to a higher value, for example 200. This can be done by editing the file etc/my.cnf (Linux) or my.ini (Windows) configuration file and adding the line set-variable = max_connect_errors=200 below the [mysqld] tag. An example file is shown below:
[mysqld]
datadir=/backup/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
set-variable = max_connections=500
set-variable = max_connect_errors=200
old_passwords=1
safe-show-database
skip-locking
skip-innodb
max_allowed_packet = 1M
set-variable = table_cache=1024
set-variable = query_cache_type=1
set-variable = query_cache_limit=1M
set-variable = query_cache_size=128M ## 32MB for every 1GB of RAM
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
[mysql]
socket=/var/lib/mysql/mysql.sock
[mysqladmin]
socket=/var/lib/mysql/mysql.sock
[mysqldump]
socket=/var/lib/mysql/mysql.sock