Users can't login to Bitbucket Server - LDAP response read timed out
Problem
Users are unable to login to Bitbucket Server.
One of the following stacktrace appears in the atlassian-bitbucket.log
:
2014-08-26 22:26:35,892 ERROR [clusterScheduler_Worker-2] c.a.c.d.DbCachingDirectoryPoller Error occurred while refreshing the cache for directory [ 229377 ].
com.atlassian.crowd.exception.OperationFailedException: Error looking up attributes for highestCommittedUSN
...
Caused by: org.springframework.ldap.UncategorizedLdapException: Uncategorized exception occured during LDAP processing; nested exception is javax.naming.NamingException: LDAP response read timed out, timeout used:120000ms.; remaining name '/'
...
Caused by: javax.naming.NamingException: LDAP response read timed out, timeout used:120000ms.
WARN [http-nio-7990-exec-128] @1UQDXDx1198x6757x85 <ip_address> "GET /scm/<project_slug>/<repository_key>.git/info/refs HTTP/1.1" c.a.s.i.s.s.PluginAuthenticationProvider Could not authenticate <username>
com.atlassian.bitbucket.auth.AuthenticationSystemException: The remote authentication server is not available. Please try again later.
...
Caused by: javax.naming.CommunicationException: <url>:<port>
...
Caused by: java.net.ConnectException: Connection timed out
Cause
There can be multiple causes for this error:
- The LDAP directory is too huge and Bitbucket Server fails to find all of the users before the timeout.
- There have been cases where the enabled 'Follow Referral' option causes the same behavior.
- Stale connections are not being closed and when used the next time, they immediately fail
Resolution
1. Huge user directory
a. Increase LDAP Read Timeout
- Go to
Administration > User Directories
- Edit the LDAP directory
- Increase the value of Read Timeout
- Go to
b. Reduce the scope of the user/group search by adding filters
Connect to an LDAP directory see "Option - Use LDAP filters to restrict the number of users and groups that are synchronised"
c. Reduce the scope of the user/group search by adding Additional User / Group DNs
Connecting Bitbucket Server to an existing LDAP directory see "LDAP Schema"
Additional User DN | This value is used in addition to the base DN when searching and loading users. If no value is supplied, the subtree search will start from the base DN. Example:
|
Additional Group DN | This value is used in addition to the base DN when searching and loading groups. If no value is supplied, the subtree search will start from the base DN. Example:
|
d. Consider adding multiple directories differing only in Additional DNs
e. Consider switching to using delegated LDAP authentication
2. Disable Follow Referral
- Go to
Administration > User Directories
- Edit the LDAP directory
- Disable the Follow Referral option
3. Set a Timeout to Automatically Close Stale LDAP Connections in the Pool
- Stop Bitbucket Server
Modify
<Bitbucket Server Installation>/bin/setenv.sh
to add the following parameter to theJVM_SUPPORT_RECOMMENDED_ARGS
:JVM_SUPPORT_RECOMMENDED_ARGS="-Dcom.sun.jndi.ldap.connect.pool.timeout=300000"
- This will timeout idle LDAP connections from the pool after 5 minutes and help clear out any stale connections from being reused
setenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0,
setenv.sh
andsetenv.bat
have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.Linux
When using the
atlbitbucket
service on Linux, the environment variables are ignored. You must set the parameters in_start-webapp.sh (or start-bitbucket.sh)
. These values will be read when the service starts.As an example, to set
JVM_SUPPORT_RECOMMENDED_ARGS
, you would add this line to the file:ExampleJVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryError
Windows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set
JVM_SUPPORT_RECOMMENDED_ARGS
, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.- Start Bitbucket Server
Further Reading
Refer to the Common User Management Errors guide for a more detailed listing of some common LDAP error codes.