Bitbucket Server Mirror becomes unresponsive when system clock is not synchronised with upstream cluster nodes
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bitbucket Mirror may become unresponsive and operations such as git clone or git fetch may be failing - for example:
1
2
3
4
5
6
7
8
9
10
11
12
$ git clone ssh://git@mirror.example.com:7999/bitbucket/my_project/my_repo.git
(...)
The authenticity of host '[mirror.example.com]:7999 ([ww.xx.yy.zz]:7999)' can't be established.
RSA key fingerprint is SHA256:EkCPi2FT6t/Rgh9KRm6UjrABbVRPsVb26sNtie5E4lR.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[mirror.example.com]:7999,[ww.xx.yy.zz]:7999' (RSA) to the list of known hosts.
git@mirror.example.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.Environment
This issue has been seen in multiple versions of Bitbucket Server.
Diagnosis
The Bitbucket Mirror's application log (atlassian-bitbucket.log) may include sequences similar to the following:
1
2
3
4
5
6
2020-12-11 19:19:22,138 WARN [http-nio-8080-exec-207] @15KJ6TZx1159x1027941x2 <list_of_IP_addresses> "POST /rest/mirroring/latest/upstreamServers/a693f399-13bf-34a9-9b1e-5c9613e5939d/addon/events HTTP/1.1" c.a.j.i.s.DefaultAuthenticationResultHandler Failure during JWT authentication
com.atlassian.jwt.exception.JwtExpiredException: Expired at Fri Dec 11 19:15:53 IST 2020 and time is now Fri Dec 11 19:19:22 IST 2020 (30 seconds leeway is allowed)
...
2021-11-02 16:50:08,695 WARN [main] c.a.b.i.m.m.DefaultUpstreamAnalyticsService Upstream and mirror clocks are out of sync by 60s, check the system clocks on the upstream and mirror nodes. This can sometimes cause JWT authentication to failCause
The Bitbucket Mirror's system clock is not synchronised with the system clocks of the upstream cluster nodes.
Solution
Check the date/time configuration on the Mirror server and on the servers of the upstream cluster nodes. Confirm that system clocks are being correctly synchronised with an NTP server, or some other mechanism, to ensure they match the current time.
See ntpd or chrony for an example program used to maintain the time on Linux systems.
Make sure the time difference between the servers does not exceed 30 seconds.
Was this helpful?