Size calculation failed warnings in Bitbucket logs
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Sometimes there are multiple occurrences of the below warning in the atlassian-bitbucket.log
1
2
2022-06-24 08:01:19,381 WARN [git:gc:thread-1] c.a.s.i.r.DefaultRepositorySizeCache CENSORED-PROJECT/censored-repo[4047]: Size calculation failed
com.atlassian.bitbucket.ServerException: '/usr/bin/git count-objects -v' timed out; exited with code 143Even though this error does not have any direct impact to the application performance, this could cause the logs to get filled up soon.
Environment
Bitbucket Version 7.14 +
Diagnosis
This is the full sample stack trace.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
2022-06-24 08:01:19,381 WARN [git:gc:thread-1] c.a.s.i.r.DefaultRepositorySizeCache CENSORED-PROJECT/censored-repo[4047]: Size calculation failed
com.atlassian.bitbucket.ServerException: '/usr/bin/git count-objects -v' timed out; exited with code 143
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:125)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:239)
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:340)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.finish(NioNuProcessHandler.java:381)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.onExit(NioNuProcessHandler.java:136)
at com.zaxxer.nuprocess.internal.BasePosixProcess.onExit(BasePosixProcess.java:315)
at com.zaxxer.nuprocess.linux.ProcessEpoll.handleExit(ProcessEpoll.java:365)
at com.zaxxer.nuprocess.linux.ProcessEpoll.cleanupProcess(ProcessEpoll.java:318)
at com.zaxxer.nuprocess.linux.ProcessEpoll.process(ProcessEpoll.java:253)
at com.zaxxer.nuprocess.internal.BaseEventProcessor.run(BaseEventProcessor.java:82)
at com.zaxxer.nuprocess.linux.ProcessEpoll.run(ProcessEpoll.java:154)
at com.zaxxer.nuprocess.linux.LinuxProcess.run(LinuxProcess.java:110)
at com.zaxxer.nuprocess.linux.LinProcessFactory.runProcess(LinProcessFactory.java:50)
at com.zaxxer.nuprocess.NuProcessBuilder.run(NuProcessBuilder.java:273)
at com.atlassian.bitbucket.internal.process.nu.NuNioProcessHelper.run(NuNioProcessHelper.java:76)
at com.atlassian.bitbucket.internal.process.NioCommand.call(NioCommand.java:46)
at com.atlassian.stash.internal.repository.DefaultRepositorySizeCache.updateSize(DefaultRepositorySizeCache.java:181)
at com.atlassian.stash.internal.repository.DefaultRepositorySizeCache.refreshSingle(DefaultRepositorySizeCache.java:155)
at com.atlassian.stash.internal.repository.DefaultRepositorySizeCache.lambda$null$3(DefaultRepositorySizeCache.java:123)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at com.atlassian.stash.internal.repository.DefaultRepositorySizeCache.lambda$maybeRefresh$4(DefaultRepositorySizeCache.java:120)
at com.atlassian.stash.internal.concurrent.ClusterMappedLock.withLock(ClusterMappedLock.java:49)
at com.atlassian.stash.internal.concurrent.ClusterRepositoryLock.withLock(ClusterRepositoryLock.java:29)
at com.atlassian.stash.internal.repository.DefaultRepositorySizeCache.maybeRefresh(DefaultRepositorySizeCache.java:119)
at com.atlassian.stash.internal.scm.git.gc.DefaultGitGarbageTruck$GcOperation.perform(DefaultGitGarbageTruck.java:492)
at com.atlassian.stash.internal.scm.git.gc.DefaultGitGarbageTruck$GcOperation.perform(DefaultGitGarbageTruck.java:444)
at com.atlassian.stash.internal.scm.git.gc.DefaultGitGarbageTruck.lambda$withLock$4(DefaultGitGarbageTruck.java:365)
at com.atlassian.bitbucket.mesh.io.UnderLock$DefaultLockedFile.edit(UnderLock.java:491)
at com.atlassian.bitbucket.mesh.io.UnderLock.edit(UnderLock.java:69)
at com.atlassian.stash.internal.scm.git.gc.DefaultGitGarbageTruck.withLock(DefaultGitGarbageTruck.java:360)
at com.atlassian.stash.internal.scm.git.gc.DefaultGitGarbageTruck.collectGarbage(DefaultGitGarbageTruck.java:234)
at com.atlassian.stash.internal.scm.git.gc.DefaultGitGarbageTruck$GcBootstrapper.runGc(DefaultGitGarbageTruck.java:424)
at com.atlassian.stash.internal.scm.git.gc.DefaultGitGarbageTruck$GcBootstrapper.run(DefaultGitGarbageTruck.java:408)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.lang.Thread.run(Thread.java:829)
... 1 frame trimmed
Caused by: com.atlassian.utils.process.ProcessTimeoutException: process timed out
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:324)
... 36 common frames omittedBrowsing to the particular repository that's throwing the error and executing the git command manually will not have any slowness or get timed out.
1
2
cd <bitbucket-home>/shared/data/repositories/<repo-id>
git count-objects -v --human-readableCause
The size calculation is a new feature of Bitbucket that got introduced in Bitbucket 7.14, and it replaces the "approximate repository size" that was previously presented in the web interface (on the Repository settings page). The current versions of Bitbucket calculate the size of each Git repository once, and store it in the database, for quick access whenever users want to see it. However, to prevent overloading the server, this calculation is subject to timeout. When the timeout is reached, this low-priority task is cancelled, to make sure that the server doesn't spend too much time just calculating the size of each repository.
Since this is a different process than the regular git commands that are being executed by Bitbucket, it has a different timeout setting. The default timeout for this is 5 seconds, which explains why we see the error often.
Solution
The message indicating a timeout in repository size calculation is not problematic unless accompanied by other issues. The calculation will be eventually retried after subsequent changes to the repository, such as another push.
The status of these jobs can be monitored by enabling trace logging for the below relevant classes.
DefaultRepositorySizeCache
RepositorySizeBackfillJob
You can use the below commands to enable the trace logging
1
2
3
curl -u admin -v -X PUT -d "" -H "Content-Type: application/json" <bitbucket_url>/rest/api/latest/logs/logger/com.atlassian.stash.internal.repository.DefaultRepositorySizeCache/trace
curl -u admin -v -X PUT -d "" -H "Content-Type: application/json" <bitbucket_url>/rest/api/latest/logs/logger/com.atlassian.stash.internal.repository.RepositorySizeBackfillJob/traceYou should be seeing entries similar to the one below in the atlassian-bitbucket.log
1
2
3
4
2021-06-09 21:12:19,892 TRACE [Caesium-1-3] c.a.s.i.r.RepositorySizeBackfillJob Acquired lock to perform backfilling of repository size cache
2021-06-09 21:12:19,893 DEBUG [Caesium-1-3] c.a.s.i.r.RepositorySizeBackfillJob Starting backfilling of repository size cache
2021-06-09 21:12:19,935 TRACE [Caesium-1-3] c.a.s.i.r.DefaultRepositorySizeCache PROJECT_1/rep_1[1]: Refreshing size
2021-06-09 21:12:20,065 TRACE [Caesium-1-3] c.a.s.i.r.DefaultRepositorySizeCache PROJECT_1/zdu[12]: Refreshing size*The entries may not be as frequent as the ones in example. Also enabling the trace logs can impact application performance and may fill up disk space due to detailed logs.
Command to disable the trace logging:
1
2
3
curl -u admin -v -X PUT -d "" -H "Content-Type: application/json" <bitbucket_url>/rest/api/latest/logs/logger/com.atlassian.stash.internal.repository.DefaultRepositorySizeCache/warn
curl -u admin -v -X PUT -d "" -H "Content-Type: application/json" <bitbucket_url>/rest/api/latest/logs/logger/com.atlassian.stash.internal.repository.RepositorySizeBackfillJob/warnAlternatively, you can query the below table in the DB also to see the last updated time for each repo
1
select * from bb_repo_size;Was this helpful?