Backup Client fails with Error reporting progress to Bitbucket

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

The backup client is failing with Error reporting progress to Bitbucket. 

Over the time it's noticed an increased length on the backup process until it gets to this stage.

Symtoms:

The following appears in the atlassian-sbc-YYYY-MM-DD-HHMM.log:

2015-11-23 01:30:32,481 INFO  [main] c.a.b.i.b.client.spring.SpringMain Initializing
2015-11-23 01:30:41,919 INFO  [main] c.a.b.i.b.c.RestBitbucketService Using Bitbucket 4.0.3
2015-11-23 01:30:42,529 INFO  [main] c.a.b.i.b.client.DefaultBackupClient Contacting Bitbucket
2015-11-23 01:30:42,779 INFO  [main] c.a.b.i.b.c.RestBitbucketService Bitbucket has been locked for maintenance. It may be unlocked with token: 923c6da4fd4ec41ffb2d994618265888d7c01981
2015-11-23 01:30:43,044 INFO  [main] c.a.b.i.b.c.RestBitbucketService Starting database backup on Bitbucket. It may be cancelled with token: 08462826515c57ab3b04b9d5156b0d7a8074763f

...
...
2015-11-23 02:02:55,832 INFO  [main] c.a.b.i.b.c.l.DefaultApplicationHome Verifying files in Bitbucket home "E:\Atlassian\ApplicationData\Bitbucket"
2015-11-23 02:02:55,910 INFO  [main] c.a.b.i.b.c.l.DefaultApplicationHome Verifying Bitbucket home
...
...
2015-11-23 05:49:50,207 INFO  [main] c.a.b.i.b.c.l.DefaultApplicationHome Verifying Bitbucket home
2015-11-23 06:00:45,454 INFO  [main] c.a.b.i.b.c.l.DefaultApplicationHome Backing up Bitbucket home
2015-11-23 06:00:50,501 INFO  [main] c.a.b.i.b.c.l.DefaultApplicationHome Backing up Bitbucket home
...
...
2015-11-23 06:40:48,348 INFO  [main] c.a.b.i.b.c.l.DefaultApplicationHome Backing up Bitbucket home
2015-11-23 06:41:53,834 WARN  [threadpool2] c.a.b.i.b.c.FixedRateProgressPublisher Error reporting progress to Bitbucket
org.apache.wink.client.ClientRuntimeException: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
	at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:241) ~[wink-client-1.4.jar:1.4]
	at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:189) ~[wink-client-1.4.jar:1.4]
	...
    ...
Caused by: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
	at org.apache.wink.client.internal.handlers.HttpURLConnectionHandler.handle(HttpURLConnectionHandler.java:57) ~[wink-client-1.4.jar:1.4]
	...
    ...
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_40]
	at java.net.SocketInputStream.socketRead(Unknown Source) ~[na:1.8.0_40]
	...
    ...


The following exception is seen on the atlassian-bitbucket.log:

2015-11-23 05:39:16,367 ERROR [http-nio-7990-exec-474] @1BRKN7Mx203x372629x46 172.16.1.170 "GET /mvc/maintenance HTTP/1.1" c.a.s.i.w.HandlerExceptionResolverImpl java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert
durch den Hostcomputer abgebrochen
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert
durch den Hostcomputer abgebrochen
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393) ~[catalina.jar:8.0.24]
	at net.bull.javamelody.FilterServletOutputStream.close(FilterServletOutputStream.java:59) ~[javamelody-1.56.0.jar:na]
	at com.atlassian.stash.internal.web.util.web.FilterServletOutputStream.close(FilterServletOutputStream.java:36) ~[FilterServletOutputStream.class:na]
	at sun.nio.cs.StreamEncoder.implClose(Unknown Source) ~[na:1.8.0_40]
	at sun.nio.cs.StreamEncoder.close(Unknown Source) ~[na:1.8.0_40]
	at java.io.OutputStreamWriter.close(Unknown Source) ~[na:1.8.0_40]
	at org.springframework.http.converter.json.GsonHttpMessageConverter.writeInternal(GsonHttpMessageConverter.java:205) ~[spring-web-4.1.7.RELEASE.jar:4.1.7.RELEASE]
	at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:208) ~[spring-web-4.1.7.RELEASE.jar:4.1.7.RELEASE]
	at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:71) ~[spring-web-4.1.7.RELEASE.jar:4.1.7.RELEASE]
	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94) [atlassian-trusted-apps-core-4.2.0.jar:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:87) [BeforeLoginPluginAuthenticationFilter.class:na]
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73) [BeforeLoginPluginAuthenticationFilter.class:na]
	at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:85) [bitbucket-service-impl-4.0.3.jar:na]
	at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38) [ConfigurableWebFilter.class:na]
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198) [javamelody-1.56.0.jar:na]
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176) [javamelody-1.56.0.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_40]
	at java.lang.Thread.run(Unknown Source) [na:1.8.0_40]
	... 122 frames trimmed
Caused by: java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert
durch den Hostcomputer abgebrochen
	at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[na:1.8.0_40]
	at sun.nio.ch.SocketDispatcher.write(Unknown Source) ~[na:1.8.0_40]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source) ~[na:1.8.0_40]
	at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.8.0_40]
	at sun.nio.ch.SocketChannelImpl.write(Unknown Source) ~[na:1.8.0_40]
	... 18 common frames omitted

Diagnosis

Diagnostic Steps

  • Application Check:
    • Generate a Support Zip (Administration -> Atlassian Support Tools -> Support Zip -> Create)
    • Verify how large your BITBUCKET_HOME disk is. An indication of such a problem (not necessarily) could be that the disk occupation is high. 
    • Count the number of files on <BITBUCKET_HOME>\shared\data\repositories
    • How many backup files exist in the directory where the backup client is storing its files? 
      • Files older than 2 or 3 days, that have a subsequent backup can safely be deleted
    • Are the backup files being stored on the Bitbucket server? If so they should be moved off immediately and stored somewhere else.

  • Generic Environment check:
    • How much RAM have you got on your server? If you have a VM could you add more RAM?
    • How much memory allocated for the JVM where Bitbucket server is running?
    • If the JVM set to the default of 768MB, and no git operations possible during a backup, what is consuming all the memory (if consumption is high)? 
      • Virus scanning? 
      • Security software? 
      • Other Atlassian applications on the same server
  • If you're on Windows:
    • Which user is the service or the manual startup using? Is this user a member of the Administrator group?
      •  If part of the Administration group, is it needs to be removed and should be a regular user.
    • Can we get perfmon data during the backup? Monitor CPU, Memory, and Disk IO?
    • You could try running "Disk Clean up and Optimization" on the drive where BITBUCKET_HOME is stored
    • You could disable any active antivirus and see how the backup goes
    • Old Git version -> Upgrade to Git for Windows 2.6.3 (https://git-for-windows.github.io/) or latest according to the Supported platforms documentation.

Cause

You can see that a long time is spent by the backup client Verifying BITBUCKET_HOME. On the logfile above, around 4 hours is spent on this stage.

This is usually caused by a large number of files on your repositories. I.e. the Backup client spends a long time going through all the individual files ( Verifying Bitbucket home step on the logs above).

Resolution

Review the Diagnostic Steps and make sure that the server hosting Bitbucket Server and the application itself has enough resources (see Supported platforms for details and Scaling Bitbucket Server).


Last modified on Nov 2, 2018

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.