Bitbucket Server fails to start after upgrade
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bitbucket fails to start after upgrade. 503: Service not available errors while trying to access Bitbucket on the browser
Diagnosis
After upgrading, you see that Bitbucket restarts with the new version:
2021-06-07 06:23:28,419 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v7.13.0 on hostname.company.com with PID 21112 (/opt/atlassian/bitbucket/7.13.0/app/WEB-INF/classes started by atlbitbucket in /)
2021-06-07 06:23:30,532 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 3.709 seconds (JVM running for 4.615)
2021-06-07 06:23:33,784 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucketpost which you might see the following indexing related errors in the atlassian-bitbucket.log file from the day of the issue:
2021-06-07 06:25:02,465 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate index
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
... 1 common frames omitted
2021-06-07 06:25:02,467 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]]''
2021-06-07 06:25:02,468 ERROR [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Update path and filename fields for file search]
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]]
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:52)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:116)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Thread.java:748)
2021-06-07 06:25:02,468 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.after a few minutes Bitbucket shuts down:
2021-06-07 06:25:48,414 INFO [SpringContextShutdownHook] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /shared_bitbucket
2021-06-07 06:25:48,428 INFO [SpringContextShutdownHook] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket
2021-06-07 06:25:48,552 INFO [SpringContextShutdownHook] c.a.b.i.boot.log.BuildInfoLogger Bitbucket 7.13.0 has shut downFor the Caused by: java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-0 errors, check the connection between Bitbucket and the Elasticsearch node, if they are able to reach each other, by running ping or telnet commands.
Cause
One of the possible causes of this issue is firewall getting enabled during upgrade, in case OS updates are also performed.
Solution
You should be able to access Bitbucket Server after disabling the firewall. This can happen if during the upgrade the OS update was also done and that installs some security patches, which would have enabled the firewall.