For your Mirror server, you can only enable DEBUG level logging by modifying the $BITBUCKET_HOME/shared/bitbucket.properties to include the following:

logging.logger.com.atlassian.bitbucket.internal.mirroring=DEBUG

In a similar way, the following can be used to enable profiling log on the smart mirror:

logging.profiling.enabled=true

While logging.logger.com.atlassian.bitbucket.internal.mirroring is usually enough for the majority of the troubleshooting, the debug log level on more classes can be enabled by adding the following:

logging.debug.enabled=true

After applying any of these settings, a restart of the mirror is required.

Existing feature request

It is currently not possible to enable additional logging without a restart. This is an existing suggestion request to provide this possibility: BSERV-11198 - Getting issue details... STATUS

The mirror server is failing to register with the upstream and the following message is logged:

Registration with upstream server failed (Could not register with the configured upstream server. Response status was 409, message 'Conflict'). Retrying in Xs

Check that the upstream server has valid Data Center license installed. Smart Mirroring is currently a Data Center only feature.

The mirror administration panel shows the connected mirrors and/or mirroring requests. When clicking a mirror, the panel shows a spinner forever and never loads.

HAR File

• Have a look at the HAR file and Browser Console output - Generating HAR files and analyzing web requests
• This can tell you if the request is failing on the user -> mirror communication, or the primary -> mirror communication

JWT

• Requests to load the Mirroring Admin are JWT signed. The tokens have expiry timestamps embedded, and require the primary and mirror's clocks to be in sync.

• This can be identified from the HAR file above. Look for the following request. If the response of the mirror is a "HTTP 401 Unathorized", you're most likely running into this issue:

  GET https://bitbucket.mirror.my.domain/plugins/servlet/admin/mirror?tz=America%2FChicago&loc=en-US&user_id=admin&user_key=1&xdm_e=https%3A%2F%2Fbitbucket.mirror.my.domain&xdm_c=channel-bitbucket.mirror.b0ht-kwf9-alq1-zyiz__mirror-config-panel&cp=&lic=none&cv=1.1.86-bitbucket-04&jwt=<jwt_code> HTTP/1.1

• If authentication issues user -> mirror exist, ensure both systems' clocks are synchronized. ntpd

If the sync occurred, you must be able to browse through your primary server and when trying to retrieve the "Clone" URL, you should have a dropdown menu with the address of the clone so people in other locations are able to clone faster.

If that is not available, it means that the synchronisation didn't happen and here are some steps to troubleshoot:

• The following appears on your Mirror logs:

  2016-10-19 20:15:41,860 DEBUG [repository-mirror-sync:thread-2]  c.a.b.i.m.m.s.RepositoryFetchProcessor Exception details: 
  com.atlassian.bitbucket.scm.git.command.fetch.GitFetchAuthorisationException: The requested remote repository does not exist, or you do not have permission to access it

• If you go to you

  • SSH to your Mirror server

    # du -sh * $BITBUCKET_MIRROR_HOME/shared/data/repositories
     120K	1
     120K	2
     
    # $BITBUCKET_MIRROR_HOME/shared/data/repositories/1> ls -ltr
    total 32
    drwxr-xr-x 4 sl000cm sysint 4096 Oct 20 00:09 refs
    drwxr-xr-x 4 sl000cm sysint 4096 Oct 20 00:09 objects
    drwxr-xr-x 2 sl000cm sysint 4096 Oct 20 00:09 info
    drwxr-xr-x 4 sl000cm sysint 4096 Oct 20 00:09 hooks
    -rw-r--r-- 1 sl000cm sysint   23 Oct 20 00:09 HEAD
    -rw-r--r-- 1 sl000cm sysint   73 Oct 20 00:09 description
    -rw-r--r-- 1 sl000cm sysint  276 Oct 20 00:09 config
    drwxr-xr-x 2 sl000cm sysint 4096 Oct 20 00:09 branches
    -rw-r--r-- 1 sl000cm sysint    0 Oct 20 00:28 FETCH_HEAD

  • That means that the mirror has created the repositories you asked to mirror but is unable to fetch its data.

  • This usually happens because SSH is blocked from the mirrors. You could test connectivity by connecting to the mirror server:

    $ git clone ssh://git@primary.url.com/<PROJECT_ID>/<REPO_SLUG>/<REPO_NAME>.git
    $ telnet primary.url.com 7999
     

    Where the clone URL can be obtained from browsing the "Clone" shortcut on the UI of your Bitbucket server

  • If the git clone hangs or there is no response from the telnet command, there is a network issue and you should investigate your firewall between the mirror and the primary.

SSL Certificate

• Check that the mirror and primary have valid SSL certificates. These certificates need to be CA signed.
  • If they are self-signed certificate, you need to get a CA certificate.
  • As a workaround for self-signed certificates you can follow this KB: Unable to connect to SSL services due to "PKIX Path Building Failed" error

• DEBUG level logging on the mirror will reveal SSL issues on attempted sync

  Enable DEBUG logging on Mirror

  For your Mirror server, you can only enable DEBUG level logging by modifying the $BITBUCKET_HOME/shared/bitbucket.properties to include the following:

  logging.logger.com.atlassian.bitbucket.internal.mirroring=DEBUG

  A restart of the mirror is required.

Network Config

• Check the mirror and primary can communicate at a system level
• curl -v https://<primary_url>/status from the mirror should return a 200 OK status code
  • Visa versa for the primary -> mirror

The primary nodes that receive the commits (i.e. when someone pushes to repos) or where you create new repos or projects are actively listening for those events to happen. Once those events are detected, the primary server will call a POST on a URL to the mirror. The Mirror in turn will try to fetch changes and populate its repository.

If you are experiencing that there is a delay in updating the mirrors repositories, amongst other reasons, it could be that the primary server is failing to listen to events in the first place and that the Mirror is updating itself (we have a mechanism implemented so the Mirrors will try to fetch the primary every 15 minutes) . If that's happening, you will likely find the RejectedExecutionException on one of the primary nodes:

2016-09-21 09:39:05,881 ERROR [AtlassianEvent::thread-2] *GVR0VMx578x1021320x19 10.193.64.223,10.192.10.8 "POST /rest/mirroring/latest/authenticate HTTP/1.1" c.a.e.i.LockFreeEventPublisher$Publisher There was an exception thrown trying to dispatch event 'com.atlassian.bitbucket.event.audit.AuditEvent[source=com.atlassian.stash.internal.audit.AuthenticationEventListener@7e0a7b85]' from the invoker 'SingleParameterMethodListenerInvoker{method=public void com.atlassian.plugins.shortcuts.internal.DefaultKeyboardShortcutManager.handleEvent(java.lang.Object), listener=com.atlassian.plugins.shortcuts.internal.DefaultKeyboardShortcutManager@14da6839}'.
java.util.concurrent.RejectedExecutionException: Task com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable@7639a428 rejected from com.atlassian.stash.internal.event.EventThreadPoolExecutor@56c25f4a[Shutting down, pool size = 16, active threads = 16, queued tasks = 1790, completed tasks = 9850331]

We've seen specifically the Code Navigator for Bitbucket (Stash) causing this issue. However in other to troubleshoot what could be causing this exception, please go through the KB above. Here is the stack trace in which we detected the plugin above causing the problem:

2016-09-28 03:42:28,863 DEBUG [AtlassianEvent::thread-11] jenkins *3N3D10x222x2329424x0 13t38fr 10.192.10.14 SSH - git-receive-pack 'game/dev.git' c.m.s.p.codenav.indexer.TagsUpdater Skipping GAME/dev:refs/heads/ody-a (doesn't match HEAD|(refs/heads/(master|develop)))
2016-09-28 03:42:30,048 ERROR [threadpool:thread-20]  c.a.s.i.c.StateTransferringExecutor Error while processing asynchronous task
java.lang.StackOverflowError: null

Pull request refs (under refs/pull-requests/) are an implementation detail of Bitbucket Server. They are not intended to be used for CI, or generally relied upon for development. This comment by Bryan Turner lists some of the reasons why it might not be a good idea to build these refs in CI.

As a result of the previous decisions not to eagerly create these refs (they are only created and updated when a pull request is viewed), we made the decision not to create webhook events for mirrors to sync these refs in real time. This is in line with the behavior on the primary Bitbucket Server, where no events are fired when these refs are created or updated.

The scheduled synchronization job that will cause the mirror to do a full sync of its configured projects every 15 minutes (by default) also does not synchronize these refs.

The refs/pull-requests are only synchronized when a push is performed. When the mirror fetches the related changed, then also these ones are synchronized.

Smart Mirrors will sync refs/heads/ and refs/tags/ in real time, by firing webhooks to the configured mirror(s) when updates to these refs are made.