Clone fails with "exited with code -1,002" error when Datadog Java agent is embedded in JVM

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

 

Summary

Clone operations will intermittently fail to complete when using Datadog Java agent to monitor application stats.

Environment

  • Bitbucket 7.5.X
  • Datadog Java agent 7.2.0 with dd-trace-java.jar 0.64.0

Diagnosis

When the clone operation fails the following messages are found in the git client machine and in Bitbucket logs.

Git client log
git clone ssh://git@<BITBUCKET_BASE_URL>:7999/test_project/Test-repo.git
Cloning into 'test_repo'...
remote: Enumerating objects: 1357, done.
remote: Counting objects: 100% (1357/1357), done.
remote: Compressing objects: 100% (4456/4456), done.
error: git upload-pack: git-pack-objects died with error. KiB/s
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
atlassian-bitbucket.log
2020-10-05 00:08:14,952 WARN  [git-hook-callback:thread-2] test_user @1GTT9VBM7x636502x1 1ygpuz7 192.168.X.X,0:0:0:0:0:0:0:1 "POST /scm/test_project/test_repo.git/git-upload-pack HTTP/1.0" c.a.s.i.s.g.h.DefaultGitHookService git-pack-objects: Hook handling failed
com.atlassian.bitbucket.scm.CommandFailedException: '/usr/bin/git pack-objects --revs --thin --stdout --progress --delta-base-offset' exited with code -1,002
	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onError(DefaultCommandExitHandler.java:43)
	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:111)
	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:285)
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.finish(NioNuProcessHandler.java:326)
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.asResult(NioNuProcessHandler.java:241)
	at com.atlassian.bitbucket.internal.process.nu.NuNioProcessHelper.run(NuNioProcessHelper.java:77)
	at com.atlassian.bitbucket.internal.process.NioCommand.call(NioCommand.java:52)
	at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.streamFromGit(CachingPackObjectsHookHandler.java:273)
	at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.access$000(CachingPackObjectsHookHandler.java:63)
	at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler$2.apply(CachingPackObjectsHookHandler.java:208)
	at com.atlassian.util.contentcache.internal.AbstractCacheEntry$BaseAccess.streamToCache(AbstractCacheEntry.java:362)
	at com.atlassian.util.contentcache.internal.AbstractCacheEntry$CacheEntryAccess.stream(AbstractCacheEntry.java:452)
	at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.streamFromCache(CachingPackObjectsHookHandler.java:245)
	at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.maybeHandle(CachingPackObjectsHookHandler.java:193)
	at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.handle(CachingPackObjectsHookHandler.java:95)
	at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.doHandleRequest(DefaultGitHookService.java:276)
	at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.handleRequest(DefaultGitHookService.java:233)
	at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.handleRawRequest(DefaultGitHookService.java:194)
	at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService$1.lambda$run$0(DefaultGitHookService.java:154)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.lang.Thread.run(Thread.java:834)
	... 1 frame trimmed
Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: -1002
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:277)
	... 19 common frames omitted

Followed by

atlassian-bitbucket.log
2020-10-05 00:08:15,321 ERROR [http-scmrequest-handler:thread-8] test_user @1GTT9VBM7x636502x1 1hclqk0 192.168.X.X,0:0:0:0:0:0:0:1 "POST /scm/test_project/test_repo.git/git-upload-pack HTTP/1.0" c.a.s.i.s.g.p.h.GitSmartExitHandler test_project/test_repo[194]: Read request from 192.168.X.X failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 1
The following was written to stderr:
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.

Datadog will also throw the following error for multiple Bitbucket packages.

launcher.log
[dd.trace 2020-10-03 21:15:17:942 -0400] [spring-startup] ERROR datadog.trace.agent.tooling.HelperInjector - Error preparing helpers while processing class com.atlassian.plugins.impl.servlet.AdminServlet for Servlet3Instrumentation. Failed to inject helper classes into instance com.atlassian.plugins.static-assets-url [113]
java.lang.IllegalStateException: Error invoking (accessor)::defineClass
    at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$UsingUnsafeInjection.defineClass(ClassInjector.java:922)

Cause

Datadog Java agent fails to inject helper classes into Bitbucket packages causing git hook to fail and impacting clone operations.

Workaround

  1. Remove Datadog JVM parameters, shown below, from the instance and restart Bitbucket;

    1. These parameters are usually set in the <Bitbucket_Install>/bin/_start-webapp.sh

      -javaagent:/opt/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.service=<SERVICE_NAME> -Ddd.env=<ENV> -Datlassian.org.osgi.framework.bootdelegation.extra=datadog.*
  2. If needed, engage with Datadog Support for further details and troubleshooting.




Last modified on Oct 28, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.