Clone fails with "exited with code -1,002" error when Datadog Java agent is embedded in JVM
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Clone operations will intermittently fail to complete when using Datadog Java agent to monitor application stats.
Environment
- Bitbucket 7.5.X
- Datadog Java agent 7.2.0 with dd-trace-java.jar 0.64.0
Diagnosis
When the clone operation fails the following messages are found in the git client machine and in Bitbucket logs.
Git client log
git clone ssh://git@<BITBUCKET_BASE_URL>:7999/test_project/Test-repo.git
Cloning into 'test_repo'...
remote: Enumerating objects: 1357, done.
remote: Counting objects: 100% (1357/1357), done.
remote: Compressing objects: 100% (4456/4456), done.
error: git upload-pack: git-pack-objects died with error. KiB/s
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
atlassian-bitbucket.log
2020-10-05 00:08:14,952 WARN [git-hook-callback:thread-2] test_user @1GTT9VBM7x636502x1 1ygpuz7 192.168.X.X,0:0:0:0:0:0:0:1 "POST /scm/test_project/test_repo.git/git-upload-pack HTTP/1.0" c.a.s.i.s.g.h.DefaultGitHookService git-pack-objects: Hook handling failed
com.atlassian.bitbucket.scm.CommandFailedException: '/usr/bin/git pack-objects --revs --thin --stdout --progress --delta-base-offset' exited with code -1,002
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onError(DefaultCommandExitHandler.java:43)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:111)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:285)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.finish(NioNuProcessHandler.java:326)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.asResult(NioNuProcessHandler.java:241)
at com.atlassian.bitbucket.internal.process.nu.NuNioProcessHelper.run(NuNioProcessHelper.java:77)
at com.atlassian.bitbucket.internal.process.NioCommand.call(NioCommand.java:52)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.streamFromGit(CachingPackObjectsHookHandler.java:273)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.access$000(CachingPackObjectsHookHandler.java:63)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler$2.apply(CachingPackObjectsHookHandler.java:208)
at com.atlassian.util.contentcache.internal.AbstractCacheEntry$BaseAccess.streamToCache(AbstractCacheEntry.java:362)
at com.atlassian.util.contentcache.internal.AbstractCacheEntry$CacheEntryAccess.stream(AbstractCacheEntry.java:452)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.streamFromCache(CachingPackObjectsHookHandler.java:245)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.maybeHandle(CachingPackObjectsHookHandler.java:193)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.handle(CachingPackObjectsHookHandler.java:95)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.doHandleRequest(DefaultGitHookService.java:276)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.handleRequest(DefaultGitHookService.java:233)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.handleRawRequest(DefaultGitHookService.java:194)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService$1.lambda$run$0(DefaultGitHookService.java:154)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.lang.Thread.run(Thread.java:834)
... 1 frame trimmed
Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: -1002
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:277)
... 19 common frames omitted
Followed by
atlassian-bitbucket.log
2020-10-05 00:08:15,321 ERROR [http-scmrequest-handler:thread-8] test_user @1GTT9VBM7x636502x1 1hclqk0 192.168.X.X,0:0:0:0:0:0:0:1 "POST /scm/test_project/test_repo.git/git-upload-pack HTTP/1.0" c.a.s.i.s.g.p.h.GitSmartExitHandler test_project/test_repo[194]: Read request from 192.168.X.X failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 1
The following was written to stderr:
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.Datadog will also throw the following error for multiple Bitbucket packages.
launcher.log
[dd.trace 2020-10-03 21:15:17:942 -0400] [spring-startup] ERROR datadog.trace.agent.tooling.HelperInjector - Error preparing helpers while processing class com.atlassian.plugins.impl.servlet.AdminServlet for Servlet3Instrumentation. Failed to inject helper classes into instance com.atlassian.plugins.static-assets-url [113]
java.lang.IllegalStateException: Error invoking (accessor)::defineClass
at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$UsingUnsafeInjection.defineClass(ClassInjector.java:922)Cause
Datadog Java agent fails to inject helper classes into Bitbucket packages causing git hook to fail and impacting clone operations.
Workaround
Remove Datadog JVM parameters, shown below, from the instance and restart Bitbucket;
These parameters are usually set in the <Bitbucket_Install>/bin/_start-webapp.sh
-javaagent:/opt/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.service=<SERVICE_NAME> -Ddd.env=<ENV> -Datlassian.org.osgi.framework.bootdelegation.extra=datadog.*
- If needed, engage with Datadog Support for further details and troubleshooting.
Last modified on Oct 28, 2020
Powered by Confluence and Scroll Viewport.