Bitbucket on AWS Amazon Linux instances fail to start due to the Hotpatch for Apache Log4j tool

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Summary

AWS started installing a Log4jHotPatch tool for JDKs shipped on Amazon Linux instances on 17 December 2021 as a means to address the CVE-2021-44228 security vulnerability with Log4j.

The tool injects a Java agent into a running JVM process and adds an "agent" jar file in Bitbucket's class path.

The jar file gets removed once Log4jHotPatch has been loaded. This causes an error during Bitbucket start up because it is unable to find the jar file.

References: 


Environment

Bitbucket Server and Data Center on AWS Amazon Linux

Diagnosis

The atlassian-bitbucket.log file shows the following error during start up:

2021-12-20 20:48:26,913 ERROR [spring-startup] l.servicelocator.ServiceLocator Cannot build ServiceLocator
liquibase.exception.UnexpectedLiquibaseException: java.io.FileNotFoundException: /tmp/agent1234567890123456789.jar (No such file or directory)
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:129)
at liquibase.servicelocator.ServiceLocator.<init>(ServiceLocator.java:59)
at liquibase.servicelocator.ServiceLocator.<clinit>(ServiceLocator.java:43)
at liquibase.database.DatabaseFactory.<init>(DatabaseFactory.java:28)
at liquibase.database.DatabaseFactory.getInstance(DatabaseFactory.java:46)
at liquibase.integration.spring.SpringLiquibase.createDatabase(SpringLiquibase.java:410)
at liquibase.database.ExtendedSpringLiquibase.createDatabase(ExtendedSpringLiquibase.java:57)
at liquibase.integration.spring.SpringLiquibase.createLiquibase(SpringLiquibase.java:375)
at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:36)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.lang.Thread.run(Thread.java:748)
... 63 frames trimmed
Caused by: java.io.FileNotFoundException: /tmp/agent1234567890123456789.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:228)
at java.util.zip.ZipFile.<init>(ZipFile.java:157)
at java.util.jar.JarFile.<init>(JarFile.java:171)
at java.util.jar.JarFile.<init>(JarFile.java:108)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:152)
at liquibase.resource.ClassLoaderResourceAccessor.getResourcesAsStream(ClassLoaderResourceAccessor.java:53)
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:115)
... 13 common frames omitted
  • In the sample stack trace above, the jar file that could not be found is: /tmp/agent1234567890123456789.jar
  • The numeric portion in the filename changes for each restart


Resolution

  • Disable the Log4jHotPatch tool by calling:

    sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill
  • Afterwards, start up Bitbucket


Last modified on Dec 21, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.