Bitbucket on AWS Amazon Linux instances fail to start due to the Hotpatch for Apache Log4j tool
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
AWS started installing a Log4jHotPatch tool for JDKs shipped on Amazon Linux instances on 17 December 2021 as a means to address the CVE-2021-44228 security vulnerability with Log4j.
The tool injects a Java agent into a running JVM process and adds an "agent" jar file in Bitbucket's class path.
The jar file gets removed once Log4jHotPatch has been loaded. This causes an error during Bitbucket start up because it is unable to find the jar file.
References:
Environment
Bitbucket Server and Data Center on AWS Amazon Linux
Diagnosis
The atlassian-bitbucket.log file shows the following error during start up:
2021-12-20 20:48:26,913 ERROR [spring-startup] l.servicelocator.ServiceLocator Cannot build ServiceLocator
liquibase.exception.UnexpectedLiquibaseException: java.io.FileNotFoundException: /tmp/agent1234567890123456789.jar (No such file or directory)
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:129)
at liquibase.servicelocator.ServiceLocator.<init>(ServiceLocator.java:59)
at liquibase.servicelocator.ServiceLocator.<clinit>(ServiceLocator.java:43)
at liquibase.database.DatabaseFactory.<init>(DatabaseFactory.java:28)
at liquibase.database.DatabaseFactory.getInstance(DatabaseFactory.java:46)
at liquibase.integration.spring.SpringLiquibase.createDatabase(SpringLiquibase.java:410)
at liquibase.database.ExtendedSpringLiquibase.createDatabase(ExtendedSpringLiquibase.java:57)
at liquibase.integration.spring.SpringLiquibase.createLiquibase(SpringLiquibase.java:375)
at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:36)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.lang.Thread.run(Thread.java:748)
... 63 frames trimmed
Caused by: java.io.FileNotFoundException: /tmp/agent1234567890123456789.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:228)
at java.util.zip.ZipFile.<init>(ZipFile.java:157)
at java.util.jar.JarFile.<init>(JarFile.java:171)
at java.util.jar.JarFile.<init>(JarFile.java:108)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:152)
at liquibase.resource.ClassLoaderResourceAccessor.getResourcesAsStream(ClassLoaderResourceAccessor.java:53)
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:115)
... 13 common frames omitted- In the sample stack trace above, the jar file that could not be found is:
/tmp/agent1234567890123456789.jar - The numeric portion in the filename changes for each restart
Resolution
Disable the
Log4jHotPatchtool by calling:sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill- Afterwards, start up Bitbucket