Bamboo fails to start with "Cannot execute <...> plugin has an extra copy of <...> classes, perhaps embedded inside the target plugin"

Summary

Bamboo Server and Agent fail to start with a reported error such as "Cannot execute <...> plugin has an extra copy of <...> classes, perhaps embedded inside the target plugin".

Environment

• Bamboo Server and Data Center

• Bamboo Agents

• External Apps (plugins) are installed

Diagnosis

Bamboo fails to start and reports the "Cannot execute <...> plugin has an extra copy of <...> classes, perhaps embedded inside the target plugin" message in the logs. It can randomly start successfully after a few retries without proper explanation. This also happens with Bamboo Agents.

Here's an example log from a server:

Cause

The two most common actions that could cause this behaviour are:

1. Bamboo was upgraded without proper validation of its apps in a test environment

2. Manual update and installation of Apps via Jar files or Download URLs outside of Atlassian Marketplace

Bamboo can get into a race condition where the apps can start up in such an order and speed that one app reads spring annotations from another that runs a different version. This causes the app initialisation to fail. This behaviour can be caused by third-party apps, which declare the Spring scanner twice. So when you restart, that app may have a chance to be at the bottom of the list to load, so it loads the critical apps and starts Bamboo. In some instances, restarting Bamboo will cause the loading order of the Apps to change and cause a successful load of the application, but that's not guaranteed.

That bootstrap failure behaviour will also manifest in Agents as Bamboo copies all its Apps to the Agents when installed or updated.

Diagnosis

To figure out each declared dependency of an app built with Maven, you must investigate each jar file located in your <bamboo-home>/shared/plugins folder and look for any dependencies that match the reported error.

1. Whenever you see such a message during Bamboo's initialisation, take note of the conflicting library. For example:

   1 ... Cannot execute atlassian-spring-scanner-runtime: plugin has an extra copy of atlassian-spring-scanner-annotation classes

2. Create the following Java code to scan your apps and report their dependencies. You need Java 11 or later to run it.

   BambooAppDependencyParser.java

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 import javax.xml.parsers.*; import org.w3c.dom.*; import java.io.*; import java.nio.file.*; import java.util.*; import java.util.zip.*; public class JarExplorer { public static void main(String[] args) { if (args.length < 1) { System.out.println("Please provide a jar file name as an argument."); return; } String fileName = args[0]; String filter = (args.length > 1) ? args[1] : null; Set<String> dependencies = new TreeSet<>(); try (ZipFile jarFile = new ZipFile(fileName)) { Path tempDir = Files.createTempDirectory("JarExplorer"); Enumeration<? extends ZipEntry> entries = jarFile.entries(); while(entries.hasMoreElements()) { ZipEntry entry = entries.nextElement(); if (entry.getName().endsWith("pom.xml")) { File outFile = new File(tempDir.toFile(), entry.getName()); outFile.getParentFile().mkdirs(); try (InputStream in = jarFile.getInputStream(entry); OutputStream out = new FileOutputStream(outFile)) { in.transferTo(out); } parsePom(outFile.getAbsolutePath(), dependencies); } } dependencies.stream().filter(dependency -> filter == null || dependency.contains(filter)).forEach(System.out::println); deleteDirectory(tempDir.toFile()); } catch(Exception e) { e.printStackTrace(); } } private static void parsePom(String fileName, Set<String> dependencies) { try { DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance(); DocumentBuilder dBuilder = dbFactory.newDocumentBuilder(); Document doc = dBuilder.parse(fileName); doc.getDocumentElement().normalize(); Map<String, String> properties = new HashMap<>(); NodeList propertiesList = doc.getElementsByTagName("properties"); for (int i = 0; i < propertiesList.getLength(); i++) { NodeList propertyNodes = propertiesList.item(i).getChildNodes(); for (int j = 0; j < propertyNodes.getLength(); j++) { Node propertyNode = propertyNodes.item(j); if (propertyNode.getNodeType() == Node.ELEMENT_NODE) { properties.put(propertyNode.getNodeName(), propertyNode.getTextContent()); } } } NodeList nList = doc.getElementsByTagName("dependency"); for (int temp = 0; temp < nList.getLength(); temp++) { Node nNode = nList.item(temp); if (nNode.getNodeType() == Node.ELEMENT_NODE) { Element eElement = (Element) nNode; String groupId = getTextContent(eElement, "groupId"); String artifactId = getTextContent(eElement, "artifactId"); String version = getTextContent(eElement, "version"); if (version != null && version.startsWith("${") && version.endsWith("}")) { String propertyName = version.substring(2, version.length() - 1); if (properties.containsKey(propertyName)) { version = properties.get(propertyName); } } dependencies.add(groupId + ":" + artifactId + ":" + version); } } } catch(Exception e) { e.printStackTrace(); } } private static String getTextContent(Element element, String tagName) { NodeList nodeList = element.getElementsByTagName(tagName); return (nodeList.getLength() > 0 && nodeList.item(0) != null) ? nodeList.item(0).getTextContent() : "N/A"; } private static void deleteDirectory(File directoryToBeDeleted) { File[] allContents = directoryToBeDeleted.listFiles(); if (allContents != null) { for (File file : allContents) { deleteDirectory(file); } } directoryToBeDeleted.delete(); } }

3. Copy the located jar files from <bamboo-home>/shared/plugins to a temporary location

4. Go to the temporary location, and for each jar file, run the script, for example. Note that we are specifying "spring-scanner" as a search string:

   1 $ java BambooAppDependencyParser.java plugin.5406341525478459069.allure-bamboo-1.15.0.jar spring-scanner

5. If there's a match, you'll see the dependency version used by the App showed twice with two possible versions. In the example below, it will load atlassian-spring-scanner-annotation:2.1.7 and another version that is the value of ${project.version}

   1 2 3 4 $ java BambooAppDependencyParser.java plugin.5406341525478459069.allure-bamboo-1.15.0.jar spring-scanner com.atlassian.plugin:atlassian-spring-scanner-annotation:${project.version} com.atlassian.plugin:atlassian-spring-scanner-annotation:2.1.7 com.atlassian.plugin:atlassian-spring-scanner-runtime:2.1.7

6. Compare the dependency version reported by the app with the one provided by Bamboo. You can locate the dependencies versions by checking Bamboo's pom.xml files. For example, for Bamboo 9.2.4, you can download thepom.xmlfile from the following link:

   • https://packages.atlassian.com/maven/repository/public/com/atlassian/bamboo/atlassian-bamboo/9.2.4/atlassian-bamboo-9.2.4.pom

7. After downloading and comparing the reported versions of the dependencies between each App and the main Bamboo pom files, we can see the version below, which means that the version used by the app (2.1.7) is earlier than the one used by Bamboo (3.0.3), and that's causing the bootstrap issue.

   1 <atlassian.spring.scanner.version>3.0.3</atlassian.spring.scanner.version>

8. Check the suspicious App for updates on Atlassian Marketplace or the Vendor's website.

Solution

1. Make sure that your Bamboo instance is using only compatible and updated apps

2. Validate the application via Bamboo Administration >> Manage Apps and make sure that all Apps are up to date

3. Atlassian has no control over how external vendors release their product updates, so each customer must reach out to each Vendor looking for an update

4. Before upgrading Bamboo, make sure to test any apps in a UAT/Staging/Test environment before proceeding with a change in Production

5. If there's no way to update an incompatible app, follow the steps in this KB article to remove it manually from Bamboo before starting it:

   1. How to start Bamboo with all or few external apps/plugins disabled

6. You can run some queries to locate Plans that may be using such Apps:

   1. How to check which plugins are being used by Bamboo plans