Bamboo Recovery after Encryption Key Loss
Bamboo stores a cipher key in the home directory which is used to decrypt sensitive data. The data that is encrypted is defined on System Wide Encryption.
If the key on the filesystem is lost Bamboo will fail to start and the following occurs in the logs (catalina.out):
//During startup 2019-05-22 17:05:28,215 FATAL [localhost-startStop-1] [InstanceSecretStorage] Filesystem part of instance key is unavailable 2019-05-22 17:05:28,217 FATAL [localhost-startStop-1] [SecretEncryptionServiceInternalImpl] java.lang.IllegalArgumentException: Unable to read cipher data for 0 at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59) at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116)... 2019-05-22 17:05:28,231 FATAL [localhost-startStop-1] [InstanceSecretStorage] Filesystem part of instance key is unavailable 2019-05-22 17:05:28,236 FATAL [localhost-startStop-1] [BambooContainer] Cannot start Bamboo java.lang.IllegalArgumentException: Unable to read cipher data for 0 at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59) at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116) ... 2019-05-22 17:05:28,241 FATAL [localhost-startStop-1] [UpgradeLauncher] Failed to initialise Bamboo container java.lang.IllegalArgumentException: Unable to read cipher data for 0 at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59) at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116) ... 22-May-2019 17:05:28.283 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file 22-May-2019 17:05:28.288 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context  startup failed due to previous errors //During usage with linked repositories com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Cannot read keypair, it doesn't contain any line with '-----BEGIN' at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) at com.atlassian.bamboo.utils.collections.AlwaysInvalidatingCacheDecorator.get(AlwaysInvalidatingCacheDecorator.java:65)Diagnosis
The cipher key files (cipher.key_x) located in the <Bamboo-Home>/xml-data/configuration/cipher folder is missing, failed to create or does not have correct permissions.
There are 3 states that the cipher files can be:
- If the cipher file(s) are present then ensure that the user that runs Bamboo has correct permissions to access the file and restart Bamboo.
- If the files are missing and a backup is available then recover the cipher.key_0 file from the backup and restore it to cipher folder and restart Bamboo.
- If the file is missing or has 0 length then you can attempt the recovery procedure described in the following section.
This section discusses the (partial) recovery of the Bamboo instance in case the cipher.key_0 file is lost and cannot be recovered from backup. For the rest of the section, we will use the following notation:
- BI - Broken Bamboo Instance that will be recovered.
- FI - Fresh Bamboo instance used to generate new encryption keys.
- The design of the System wide encryption feature of Bamboo does not allow for data recovery in case the cipher.key_x file is lost. All encrypted data is lost.
- This procedure only cleans up encrypted data and generates new encryption keys. All data that was encrypted (credentials, SSH keys, variables, etc.) needs to be re-populated manually.
- We highly recommend performing a backup before proceeding with the steps below!
1. Backup BI
- Stop BI.
- Save a copy of the BI install directory.
- Save a copy of the BI home directory.
- Create a database dump of the BI database using the native database backup tools.
2. Disable Encryption and automatic builds during startup in BI
In order to prevent BI from running automated builds during the startup, place the following arguments in the <BI-Install-dir>/bin/setenv.sh file under JVM_SUPPORT_RECOMMENDED_ARGS (see Configuring your system properties):
3. List all the encrypted data in BI (database and filesystem)
We perform these steps to identify which data is encrypted with the lost keys and needs to be cleaned up. Bamboo encrypts data in 4 separate locations:
Shared Credentials with passwords.
Global / Plan variables which have been masked (*****) - These are variables that include keywords such as "secret" and "password"
Passwords and SSH private keys in Repository configurations
- Mail SMTP server encryption.
All data that is encrypted has a prefix of BAMSCRT. Each subsection below lists the encrypted data. After Identifying the encrypted data, you need to update it with a non-encrypted string from the UI.
Alwaysyour data before performing any modifications to the database. If possible, test any alter, insert, update, or delete SQL commands on a staging server first.
Please keep track of all the results/items below in a list, as you will need to reset their correct passwords after the recovery process.
3a. Shared Credentials
To find if there are any shared credentials with an encrypted password, run the following Database Query on the BI database:
SELECT * FROM credentials WHERE xml LIKE '%BAMSCRT%';
3b. Global / Plan Variables
To find if there are any variables with an encrypted value, Run the following Database Query on BI database (Tested in PostgreSQL):
SELECT variable_definition.variable_key, variable_definition.variable_type, build.full_key, build.title, variable_definition.variable_value FROM variable_definition LEFT JOIN build ON (build.build_id=variable_definition.plan_id AND build.marked_for_deletion = false) WHERE variable_definition.variable_value LIKE '%BAMSCRT%';
3c. Repository Configurations
To find if there are any variables with an encrypted value, run the following Database Query on BI database:
SELECT v.name, v.is_global, REPLACE(v.plugin_key,'com.atlassian.bamboo.plugins.','') as repo_type, v.marked_for_deletion, build.full_key, build.title FROM vcs_location AS v LEFT JOIN plan_vcs_location ON v.vcs_location_id=plan_vcs_location.vcs_location_id LEFT JOIN build ON (build.build_id=plan_vcs_location.plan_id AND build.marked_for_deletion = false) WHERE v.xml_definition_data LIKE '%BAMSCRT%';
3d. Mail SMTP password
Unlike the other items on this list, the Mail password is stored on disk. If you have a SMTP server setup, this is located in the <BI-home-dir>/bamboo-main.cfg.xml file under the <password> tags. If you see a string prefixed with BAMSCRT either update this file or update it in the UI by navigating to:
- Bamboo administration > Mail server > Edit
4. Cleanup encrypted data in BI
In this section, we cleanup the encrypted repository entries that are listed in section 3c.
- Run the following query against the BI database to replace the encryption identifier "BAMSCRT" with a dummy value:
UPDATE vcs_location SET xml_definition_data = REPLACE(xml_definition_data,'BAMSCRT','-----BEGIN KEY----- OLD -----END KEY-----') WHERE xml_definition_data LIKE '%BAMSCRT%';
Note: The replacement value must be "-----BEGIN KEY----- OLD -----END KEY-----" to recover certain types of repositories
Run the vcs_location select query (from 3c) again to confirm no variables are left, keeping the original output for later.
- If the <BI-Home>/bamboo-main.cfg.xml does not contain a smtp server with a <password> value defined, backup up the original file and replace the contents with:
<?xml version="1.0" encoding="UTF-8"?> <mail-servers> <smtp-server> <name>Bamboo</name> <password>password</password> <defaultFrom>firstname.lastname@example.org</defaultFrom> <prefix>Bamboo</prefix> <hostname>bamboo</hostname> <sessionServer>false</sessionServer> <port>25</port> <tlsRequired>false</tlsRequired> </smtp-server> </mail-servers>
All the remaining data listed in section 3 will be cleaned up in the Bamboo UI as shown in section 6.
5. Generate new encryption keys in FI
To generate new encryption keys you will need to use another fresh installation (FI) of Bamboo. These new keys will overwrite the keys that were lost in the Broken Instance (BI).
Install a new Bamboo instance with the same version of BI (using different home and install directories).
To force the initial key generation, create a global variable with the Variable name containing 'password' in it and a dummy value and 'add'.
Extract the encryption keys from this FI. The below query in FI database should list the database part of encryption keys.
SELECT * FROM BANDANA WHERE BANDANA_KEY LIKE '%cipher%';
The file system part of the key is located in <FI-home-dir>/xml-data/configuration/cipher
6. Finalize recovery stage
6a. Replace the cipher key in the corrupt installation with the regenerated cipher key from FI.
Run the below queries to update encryption keys (Tested this query in PostgreSQL).
UPDATE bandana SET serialized_data = '<string>ReplaceThisWithKeyStringFromFI</string>' WHERE bandana_key='com.atlassian.restricted.instance.cipher.key_0'; UPDATE bandana SET serialized_data = '<string>ReplaceThisWithIVStringFromFI</string>' WHERE bandana_key='com.atlassian.restricted.instance.cipher.iv_0';
- copy the cipher file from <FI-home-dir>/xml-data/configuration/cipher to <BI-home-dir>/xml-data/configuration/cipher
6b. Start BI with encryption disabled
Ensure Bamboo starts correctly. If Bamboo fails to start with errors related to cipher data, check for any entries that were missed from the previous section (Identify affected data and update it if necessary)
6c. Update the rest of the previously encrypted data in BI.
Now that we have BI running, we will updated the passwords listed in section 3.
- Navigate and update manually Shared Credentials and Global / Plan Variables in BI.
- Navigate and update Repository Configurations using the output of query in section 3c:
If "marked_for_deletion" is "true"
These items can be ignored, as they are only used when re-running old builds
If "is_global" is "true"
Navigate to Bamboo administration > Linked Repositories
Select the Repository that requires an update
- Check "Change Password?" and set set the correct password
- If "is_global" is "false"
Determine the Plan where the repository is located, defined by the "full_key" and "title" columns
- Navigate to Plans > All Build Plans > and configure the plan shown from the Query
- Select the "Repositories" tab and select the repository that requires an update
- Check "Change Password?" and set the correct password
- If "type" starts with "stash.atlassian-bamboo-plugin-stash"
- Use steps b. or c. to navigate to the repository in the UI
- You should see an error such as: "Could not access this repository in Bitbucket Server"
- Press "Save Repository" to clear the error and regenerate a new encrypted SSH key
7. Re-enable encryption in BI
Now that we have repopulated all the credentials , we restart BI with encryption enabled, so that BI will encrypt all credentials with the new keys.
- Stop BI
- Remove the argument -Datlassian.bamboo.build.disable=true -Dbamboo.security.encryption.disable=true from the setenv.sh file (set in section 2)
- Start Bamboo
The instance is now recovered. You may now remove the Fresh Instance(FI) from your system.
If you require any further support with these steps please contact our friendly support.