Hipchat Server upgrade fails with Chef error
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
An upgrade of Hipchat Server fails and throws the following line to /var/log/hipchat/update.log:
There was an unexpected error: Command '['sudo', '/opt/atlassian/hipchat/sbin/cs']' returned non-zero exit status 1The following appears in the /var/log/chef.log:
[2016-10-17T08:17:14+00:00] INFO: Forking chef instance to converge...
[2016-10-17T08:17:14+00:00] INFO: *** Chef 11.8.2 ***
[2016-10-17T08:17:14+00:00] INFO: Chef-client pid: 16048
[2016-10-17T08:17:15+00:00] INFO: Setting the run_list to ["role[btf]"] from JSON
[2016-10-17T08:17:15+00:00] ERROR: Role btf (included by 'top level') is in the runlist but does not exist. Skipping expand.
[2016-10-17T08:17:15+00:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out
[2016-10-17T08:17:15+00:00] ERROR: The expanded run list includes nonexistent roles: btf
================================================================================
Error expanding the run_list:
================================================================================
Missing Role(s) in Run List:
----------------------------
btf included by 'top level'
Original Run List
-----------------
role[btf]
[2016-10-17T08:17:14+00:00] INFO: Forking chef instance to converge...
[2016-10-17T08:17:15+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
allow custom /etc files
re-enabling monitoring
-----------------------------Diagnosis
Environment
- This can happen in any Hipchat Server environment.
Diagnostic Steps
Run a health check of the Hipchat Server Chef package:
dpkg -s hc-chef-repoEven if the command returns an "Status: install ok installed" please proceed to the workaround steps.
Cause
The root cause of the Chef failure is a broken/corrupt Chef repo in Hipchat Server.
Workaround
Re-install the Chef repo:
sudo dont-blame-hipchat -c 'apt-get --reinstall install hc-chef-repo'Restart the upgrade:
hipchat upgrade --upgradeIf the upgrade fails again, force it:
hipchat upgrade --force-upgrade- If the upgrade fails again, run it offline.
Resolution
There's no permanent resolution to this upgrade failure as it might be caused by several events. The workaround has to be executed every time this issue is encountered.
The error is nevertheless logged as a bug in HCPUB-1197.