Problem

An upgrade to 2.4.x fails and renders the system unusable. Accessing the website shows the error below:

The following appears at the end of /var/log/chef.log:

[2018-06-04T10:58:41+00:00] ERROR: bash[update_mysql_password] (mariadb::server line 47) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20180604-21537-1db22zk" ----
STDOUT: [mariadb_grants] Validating password specified is indeed set
STDERR: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
---- End output of "bash"  "/tmp/chef-script20180604-21537-1db22zk" ----
Ran "bash"  "/tmp/chef-script20180604-21537-1db22zk" returned 1
[2018-06-04T10:58:41+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2018-06-04T10:58:41+00:00] INFO: Allow custom /etc files
[2018-06-04T10:58:42+00:00] INFO: Re-enabling monitoring
[2018-06-04T10:58:42+00:00] ERROR: Error: Configuration failure

Diagnosis

We've only seen this issue happening when upgrading to version 2.4.x:

HCPUB-18321 - Getting issue details... STATUS

Cause

We have not be able to reproduce the issue so we still don't know what the root cause is. 

Workaround

1. Log into the Hipchat Server terminal/command-line interface

2. Run the following commands to reset the database passwords, then restart all services:

   wget https://s3.amazonaws.com/hipchat-server-stable/utils/_mariadb_change_password.sh
   sudo dont-blame-hipchat
   chmod +x _mariadb_change_password.sh
   /bin/bash _mariadb_change_password.sh
   hipchat upgrade --restart
   exit

   The commands above downloads a script, gains root access to the system, sets the script to be executable, runs the script to reset the database password, resets the Hipchat Server services, then relinquishes root access.

In case problem persists, please contact Hipchat Server support for assistance.