Hipchat Server upgrades fail due to missing lvm2 package
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
This is for an outdated version of Hipchat Server
This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.
Problem
Upgrading Hipchat Server from 1.4.2 to 1.4.3 fails during the Chef run, complaining that it cannot install the lvm2 package.
Trying to force an upgrade after the failure may throw the following errors:
Calculated sha512sum: b55f7c1b139ac6cba89eef57164d76ed60d70dcc4ecab70fbfe4fe7c5364ca51c6b64e9e9d423ecdcb5231d658847372470d90e585c678bf0e032e924f0cf395 hcs-1.4.3-2016.07.11.082020.tar.gz
coercing to Unicode: need string or buffer, NoneType found
Updating system configuration, this may take a few minutes...
There was an unexpected error: Command '['sudo', '/opt/atlassian/hipchat/sbin/cs']' returned non-zero exit status 1Diagnosis
Environment
- Hipchat Server 1.4.3 and earlier
Cause
The following is seen in /var/log/chef.log which indicates that the upgrade did not complete successfully due to the failure to fetch the lvm2 package from the on-disk Ubuntu archive. Note the FILE NOT FOUND error below for lvm2_2.02.98-6ubuntu2_amd64.deb:
[2016-07-22T05:47:22+00:00] ERROR: bash[apt_dist_upgrade_non_interactive] (hipchat_server_cleanup::apt_dist_upgrade_non_interactive line 2) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '100'
---- Begin output of "bash" "/tmp/chef-script20160722-2657-1pi26j0" ----
STDOUT: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
consolekit guile-1.8-libs libavcodec53 libavformat53 libavutil51
libboost-date-time1.46.1 libboost-iostreams1.46.1
libboost-program-options1.46.1 libboost-serialization1.46.1
libboost-thread1.46.1 libclass-isa-perl libcloog-ppl0 libdconf0 libgdu0
libgearman6 libgnome-keyring-common libgnome-keyring0 liblcms1 libllvm3.0
liblua5.1-0 liblvm2app2.2 libmagickcore4 libmagickwand4 libmpc2 libodbc1
libpam-ck-connector libpango1.0-0 libpangoxft-1.0-0 libpostproc52 libppl9
libpwl5 libsgutils2-2 libswitch-perl libtcl8.5 libtiff4 libtokyocabinet8
python-boto python-crypto python-m2crypto python-paramiko python-pycurl
python-software-properties python-support tcl8.5 tcl8.5-dev udisks
Use 'apt-get autoremove' to remove them.
The following packages will be upgraded:
lvm2
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/470 kB of archives.
After this operation, 290 kB of additional disk space will be used.
Err file:/file_store/archive/ bintray-trusty-btf/main lvm2 amd64 2.02.98-6ubuntu2
File not found
STDERR: E: Failed to fetch file:/file_store/archive/pool/main/l/lvm2/lvm2_2.02.98-6ubuntu2_amd64.deb File not found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
---- End output of "bash" "/tmp/chef-script20160722-2657-1pi26j0" ----
Ran "bash" "/tmp/chef-script20160722-2657-1pi26j0" returned 100
[2016-07-22T05:47:23+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
allow custom /etc filesThis puts the server in an unknown state and renders Hipchat unusable.
Workaround
For users affected by this problem, please follow the steps below. Log in to the Hipchat Server's command line and execute the following commands:
Obtain access as a root user
sudo dont-blame-hipchatNavigate to the directory where lvm2 is located:
cd /file_store/archive/pool/main/l/lvm2/Retrieve the missing file from the S3 storage of the hosted server in the lvm2 directory:
wget https://s3.amazonaws.com/hipchat-server-stable/utils/lvm2_2.02.98-6ubuntu2_amd64.debKick off a Chef Solo run using the cs command to correctly install the previously missing lvm2 package:
Once the command completes, please log in to the Hipchat clients and confirm things are working again.
This issue is permanently fixed in Hipchat Server 2.1.3. Please refer to the bug below for more information:
HCPUB-1200 - Getting issue details... STATUS