Hipchat Server upgrade stalled and eventually timed out while executing apt-get-update-build-essentials
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
When running a Hipchat Server upgrade through the terminal/command-line, admin noticed the upgrade stalled and there is not much progress on the console. The process eventually failed and the following appears in /var/log/chef.log
[2016-11-23T13:28:38+08:00] INFO: Processing execute[apt-get-update-build-essentials] action run (build-essential::debian line 22)
[2016-11-23T14:29:32+08:00] ERROR: Running exception handlers
[2016-11-23T14:29:32+08:00] ERROR: Exception handlers complete
[2016-11-23T14:29:32+08:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out
[2016-11-23T14:29:32+08:00] ERROR: execute[apt-get-update-build-essentials] (build-essential::debian line 22) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command execeded allowed execution time, killed by TERM signal.
---- Begin output of apt-get update ----
STDOUT: Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en
Ign file: hipchat/main Translation-en
Ign http://ppa.launchpad.net trusty InRelease
Err http://ppa.launchpad.net trusty Release.gpg
Connection failed
Ign http://ppa.launchpad.net trusty Release
Ign http://ppa.launchpad.net trusty/main amd64 Packages/DiffIndex
Ign http://ppa.launchpad.net trusty/main i386 Packages/DiffIndex
STDERR:
---- End output of apt-get update ----
Ran apt-get update returned
[0m
================================================================================[0m
[31mError executing action `run` on resource 'execute[apt-get-update-build-essentials]'[0m
================================================================================[0m
[0m
Mixlib::ShellOut::CommandTimeout[0m
--------------------------------[0m
Command timed out after 3600s:
Command execeded allowed execution time, killed by TERM signal.
---- Begin output of apt-get update ----
STDOUT: Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en
Ign file: hipchat/main Translation-en
Ign http://ppa.launchpad.net trusty InRelease
Err http://ppa.launchpad.net trusty Release.gpg
Connection failed
Ign http://ppa.launchpad.net trusty Release
Ign http://ppa.launchpad.net trusty/main amd64 Packages/DiffIndex
Ign http://ppa.launchpad.net trusty/main i386 Packages/DiffIndex
STDERR:
---- End output of apt-get update ----
Ran apt-get update returned [0m
[0m
Cookbook Trace:[0m
---------------[0m
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:30:in `from_file'
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/default.rb:21:in `from_file'[0m
[0m
Resource Declaration:[0m
---------------------[0m
# In /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb
22: execute "apt-get-update-build-essentials" do
23: command "apt-get update"
24: action :nothing
25: # tip: to suppress this running every time, just use the apt cookbook
26: not_if do
27: ::File.exists?('/var/lib/apt/periodic/update-success-stamp') &&
28: ::File.mtime('/var/lib/apt/periodic/update-success-stamp') > Time.now - 86400*2
29: end
30: end.run_action(:run) if node['build_essential']['compiletime']
31:
32: %w{
33: autoconf
34: binutils-doc
35: bison
36: build-essential
37: flex
38: }.each do |pkg|
39:
40: r = package pkg do
41: action( node['build_essential']['compiletime'] ? :nothing : :install )
42: end
43: r.run_action(:install) if node['build_essential']['compiletime']
44:
45: end
[0m
[0m
Compiled Resource:[0m
------------------[0m
# Declared in /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:22:in `from_file'
execute("apt-get-update-build-essentials") do
action [:nothing]
retries 0
retry_delay 2
command "apt-get update"
backup 5
returns 0
cookbook_name :"build-essential"
recipe_name "debian"
not_if { #code block }
end
[0m
[0m
[0m
================================================================================[0m
[31mRecipe Compile Error in /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/default.rb[0m
================================================================================[0m
[0m
Mixlib::ShellOut::CommandTimeout[0m
--------------------------------[0m
execute[apt-get-update-build-essentials] (build-essential::debian line 22) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command execeded allowed execution time, killed by TERM signal.
---- Begin output of apt-get update ----
STDOUT: Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en
Ign file: hipchat/main Translation-en
Ign http://ppa.launchpad.net trusty InRelease
Err http://ppa.launchpad.net trusty Release.gpg
Connection failed
Ign http://ppa.launchpad.net trusty Release
Ign http://ppa.launchpad.net trusty/main amd64 Packages/DiffIndex
Ign http://ppa.launchpad.net trusty/main i386 Packages/DiffIndex
STDERR:
---- End output of apt-get update ----
Ran apt-get update returned [0m
[0m
Cookbook Trace:[0m
---------------[0m
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:30:in `from_file'
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/default.rb:21:in `from_file'[0m
[0m
Relevant File Content:[0m
----------------------[0m
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:
23: command "apt-get update"
24: action :nothing
25: # tip: to suppress this running every time, just use the apt cookbook
26: not_if do
27: ::File.exists?('/var/lib/apt/periodic/update-success-stamp') &&
28: ::File.mtime('/var/lib/apt/periodic/update-success-stamp') > Time.now - 86400*2
29: end
30>> end.run_action(:run) if node['build_essential']['compiletime']
31:
32: %w{
33: autoconf
34: binutils-doc
35: bison
36: build-essential
37: flex
38: }.each do |pkg|
39:
[0m
[0m
[2016-11-23T13:28:35+08:00] INFO: Forking chef instance to converge...
[2016-11-23T14:29:32+08:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Diagnosis
- From server's terminal/command-line, run cat /etc/apt/sources.list to verify if there are other resources in the APT configuration
From server's terminal/command-line, run apt-get update; echo $? and check the returned output. The expected output is as follows:
~$ sudo dont-blame-hipchat -c "apt-get update; echo $?" Get:1 file: trusty InRelease [1781 B] Get:2 file: hipchat InRelease [1753 B] Ign file: trusty/main Translation-en Ign file: hipchat/main Translation-en Reading package lists... Done 0
If the command didn't complete and/or it's trying to reach out to an external URL, proceed with the Resolution section below
Cause
APT (Advanced Package Tool) which is a set of tools for managing Debian packages configuration on the server is messed up or there are other resources in the APT configuration. Hipchat Server is not meant to reach out to any external sources during an apt-get operation (apt-get should only hit local files) but as you can see from the stack trace above, the apt-get is reaching out to external URL http://ppa.launchpad.net
Resolution
- Backup Hipchat Server for precautionary measure
From the server's terminal/command-line, run the followings to remove unwanted files from these locations:
sudo dont-blame-hipchat apt-get clean rm -f /etc/apt/sources.list.d/* rm -rf /var/lib/apt/lists/*
After that, run these commands, to check that the new Chef (configuration manager) .deb files made it:
dpkg -l | grep hc- ls -ahl /file_store/archive/pool/main/h/hc-chef-repo/ cat /etc/apt/sources.list ls -ahl /etc/apt/sources.list.d/
The output should be similar to these:
~$ dpkg -l | grep hc- ii hc-chef-repo 2016.11.23.192154-34ee8fd all config management for HipChat ii hc-client-bundle 2016.11.09.162543-366c47e all HipChat desktop and mobile clients ii hc-consul 2016.11.15.023757-45df12a all Consul for HipChat ii hc-crowd 2016.11.01.234254-19fb239 all Atlassian Crowd for HipChat ii hc-cumulus 2016.09.02.191338-4631a52 all S3 functionality for HipChat ii hc-curler 2016.11.23.145246-8149c13 all HipChat Curler ii hc-emoticons 2016.11.23.020131-caa9654 all HipChat Emoticons ii hc-fishtank 2016.11.23.141140-91c4d48 all HipChat Services ii hc-punjab 2016.09.06.160718-7c3655d all BOSH server ii hc-s3-signer 2016.11.22.143244-02f772b all S3 Read request signer for HipChat ii hc-scissortail 2016.11.23.141755-ab3e194 all HipChat data migration ii hc-server-cli 2016.11.22.183242-ca295ec all HipChat Server CLI ii hc-web 2016.11.23.015636-fe5180c all web-based functionality for HipChat ii hc-webcore 2016.11.02.003441-61a7270 all javascript client for HipChat admin@coco:~$ ls -ahl /file_store/archive/pool/main/h/hc-chef-repo/ total 412K drwxr-xr-x 2 root root 4.0K Nov 23 19:29 . drwxr-xr-x 23 root root 4.0K Nov 23 19:29 .. -rw-r--r-- 1 root root 400K Nov 23 19:29 hc-chef-repo_2016.11.23.192154-34ee8fd_all.deb admin@coco:~$ cat /etc/apt/sources.list # # Ubuntu trusty - Generated by Chef # deb [arch=amd64] file:///file_store/archive trusty main deb [arch=amd64] file:///file_store/archive hipchat main admin@coco:~$ admin@coco:~$ ls -ahl /etc/apt/sources.list.d/ total 8.0K drwxr-xr-x 2 root root 4.0K Mar 22 2016 . drwxr-xr-x 7 root root 4.0K Nov 23 20:33 .. admin@coco:~$
Next, run these commands to ensure the right keys are in place:
apt-key add /file_store/archive/archive-hcsinf-pub.gpg dpkg --configure -a apt-get update
If you receive gpg: can't open `/file_store/archive/archive-hcsinf-pub.gpg': No such file or directory when running the apt-key command above, directly download the apt key to your server following these commands:
cd /file_store/archive wget https://s3.amazonaws.com/hipchat-server-stable/utils/archive-hcsinf-pub.gpg apt-key add /file_store/archive/archive-hcsinf-pub.gpg
The output should look like this:
# apt-key add /file_store/archive/archive-hcsinf-pub.gpg OK # dpkg --configure -a # apt-get update Get:1 file: trusty InRelease [1781 B] Get:2 file: hipchat InRelease [1753 B] Ign file: trusty/main Translation-en Ign file: hipchat/main Translation-en Reading package lists... Done
Finally, re-run the upgrade and wait till it completes:
hipchat upgrade --force-upgrade
You may tail both update.log and chef.log to check on the progress:
tail -f /var/log/hipchat/update.log /var/log/chef.log