Chat services are down due to broken ./virtualenv/*
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
This version of Hipchat Server is no longer supported
This article applies to a version of Hipchat Server which is beyond the Atlassian End of Life policy, and is no longer supported.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes. If possible, you should also consider deploying Hipchat Data Center instead.
Problem
Chat services are down. Server restart and/or server upgrade didn't make any difference.
The following appears repeatedly in the /var/log/chef.log
[2015-06-17T13:00:09+00:00] ERROR: Running exception handlers
[2015-06-17T13:00:09+00:00] ERROR: Exception handlers complete
[2015-06-17T13:00:09+00:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out
[2015-06-17T13:00:09+00:00] ERROR: python_pip[wheel==0.24.0] (hcapp::btf-setup line 71) had an error: Errno::ENOENT: No such file or directory - /hipchat/dory/current/vendor/virtualenv/bin/pip install --no-deps --no-index --find-links=/opt/atlassian/hipchat/dependencies/python --upgrade wheel==0.24.0
[2015-06-17T13:00:09+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)Cause
Somehow, ./virtualenv/* wasn't populated correctly and Chef runs (hipchat upgrade --restart) doesn't fix broken ./virtualenv/* contents.
Workaround
SSH into the server and run the following commands to bring chat services up again:
sudo dont-blame-hipchat
mv /hipchat/dory/current/vendor /hipchat/dory/current/vendor.broken
cs && hup
exitThis issue should be fixed in 1.4.x and above.