monit scissortail alerts are received after upgrade from 1.1.x to 2.x
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
This is for an outdated version of Hipchat Server
This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.
Problem
After a successful upgrade from 1.1.x to 2.x, Hipchat Server is sending monitoring alerts similar to the following:
Here are the details,
Action: alert
Desc: failed to start (exit status -1) – Program /hipchat/scissortail/current/wrapper.sh failed: File '/hipchat/scissortail/current/wrapper.sh' does not exist
Event: Execution failed
Service: scissortail-rqworker
Host: hipchat.example.com
Date: Thu, 02 Mar 2017 13:38:52
Here are the details,
Action: alert
Desc: failed to start (exit status 0) – /etc/init.d/scissortail: Traceback (most recent call last):
File "/hipchat-scm/scissortail/vendor/virtualenv/lib/python3.5/site-packages/gunicorn/app/base.py", line 92, in get_config_from_filename
execfile_(filename, cfg, cfg)
File "/hipchat-scm/s
Event: Execution failed
Service: scissortail-gunicorn
Host: hipchat.example.com
Date: Thu, 02 Mar 2017 13:39:23
Looking at /var/log/hipchat/runtime.log
or its rotated version /var/log/hipchat/runtime.log.1.gz
, you will see
/etc/monit.d/scissortail.monitrc:2: Program does not exist: '/hipchat/scissortail/current/wrapper.sh'
/etc/monit.d/scissortail.monitrc:3: Program does not exist: '/hipchat/scissortail/current/wrapper.sh'
Diagnosis
Validate the symptoms by checking the presence of /etc/monit.d/scissortail.monitrc
.
Cause
The upgrade doesn't properly cleanup /etc/monit.d/scissortail.monitrc
. This config file tells monit to monitor for a process that no longer exists in Hipchat Server.
When monit doesn't find this process running, it will try to restart it and will fail.
Workaround
Move the monit config file out of the way:
sudo dont-blame-hipchat mv -v /etc/monit.d/scissortail.monitrc /home/admin/scissortail.monitrc.bak # Pick any other location if needed
Restart monit:
service monit restart
Verify monit doesn't return any process as failed:
monit status
Resolution
The issue is logged as a bug in our public issue tracker: - HCPUB-2779Getting issue details... STATUS