Logs are stored in the directory:
/var/log/hipchat. A symlink to this directory exists at
/home/admin/logs, as well.
Hipchat Server includes built-in commands to tail, or continuously output, and highlight the content of particular logs:
wlog– Web functionality
tlog– XMPP Chat Back-end
clog– APIv2 Services
mlog– Outbound email server
Configuration management is managed by chef-solo. It is run at boot, upgrade, and during service restarts. You can find the chef-solo log file in the following location:
|hipchat log --rotate||Force a log rotation||This will force all logs to conform to the log rotation configuration specified in /etc/logrotate.conf and /etc/logrotate.d|
|hipchat log --purge||Truncates the contents of all logs in /var/log||Be sure to backup any logs required for troubleshooting before executing this command.|
Log File Reference
|/var/log/chef.log||chef runs for installing/updating/configuring||Logging starts from first boot. Most system configuration changes will trigger a chef run.|
|/var/log/cumulus.log||S3-compatible file service logs||Raw logs of attempted access but almost all connections are mediated through nginx|
|/var/log/hipchat/nginx.log||nginx logs AND coral logs||
Includes nginx-access entries alongside coral entries. nginx.err.log only logs ERROR and above.
Any entries in nginx.err.log are indicative of a problem.
|/var/log/hipchat/kern.log||Ubuntu kernel logging|
|/var/log/schema_upgrade.log||Logs any schema upgrade changes that occur during upgrades||Useful for seeing upgrade history.|
External directory (Crowd/AD/LDAP) integration and authentication
|Related to user authentication and external directory synchronization.|
Many services rely on coral for authentication, so this log is often referenced while tracing a problem. coral.err.log only logs ERROR and above.
Any entries in coral.err.log are indicative of a problem.
|/var/log/hipchat/files.log||Entries related to daily cronjob for purging export files|
|/var/log/hipchat/web.log||WebUI logging (i.e. the php-based administration)||
Good starting point for any error messages or stack traces occurring in the web interface. web.err.log only logs ERROR and above.
Any entries in web.err.log are indicative of a problem.
|/var/log/hipchat/update.log||Detailed output of upgrades (and errors)||Critical for troubleshooting upgrade issues, along with chef.log.|
Core chat service log
Any errors here are usually pretty critical. tetra.err.log only logs ERROR and above.
Any entries in tetra.err.log are indicative of a problem.
|/var/log/hipchat/hup.log||Logging of when the services are restarted||
Helpful for troubleshooting a broken service/upgrade.
"services starting" is to prevent access to the system before it is fully initialized, the hup.log is the orderly start - the last statement should be "maintenance_mode now OFF".
||Entries include associated service name for easy parsing, such as:
|/var/log/hipchat/database.log||redis master log, there is another redis log for stats||If this file is huge then most likely sudo /bin/dont-blame-hipchat; chown redis /mnt is required|
|/var/log/hipchat/daemon.log||Contain logs for the various daemons, including monit and mysqld||Useful for observing emergency service restarts via monit. Entries include daemon names for parsing, similar to hcapp.log|
|/var/log/hipchat/runtime.log||List out server processes, disk space, server status (including CPU, memory, active user counts, etc.)||This is a great place to start for root cause analyzes.|