Log files
Logs are stored in the directory: /var/log/hipchat
. A symlink to this directory exists at /home/admin/logs
, as well.
Hipchat Server includes built-in commands to tail, or continuously output, and highlight the content of particular logs:
wlog
– Web functionalitytlog
– XMPP Chat Back-endclog
– APIv2 Servicesmlog
– Outbound email server
Configuration management is managed by chef-solo. It is run at boot, upgrade, and during service restarts. You can find the chef-solo log file in the following location: /var/log/chef.log
Logs Commands
Command | Use | Notes |
---|---|---|
hipchat log --rotate | Force a log rotation | This will force all logs to conform to the log rotation configuration specified in /etc/logrotate.conf and /etc/logrotate.d |
hipchat log --purge | Truncates the contents of all logs in /var/log | Be sure to backup any logs required for troubleshooting before executing this command. |
Log File Reference
Resource | Use | Notes |
---|---|---|
/var/log/chef.log | chef runs for installing/updating/configuring | Logging starts from first boot. Most system configuration changes will trigger a chef run. |
/var/log/cumulus.log | S3-compatible file service logs | Raw logs of attempted access but almost all connections are mediated through nginx |
/var/log/hipchat/nginx.log | nginx logs AND coral logs | Includes nginx-access entries alongside coral entries. nginx.err.log only logs ERROR and above. Any entries in nginx.err.log are indicative of a problem. |
/var/log/hipchat/kern.log | Ubuntu kernel logging | |
/var/log/schema_upgrade.log | Logs any schema upgrade changes that occur during upgrades | Useful for seeing upgrade history. |
/var/log/hipchat/atlassian-crowd.log | External directory (Crowd/AD/LDAP) integration and authentication | Related to user authentication and external directory synchronization. |
/var/log/hipchat/coral.log | APIv2 logs | Many services rely on coral for authentication, so this log is often referenced while tracing a problem. coral.err.log only logs ERROR and above. Any entries in coral.err.log are indicative of a problem. |
/var/log/hipchat/files.log | Entries related to daily cronjob for purging export files | |
/var/log/hipchat/web.log | WebUI logging (i.e. the php-based administration) | Good starting point for any error messages or stack traces occurring in the web interface. web.err.log only logs ERROR and above. Any entries in web.err.log are indicative of a problem. |
/var/log/hipchat/update.log | Detailed output of upgrades (and errors) | Critical for troubleshooting upgrade issues, along with chef.log. |
/var/log/hipchat/tetra.log | Core chat service log | Any errors here are usually pretty critical. tetra.err.log only logs ERROR and above. Any entries in tetra.err.log are indicative of a problem. |
/var/log/hipchat/hup.log | Logging of when the services are restarted | Helpful for troubleshooting a broken service/upgrade. "services starting" is to prevent access to the system before it is fully initialized, the hup.log is the orderly start - the last statement should be "maintenance_mode now OFF". |
/var/log/hipchat/hcapp.log | Hipchat-specific subprocesses:
| Entries include associated service name for easy parsing, such as:grep scissortail hcapp.log |
/var/log/hipchat/database.log | redis master log, there is another redis log for stats | If this file is huge then most likely sudo /bin/dont-blame-hipchat; chown redis /mnt is required |
/var/log/hipchat/daemon.log | Contain logs for the various daemons, including monit and mysqld | Useful for observing emergency service restarts via monit. Entries include daemon names for parsing, similar to hcapp.log |
/var/log/hipchat/runtime.log | List out server processes, disk space, server status (including CPU, memory, active user counts, etc.) | This is a great place to start for root cause analyzes. |