Users are not receiving emails or push notifications in Data Center
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Users do not receive email or push notifications when they're offline.
It is also possible that some users do receive notifications and others don't depending on the node they're connected to.
Diagnosis
Environment
- Hipchat Data Center
Diagnostic Steps
- Test your SMTP relay configuration by sending a test email from System > Email server > Test configuration > Send test email
Verify the
gearmand
process is running on all nodes,gearmand
is responsible for queuing the notifications:sudo dont-blame-hipchat -c "monit status | grep -A1 \"Process 'gearmand'\""
Verify the
curler
process is running on all nodes,curler
is responsible for polling thegearmand
queue and processing the notifications:sudo dont-blame-hipchat -c "monit status | grep -A1 \"Process 'curler'\""
Cause
No notifications will be sent to your SMTP relay server if gearmand
or curler
are down.
It is possible that one of those two services has always been down since you setup Hipchat Data Center. It is more likely to impact Gearman and we have an open bug about it - HCPUB-3758Getting issue details... STATUS .
Resolution
If curler
or gearmand
are down on a given node, run the following steps:
Gain root access:
sudo dont-blame-hipchat
If
gearmand
is down, start it up:/usr/sbin/service gearman-job-server start
Verify
gearmand
is up:/usr/sbin/service gearman-job-server status
Restart
curler
:/etc/init.d/curler restart
There's a chance monit gave up on monitoring gearman, monitor it again:
monit monitor gearmand
Wait for a few minutes and verify
monit
seesgearmand
andcurler
as up,monit
is the service monitoring the Hipchat services:monit status | grep -A1 "Process 'gearmand'" monit status | grep -A1 "Process 'curler'"
- Retest the notifications