Users are not receiving emails or push notifications in Data Center

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

Users do not receive email or push notifications when they're offline. 

It is also possible that some users do receive notifications and others don't depending on the node they're connected to. 

Diagnosis

Environment

  • Hipchat Data Center

Diagnostic Steps

  • Test your SMTP relay configuration by sending a test email from System > Email server > Test configuration > Send test email
  • Verify the gearmand process is running on all nodesgearmand is responsible for queuing the notifications: 

    sudo dont-blame-hipchat -c "monit status | grep -A1 \"Process 'gearmand'\""
  • Verify the curler process is running on all nodes, curler is responsible for polling the gearmand queue and processing the notifications:

    sudo dont-blame-hipchat -c "monit status | grep -A1 \"Process 'curler'\""

Cause

No notifications will be sent to your SMTP relay server if gearmand or curler are down.

It is possible that one of those two services has always been down since you setup Hipchat Data Center. It is more likely to impact Gearman and we have an open bug about it  HCPUB-3758 - Getting issue details... STATUS

Resolution

If curler or gearmand are down on a given node, run the following steps: 

  1. Gain root access:

    sudo dont-blame-hipchat 
  2. If gearmand is down, start it up:

    /usr/sbin/service gearman-job-server start
  3. Verify gearmand is up:

    /usr/sbin/service gearman-job-server status
  4. Restart curler:

    /etc/init.d/curler restart
  5. There's a chance monit gave up on monitoring gearman, monitor it again: 

    monit monitor gearmand
  6. Wait for a few minutes and verify monit sees gearmand and curler as up, monit is the service monitoring the Hipchat services:

    monit status | grep -A1 "Process 'gearmand'"
    monit status | grep -A1 "Process 'curler'"
  7. Retest the notifications


Last modified on Mar 13, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.