Integrations no longer send notifications to rooms after server upgrade

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.

This is for an outdated version of Hipchat Server

 This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.

When will my version be deprecated?

The following versions have been deprecated:

  • Hipchat Server 1.3 (EOL Date: Aug 17, 2017)
  • Hipchat Server 2.0 (EOL Date: Jun 17, 2018)
  • Hipchat Server 2.1 (EOL Date: Dec 8, 2018)

The following version will be deprecated soon:

  • Hipchat Server 2.2 (EOL Date: May 30, 2019)

You can read more about Atlassian's End of Life policy here.

You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.


Integrations are no longer sending notifications to rooms. Some examples are:

  • Jira no longer send notifications to Hipchat rooms
  • Confluence no longer send notifications to Hipchat rooms

The following appears in the /home/admin/logs/nginx.log:

2016-03-17T15:07:01.672700+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] =====> New API request: #1522, uri=/room/3/notification, method=POST,  from=, job_handle=none
2016-03-17T15:07:01.672888+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Fetching an oauth session for token XXXXXXXXUsu7SWiUjPvZpaJdmt1DYNOtxwAvcXC2
2016-03-17T15:07:01.673066+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query: SELECT * FROM oauth_sessions WHERE access_token=%(access_token)s, args={'access_token': 'XXXXXXXXUsu7SWiUjPvZpaJdmt1DYNOtxwAvcXC2'}
2016-03-17T15:07:01.673986+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query completed in time=1ms
2016-03-17T15:07:01.674263+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Authenticated session 8181 by access token XXXXXXXXUsu7SWiUjPvZpaJdmt1DYNOtxwAvcXC2
2016-03-17T15:07:01.675277+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Authenticated client c9b73251-34ef-4ef9-b454-87eeea5e29e6
2016-03-17T15:07:01.677322+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query:  SELECT group_id, name FROM rooms WHERE group_id=%(group_id)s AND id = %(room_id)s AND !is_deleted , args={'pretty_name': u'3', 'group_id': 1, 'room_id': 3}
2016-03-17T15:07:01.677910+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query completed in time=1ms
2016-03-17T15:07:01.679008+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Sending system message to rid-1_dropcam_devops, data={'color': 'yellow', 'from_nick': 'Frodo', 'notify': False, 'message_format': u'text'}
2016-03-17T15:07:01.679507+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] ERROR: Error calling broadcast_room_system_message: Service Unavailable, try back later
2016-03-17T15:07:01.679666+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] ERROR: Sending API error: 503 - Service Unavailable, try back later
2016-03-17T15:07:01.680065+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Traceback (most recent call last):
2016-03-17T15:07:01.680107+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/", line 1099, in _inlineCallbacks
2016-03-17T15:07:01.680147+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     result = g.send(result)
2016-03-17T15:07:01.680186+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/libs/", line 158, in proxy
2016-03-17T15:07:01.680224+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     resp = yield self.call_remote_server(func_name, body)
2016-03-17T15:07:01.680262+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/dace/", line 194, in __call__
2016-03-17T15:07:01.680299+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     d = self._run(*args, **kwargs)
2016-03-17T15:07:01.680337+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/", line 1237, in unwindGenerator
2016-03-17T15:07:01.680373+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     return _inlineCallbacks(None, gen, Deferred())
2016-03-17T15:07:01.680409+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] --- <exception caught here> ---
2016-03-17T15:07:01.680446+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/", line 1099, in _inlineCallbacks
2016-03-17T15:07:01.680482+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     result = g.send(result)
2016-03-17T15:07:01.680519+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/dace/", line 97, in call
2016-03-17T15:07:01.680555+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     d = yield func(*args, **kwargs)
2016-03-17T15:07:01.680592+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/dace/", line 129, in call
2016-03-17T15:07:01.680629+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     scheme, netloc = load_balancer()
2016-03-17T15:07:01.680666+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/tetra/", line 41, in __call__
2016-03-17T15:07:01.680701+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG]     raise ServiceUnavailableError("No tetra hosts found")
2016-03-17T15:07:01.680738+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] txhipchat.error.ServiceUnavailableError: Service Unavailable, try back later
2016-03-17T15:07:01.680813+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] WARN: =====> API request failed: #1522, in-progress=0, code=503
2016-03-17T15:07:03.468886+00:00 hipchat nginx-access: - - [17/Mar/2016:15:07:01 +0000]  "POST /v2/room/3/notification HTTP/1.1" 503 136 "-" "python-requests/2.9.1" "-" -

From the log snippet above, you can see there's an error sending the notification and the API request failed. It occurs when hitting the '/room/<ID>/notification' REST endpoint and it failed when trying to connect to a tetra-app service which responsible for a big piece of actually sending the notification.


Re-configure the integrations to isolate caching issue.


tetra-app cache entries in the server are broken.


Restart the tetra-app and related services with the following command:

hipchat service --restart

The above command should re-register cached 'tetra-app' entries we're looking for, allowing the process to send notifications to complete successfully.

 This will cause a brief service outage, so it is advisable to notify your team upfront or set a small maintenance window to avoid service disruption.

If there's still an issue, please contact Hipchat Server Support here.


Last modified on Jan 19, 2018

Was this helpful?

Provide feedback about this article
Powered by Confluence and Scroll Viewport.