ResourceError("Redis get failed. args=['group:1'], host='localhost'",) Causing the Hipchat Server Web Interface to be Inaccessible

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

This version of Hipchat Server is no longer supported

This article applies to a version of Hipchat Server which is beyond the Atlassian End of Life policy, and is no longer supported.

When was my version deprecated?

The following versions have been deprecated:

  • Hipchat Server 1.3 (EOL Date: Aug 17, 2017)

The following versions will be deprecated soon:

  • Hipchat Server 2.0 (EOL Date: Jun 17, 2018)
  • Hipchat Server 2.1 (EOL Date: Dec 8, 2018)
  • Hipchat Server 2.2 (EOL Date: May 30, 2019)

You can read more about Atlassian's End of Life policy here

You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes. If possible, you should also consider deploying Hipchat Data Center instead.

Problem

The web interface is no longer accessible from the FQDN of Hipchat Server. When accessing the web interface, the Services Starting will be shown and it gets stuck there.

The following appears in the /var/log/hipchat/tetra.log:

2016-08-03T08:36:28.319099+00:00 chat tetra-proxy-2: [] ERROR: Unable to connect to Redis at localhost:6379 - [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by other side: 111: Connection refused.\n]
2016-08-03T08:36:28.319322+00:00 chat tetra-proxy-2: [] ERROR: Error checking tetra-app cluster: ResourceError("Redis smembers failed. args=['tetra-app-services'], host='localhost'",)
2016-08-03T08:36:28.362655+00:00 chat tetra-proxy-3: [] ERROR: Unable to connect to Redis at localhost:6379 - [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by other side: 111: Connection refused.\n]
2016-08-03T08:36:28.363016+00:00 chat tetra-proxy-3: [] ERROR: Error checking tetra-app cluster: ResourceError("Redis smembers failed. args=['tetra-app-services'], host='localhost'",)
2016-08-03T08:36:28.727071+00:00 chat tetra-proxy-1: [] ERROR: Unable to connect to Redis at localhost:6379 - [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by other side: 111: Connection refused.\n]
2016-08-03T08:36:28.727295+00:00 chat tetra-proxy-1: [] ERROR: Error checking tetra-app cluster: ResourceError("Redis smembers failed. args=['tetra-app-services'], host='localhost'",)
2016-08-03T08:36:29.178199+00:00 chat tetra-proxy-0: [] ERROR: Unable to connect to Redis at localhost:6379 - [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by other side: 111: Connection refused.\n]
2016-08-03T08:36:29.178714+00:00 chat tetra-proxy-0: [] ERROR: Error checking tetra-app cluster: ResourceError("Redis smembers failed. args=['tetra-app-services'], host='localhost'",)

You will notice the 500 Error entry in /var/log/hipchat/nginx.log:

2016-08-03T08:32:54.555819+00:00 chat coral-0: [/user#REQU0YsDb] ALERT: 500 error processing request /user?expand=items&start-index=0&max-results=1 : ResourceError("Redis get failed. args=['group:1'], host='localhost'",)
2016-08-03T08:32:54.555908+00:00 chat coral-0: [/user#REQU0YsDb] ERROR: Sending API error: 500 - An internal error occurred.  Please try again later.
2016-08-03T08:34:05.555812+00:00 chat coral-1: [-] Traceback (most recent call last):
2016-08-03T08:34:05.555856+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 423, in errback
2016-08-03T08:34:05.555937+00:00 chat coral-1: [-]     self._startRunCallbacks(fail)
2016-08-03T08:34:05.556005+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
2016-08-03T08:34:05.556084+00:00 chat coral-1: [-]     self._runCallbacks()
2016-08-03T08:34:05.556175+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
2016-08-03T08:34:05.556244+00:00 chat coral-1: [-]     current.result = callback(current.result, *args, **kw)
2016-08-03T08:34:05.556315+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1155, in gotResult
2016-08-03T08:34:05.556405+00:00 chat coral-1: [-]     _inlineCallbacks(r, g, deferred)
2016-08-03T08:34:05.556470+00:00 chat coral-1: [-] --- <exception caught here> ---
2016-08-03T08:34:05.556560+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2016-08-03T08:34:05.556626+00:00 chat coral-1: [-]     result = result.throwExceptionIntoGenerator(g)
2016-08-03T08:34:05.556715+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2016-08-03T08:34:05.556781+00:00 chat coral-1: [-]     return g.throw(self.type, self.value, self.tb)
2016-08-03T08:34:05.556869+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/api/api_filters.py", line 557, in _auth
2016-08-03T08:34:05.556935+00:00 chat coral-1: [-]     result = yield self._auth_bearer_token(core, clients, request, token)
2016-08-03T08:34:05.557023+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2016-08-03T08:34:05.557089+00:00 chat coral-1: [-]     result = result.throwExceptionIntoGenerator(g)
2016-08-03T08:34:05.557178+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2016-08-03T08:34:05.557244+00:00 chat coral-1: [-]     return g.throw(self.type, self.value, self.tb)
2016-08-03T08:34:05.557332+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/api/api_filters.py", line 571, in _auth_bearer_token
2016-08-03T08:34:05.557398+00:00 chat coral-1: [-]     client = yield clients.get_client(session.client_id)
2016-08-03T08:34:05.557486+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2016-08-03T08:34:05.557552+00:00 chat coral-1: [-]     result = result.throwExceptionIntoGenerator(g)
2016-08-03T08:34:05.557639+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2016-08-03T08:34:05.557705+00:00 chat coral-1: [-]     return g.throw(self.type, self.value, self.tb)
2016-08-03T08:34:05.557868+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/oauth/oauth_client_manager.py", line 124, in get_client
2016-08-03T08:34:05.558067+00:00 chat coral-1: [-]     client = yield self._load_client(client_id, group_id)
2016-08-03T08:34:05.558138+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2016-08-03T08:34:05.558177+00:00 chat coral-1: [-]     result = result.throwExceptionIntoGenerator(g)
2016-08-03T08:34:05.558213+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2016-08-03T08:34:05.558282+00:00 chat coral-1: [-]     return g.throw(self.type, self.value, self.tb)
2016-08-03T08:34:05.558421+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/oauth/oauth_client_manager.py", line 233, in _load_client
2016-08-03T08:34:05.558525+00:00 chat coral-1: [-]     client = yield OauthClient(self.core, client_id).load()
2016-08-03T08:34:05.558626+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2016-08-03T08:34:05.558697+00:00 chat coral-1: [-]     result = result.throwExceptionIntoGenerator(g)
2016-08-03T08:34:05.558745+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2016-08-03T08:34:05.558813+00:00 chat coral-1: [-]     return g.throw(self.type, self.value, self.tb)
2016-08-03T08:34:05.558903+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/libs/cached_record.py", line 215, in load
2016-08-03T08:34:05.559083+00:00 chat coral-1: [-]     data = yield self.fetch_from_cache()
2016-08-03T08:34:05.559155+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2016-08-03T08:34:05.559195+00:00 chat coral-1: [-]     result = result.throwExceptionIntoGenerator(g)
2016-08-03T08:34:05.559232+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2016-08-03T08:34:05.559302+00:00 chat coral-1: [-]     return g.throw(self.type, self.value, self.tb)
2016-08-03T08:34:05.559367+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/libs/cached_record.py", line 137, in fetch_from_cache
2016-08-03T08:34:05.559412+00:00 chat coral-1: [-]     e = yield self.cache.get(self.cache_key)
2016-08-03T08:34:05.559477+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2016-08-03T08:34:05.559569+00:00 chat coral-1: [-]     result = result.throwExceptionIntoGenerator(g)
2016-08-03T08:34:05.559634+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2016-08-03T08:34:05.559720+00:00 chat coral-1: [-]     return g.throw(self.type, self.value, self.tb)
2016-08-03T08:34:05.559785+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/storage/redis.py", line 305, in get
2016-08-03T08:34:05.559872+00:00 chat coral-1: [-]     result = yield self.do('get', key)
2016-08-03T08:34:05.559937+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
2016-08-03T08:34:05.560026+00:00 chat coral-1: [-]     result = g.send(result)
2016-08-03T08:34:05.560090+00:00 chat coral-1: [-]   File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/storage/redis.py", line 163, in do
2016-08-03T08:34:05.560178+00:00 chat coral-1: [-]     % (command, list(args), self.host))
2016-08-03T08:34:05.560248+00:00 chat coral-1: [-] txhipchat.error.ResourceError: Redis get failed. args=['OauthClient:9de930c0-a22c-4e2d-bb60-5c5bb8765036'], host='localhost'
2016-08-03T08:34:05.560375+00:00 chat coral-1: [/readstate#REQm5hOT4] ALERT: 500 error processing request /readstate : ResourceError("Redis get failed. args=['OauthClient:9de930c0-a22c-4e2d-bb60-5c5bb8765036'], host='localhost'",)
2016-08-03T08:34:05.560464+00:00 chat coral-1: [/readstate#REQm5hOT4] ERROR: Sending API error: 500 - An internal error occurred.  Please try again later.
2016-08-03T08:34:22.854905+00:00 chat nginx-error: 2016/08/03 08:34:22 [error] 5868#0: *123 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.1.30, server: , request: "GET / HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "hipchat.danial.com"
2016-08-03T08:35:02.884490+00:00 chat nginx-error: 2016/08/03 08:35:02 [error] 5867#0: *161 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.1.30, server: , request: "GET / HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "hipchat.danial.com"
2016-08-03T08:35:57.929248+00:00 chat nginx-error: 2016/08/03 08:35:55 [error] 5867#0: *215 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.1.30, server: , request: "GET / HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "hipchat.danial.com"

Diagnosis

Environment:

  • Hipchat Server 1.4.3 and higher. 

Diagnostics Step

There is a chance that you might be seeing the following error on the web trying to access the page:

Fatal error: Maximum execution time of 30 seconds exceeded in /hipchat-scm/web/application/libraries/RedisCache.php on line 184

 

Cause

While the log entries for tetra and nginx indicates that there is an issue with Redis, no specific causes are known.

Resolution

  1. Log in to the Hipchat Server terminal / SSH console

  2. Gain root access by running the following command:

    sudo dont-blame-hipchat

  3. Create a directory in /file_store/redis named redis-backup:

    mkdir /file_store/redis/redis-backup

  4. Stop redis if it is still running:

    service redis stop

  5. Move the redis dump.rdb file to the backup directory:

    mv /file_store/redis/dump.rdb /file_store/redis/redis-backup/dump-backup.rdb

  6. Invoke a full restart of all services:

    hipchat upgrade --restart

  7. If the process of updating configurations and restarting services does not help, please reboot the server and confirm that the web interface is accessible again:

    sudo reboot


Last modified on Jan 19, 2018

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.