Hipchat Server is inaccessible after an upgrade to 1.3.7 or later due to services unable to connect to database
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
This version of Hipchat Server is no longer supported
This article applies to a version of Hipchat Server which is beyond the Atlassian End of Life policy, and is no longer supported.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes. If possible, you should also consider deploying Hipchat Data Center instead.
Problem
After an upgrade to v1.3 build 1.3.7 or 1.3.8, Hipchat Server becomes inaccessible. You may see the following when trying to access the server from the browser:
The following appears in /var/log/hipchat/web.err.log:
2015-12-04T03:23:54.201165+00:00 hipchat web[14402]: users/sign_in#REQ80AB9F uid-anon 9ms [alert] Uncaught Kohana_Database_Exception: There was an error connecting to the database: Access denied for user 'root'@'localhost' (using password: YES) in file /hipchat-scm/web/system/libraries/Database.php on line 218\n\n0: /hipchat-scm/web/system/libraries/Database.php:1172 -> connect()\n1: /hipchat-scm/web/system/libraries/ORM.php:902 -> list_fields()\n2: /hipchat-scm/web/system/libraries/ORM.php:155 -> reload_columns()\n3: /hipchat-scm/web/system/libraries/ORM.php:106 -> __initialize()\n4: /hipchat-scm/web/application/models/session.php:181 -> __construct()\n5: /hipchat-scm/web/application/models/session.php:200 -> hydrate_from_cache()\n6: /hipchat-scm/web/application/models/session.php:81 -> load_from_cache()\n7: /hipchat-scm/web/system/libraries/drivers/Session/HipchatDb.php:69 -> find_by_id()\n8: in Session_HipchatDb_Driver::read()\n9: /hipchat-scm/web/system/libraries/Session.php:160 -> session_start()\n10: /hipchat-scm/web/system/libraries/Session.php:73 -> create()\n11: /hipchat-scm/web/system/libraries/Session.php:38 -> __construct()\n12: /hipchat-scm/web/application/controllers/application.php:803 -> instance()\n13: /hipchat-scm/web/application/controllers/application.php:63 -> load_userdata_from_session_cookie()\n14: /hipchat-scm/web/application/controllers/users.php:23 -> __construct()\n15: in Users_Controller::__construct()\n16: /hipchat-scm/web/system/core/Kohana.php:251 -> newInstance()\n17: in Kohana::instance()\n18: /hipchat-scm/web/system/core/Event.php:209 -> call_user_func()\n19: /hipchat-scm/web/system/core/Bootstrap.php:55 -> run()\n20: /hipchat-scm/web/www/index.php:109 -> require()\n\nurl: https://hipchat.itnj.us/users/sign_in\nmethod: GET\nreferrer: none\nserver: \nremote_addr: 10.0.50.103\nuser_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.22 Safari/537.36\nx_hipchat_request: public
The following appears in /var/log/hipchat/coral.err.log:
2016-03-11T18:51:13.585796+00:00 hipchat coral-2: [] ERROR: SQL query failed: (1045, "Access denied for user 'root'@'localhost' (using password: YES)")
2016-03-11T18:51:13.585961+00:00 hipchat coral-2: [] ERROR: Exception while trying to fetch features: SQL query failed: (1045, "Access denied for user 'root'@'localhost' (using password: YES)")
The following appears in /var/log/hipchat/hcapp.log:
2016-03-11T18:52:43.166649+00:00 hipchat dory-0: [] Usage: cpu=0.20%, mem_rss=84.59MB, mem_vms=472.32MB
2016-03-11T18:52:43.167135+00:00 hipchat dory-0: [] SQL query: SELECT * FROM features, args=None
2016-03-11T18:52:43.170691+00:00 hipchat dory-0: [] ERROR: SQL query failed: (1045, "Access denied for user 'root'@'localhost' (using password: YES)")
2016-03-11T18:52:43.170933+00:00 hipchat dory-0: [] ERROR: Exception while trying to fetch features: SQL query failed: (1045, "Access denied for user 'root'@'localhost' (using password: YES)")
Cause
The script that generates the password for the database may have failed to run completely during the upgrade. This prevented the database password from updating appropriately, causing failures when services connect to the database.
Resolution
- Log into the Hipchat Server terminal/command-line interface
Run the following command to re-attempt the server upgrade:
wget https://s3.amazonaws.com/hipchat-server-stable/utils/_mariadb_change_password.sh sudo dont-blame-hipchat chmod +x _mariadb_change_password.sh /bin/bash _mariadb_change_password.sh hipchat upgrade -f exit
The commands above download a script, gain root access to the system, set the script to be executable, run the script to reset the database password, re-attempt the Hipchat Server upgrade, then exit from the root user.
In case problem persists, please file a support ticket.