Set up a small-scale Hipchat Data Center cluster

This guide contains step-by-step instructions for setting up the component services for a small-scale instance of Hipchat Data Center using VMware hosts. This type of deployment uses one VMWare host for the Hipchat node, another VMWare host for all three data stores, and a reverse proxy.

tip/resting Created with Sketch.

Use this deployment configuration if:

  • You have 2000 users or fewer and you do not require High Availability (HA) in your deployment.
    (If you require HA, you must use an an Enterprise-scale instance.)
  • You're deploying a temporary evaluation, test, or development instance of Hipchat Data Center.

On this page:

Small-scale deployment architecture notes

A small-scale deployment can save you administrative and maintenance time and reduce infrastructure costs. However, it can also introduce some single points of failure which you should be aware of as an administrator. Because there is only one of each component in the cluster, there are no failover mechanisms, and a single node failure can make the entire Hipchat deployment unavailable. For example, if the node that hosts your data stores fails, all three data stores are affected and the deployment will immediately stop. Similarly, a problem on the reverse proxy will prevent users from accessing the service, and a failure on the Hipchat node itself will stop the service.

If you are deploying a small-scale instance in production, make sure that you have good monitoring set up to alert you if there are problems, make sure your configuration files are backed up, and take frequent backups of the data stores node.

Planning a small-scale Hipchat Data Center deployment

Depending on your organization's needs, you might decide to plan for the future when deploying Hipchat Data Center.

Reverse proxy vs load balancer

In both the small- and Enterprise-scale deployment options, clients access the Hipchat node(s) through a service which terminates SSL. This is required regardless of which deployment type you choose. 

In a small-scale deployment all client connections go to a single Hipchat node, so this service can be a reverse proxy. When you add nodes to create an Enterprise-scale deployment, the service must also function as a load-balancer which distributes client connections among the Hipchat nodes. If you think you might later increase your small-scale deployment to Enterprise-scale, you can choose one of several services (such as NGINX, HAProxy or Apache) that can function as both a load balancer and a reverse proxy.

Planning for apps, add-ons, and integrations

If your deployment will use apps, add-ons, or integrations which are hosted on the public internet, you must make sure that your cluster allows the Hipchat nodes to access them. You can set up a forward proxy (sometimes called an "outbound" proxy) to allow the Hipchat nodes to reach external apps, and configure your network so that the apps can reach your reverse proxy (or load balancer). 

If you are unable to connect your cluster to the internet for security reasons, some open source add-ons allow you to download their source code, inspect it, and rebuild it to host it on your network as a private service. 

Sizing for migration

If you are migrating from a Hipchat Server deployment you might need to increase the data store node's available storage based on the number and size of messages in your exported server data.

A small-scale deployment of Hipchat Data Center


Hardware requirements for a small-scale Hipchat Data Center deployment

  • The reverse proxy host can be small: 2 CPU cores 1GHz (or faster), with 2GB of RAM.
  • The Hipchat application node must have 8 CPU cores 2.8GHz (or faster),16GB RAM, and a minimum of 40GB of (non-RAM) storage space.

  • The data stores node must have 8 CPU cores 2.8GHz (or faster), 16GB RAM, and at least 60GB of non-RAM storage. 

    tip/resting Created with Sketch.

    If you are migrating from a Hipchat Server deployment, you might need to increase the data store node's available storage based on the number and size of messages in your exported server data.

Network configuration requirements

  • Hipchat nodes must be deployed on a private network. You should only be able to access the nodes through a reverse proxy or jumpbox.
  • Your private network should only allow inbound connections on port 80 (HTTP), 443 (HTTPS), and 22 (SSH). (See below for details.)
  • SSL must be terminated at a load balancer or reverse proxy.
  • Hipchat nodes must have unrestricted network access to any other Hipchat nodes in the cluster. 
  • Hipchat nodes must have unrestricted network access to the data stores. 
  • Hipchat nodes also require access to ports 53 for DNS and 123 for NTP, and may require additional access to enable optional features such as email notifications, Add-ons, Hipchat Video, and mobile notifications. You might need to write additional DNS-based firewall or proxy rules to allow this access. See the Open and Outgoing ports for more information. 
  • The private network on which you deploy should allow outbound access or have a forward proxy. If your organization uses a DMZ, you can deploy the reverse proxy there.

Open and outgoing ports 

Depending on which Hipchat features you choose to enable, you may need to unblock or write firewall rules for the following ports. If you are using a forward proxy, you can use it to access most of these services without writing rules.

Feature Optional or required? Port/Protocol Notes
DNS Required 53 TCP/UDP Used for DNS resolution of hostnames and to set up SSL trust.
NTP (Network Time Protocol) Required 123 TCP/UDP Keeps the cluster members on the same schedule.
Email notifications

Optional

(recommended)

25 TCP If your SMTP server is accessible by the Hipchat nodes from inside the network, you shouldn't need to open additional ports.
Native mobile notifications Optional 443 TCP to barb.hipch.at

You can whitelist all of barb.hipch.at or barb.hipch.at/android and barb.hipch.at/ios individually to enable mobile push for iOS and Android mobile devices

Hipchat video Optional

Server: 443 TCP to video.hipchatserver.com

Clients: 10000 UDP and 443 TCP to hipchat.me

The Hipchat nodes require one-time access to the central Video server to register themselves and create a trust keypair.

Clients require additional open access on ports 10000 (with 443 as a fallback) to access the video service.

Add-ons Optional 443 TCP to marketplace.atlassian.com and marketplace-cdn.atlassian.com Used for retrieving Add-Ons listings from the Atlassian Marketplace. Add-ons may require additional access to function correctly.
Analytics reporting ("Phone home")

Optional

(recommended, please!)

443 TCP to https://hipchat-server-stable.s3.amazonaws.com/ohaibtf.html We use the statistics reported to these servers to help make Hipchat better!

Cluster configuration requirements 

Hipchat node configuration

  • The node must have a static IPv4 address. (This IP should not be accessible by the public internet.) See the official Ubuntu network configuration documentation for more information.
  • The node must be configured in the UTC timezone, and must keep the time synchronized using NTP. (This requires access to port 123 over TCP/UDP.)
  • The node must have a unique hostname. This name must be unique among all members of the cluster.  (This requires access to port 53 over TCP/UDP.)

    tip/resting Created with Sketch.

    Hostname hints

    For clarity in logs and troubleshooting, the host name should also be different from the public DNS entry used to access the cluster. (We know it's tempting to just call it hipchat, but resist!)

    You can set the hostname on a Hipchat node by logging in with SSH and using the following command:

    sudo dont-blame-hipchat -c "hostnamectl set-hostname hipchat1.example.com"

    If your DNS server does not resolve the hostname you set, you can also add an entry similar to the one below to the /etc/hosts/ file on each node.

    127.0.0.1   hipchat1.example.com

Reverse proxy configuration

You may use any reverse proxy that meets the following requirements:

  • Must be configured for HTTP persistent connections (also called http-keepalive).

  • Must support HTTPs endpoints and SSL offloading. 
  • This will terminate SSL for the cluster, so that all communication within the cluster is unencrypted.

  • Must be configured to forward traffic from port 443 to port 80.
  • Must be configured with a DNS record to establish SSL trust, and so the chat clients can access it.
  • Must be configured with an SSL certificate for connections on port 443. 

If you will be using publicly hosted Apps, add-ons, or integrations, the service should be accessible from the public internet so that these external services can send responses back to the cluster.

tip/resting Created with Sketch.

See the Hipchat Knowledge Base article for more detailed instructions on setting up a basic NGINX reverse proxy for a small-scale Hipchat Data Center.

Redis cache requirements

  • Deploy a Redis cache using version 3.2. 
    The official Redis basic deployment documentation is here, however you may also use a package manager for your environment.  
  • Use the default Redis port of 6379.
  • Record the address of the instance. 
  • For security purposes, we recommend that you enable authentication, and then change the default password. (Make sure you record these credentials for later.) 
  • Enable persistence by setting the following configuration values:

    appendonly no
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-min-size 64mb
    auto-aof-rewrite-percentage 100
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    maxclients 10000
    repl-timeout 60

tip/resting Created with Sketch.

See the Hipchat Knowledge Base article for more detailed instructions on setting up a basic Redis cache for Hipchat Data Center.

NFS volume requirements

  • Must be NFS v4.
  • Must be a minimum of 40GB.
  • This volume must be accessible anonymously (with read and write permissions) by the Hipchat nodes.
  • Root squashing must be disabled. (Learn more.)

tip/resting Created with Sketch.

See the Hipchat Knowledge Base article for more detailed instructions on setting up a basic NFS server for Hipchat Data Center.

Postgres database requirements

  • Deploy a Postgres instance using version 9.5. Use the the official Postgres installation guides found here.
  • The instance should be configured in the UTC timezone, and must use NTP to stay synchronized with the rest of the cluster.
  • Record the IP or DNS address of the host, or an endpoint that can be used to access it (such as a dedicated load balancer for the database). 

  • Set the database to use UTF-8. 
  • Set the max_connections to 1000.

tip/resting Created with Sketch.

See the Hipchat Knowledge Base article for more detailed instructions on setting up a basic Postgres database for Hipchat Data Center.

Once the Postgres instance is running, create a database and service user:

  1. Create a database on the Postgres instance. The database name must:
    • start with a letter
    • be between 8 and 30 characters
    • only include letters, numbers, dashes, periods, and underscores
  2. Create a user to access the database. (Do not use the Postgres SUPERUSER account.) 
  3. Make sure that the new user has GRANTS ALL access to the database you just created.
tip/resting Created with Sketch.

For clarity, we recommend putting the word 'hipchat' in the database name, and in the name of the user that you create to access it. This will help you if you ever need to troubleshoot.

ℹ️   Need some help? Click here for some example commands
# create hipchat_postgres database
sudo -u postgres psql postgres -c "CREATE DATABASE hipchat_postgres"

# create dedicated user, set $PASSWORD (replace the variable with the desired password)
sudo -u postgres psql postgres -c "CREATE USER hipchat_user"
sudo -u postgres psql postgres -c "ALTER USER hipchat_user PASSWORD '$PASSWORD';"

# give hipchat_user access to database
sudo -u postgres psql postgres -c "ALTER ROLE hipchat_user WITH LOGIN;"
sudo -u postgres psql postgres -c "GRANT ALL ON DATABASE hipchat_postgres TO hipchat_user;"

Set up the Hipchat VM

In this step you'll create the virtual machine using the OVA file, and configure it. 

Download and deploy the Hipchat OVA

  1. Create the VM on your VMware host. This process may vary depending on your VMware environment and your organization's standard procedures. 
  2. Load the OVA file to create the new VM.
    If you're deploying to a VMware instance that has access to the internet, you can sometimes download the binary directly using the URL in the release notes
    Otherwise, you can download the file locally, and then transfer it to the VMware host to deploy the VM. Links to the binaries for both the Production and Beta versions are in the Hipchat Data Center release notes

    tip/resting Created with Sketch.

    If you're having trouble downloading the Hipchat OVA file using your browser, you can use a command like curl or wget.

    $ curl https://s3.amazonaws.com/hipchat-server-stable/dc/HipChat.ova --output HipChat.ova
    $ wget https://s3.amazonaws.com/hipchat-server-stable/dc/HipChat.ova

Configure the Hipchat VM network

Once you've deployed the VM, you'll configure networking.

From the VM's console:

  1. Edit the etc/network/interfaces file. (You can learn more about this file in the official Ubuntu 14.04 network configuration documentation.)
    You can use the following command to edit it in vim, but you can use another text editor if you prefer.

    vim etc/network/interfaces


  2. Change the iface eth0 inet line from dchp to static.

    tip/resting Created with Sketch.

    If you're using vim, you'll need to press i to get into "insert" mode which allows you to type.

  3. Add a line for address, and set its value to the IP address you allocated for your Hipchat node. (If there was already an address line, replace the existing/default IP with the one you'll be using.)

  4. If they're not already included, make sure you have lines for netmask and gateway. (These are usually the same for any server on your network.) 
    When you're done, your file should look something like this:

    auto eth0                  # turn on automatically
    iface eth0 inet static     # set this to static
    address 10.0.0.100         # the IP address you allocated for this Hipchat node
    netmask 255.255.255.0      # network mask 
    gateway 10.0.0.1           # outgoing traffic 
  5. Save the file.

    tip/resting Created with Sketch.

    If you're using vim, you'll need to press Esc to exit insert mode, then type colon ( : ) and x.

  6. Next run the following two commands to restart networking and apply your changes.

    sudo ifdown eth0
    sudo ifup eth0

Set the node hostname

The node must have a unique hostname, which must be unique among all members of the cluster. (We know it's tempting to just call it hipchat, but resist!)

The Hipchat node's name should also be different from the public DNS entry used to access the cluster. (The node requires access to port 53 over TCP/UDP to resolve DNS.)

sudo dont-blame-hipchat -c "hostnamectl set-hostname hipchat-worker-1.example.com"

If your DNS server does not resolve the hostname you set, you can also add an entry similar to the one below to the /etc/hosts/ file on each node.

127.0.0.1   hipchat-worker-1.example.com

Optional - Set custom NTP servers

The Hipchat node must be configured in the UTC timezone, and must keep the time synchronized using NTP. (This requires access to port 123 over TCP/UDP.)

If your deployment has access to the internet, you can skip this configuration step and use the default Atlassian NTP servers provided. (These are 0.atlassian.pool.ntp.org, and 1.atlassian.pool.ntp.org

If your environment does not have access to the internet, or if your organization runs its own NTP servers and you wish to use them, you should configure your Hipchat node to use an NTP service inside your network.

To change the NTP servers use the following command. Replace "time1.example.com,time2.example.com" with a comma-separated list of your own NTP servers.

hipchat service --ntp-servers time1.example.com,time2.example.com 


What's next? - The Hipchat Data Center deployment process

Once you've deployed the component services for your Hipchat Data Center cluster, you'll follow the instructions at Configure Hipchat Data Center nodes, then Configure optional Hipchat Data Center features, and Verify and troubleshoot your Hipchat Data Center deployment to complete the process. Then allow users to get online and get chatting!

Last modified on Jun 19, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.