Performance Testing Framework for Confluence

Before you begin

Before you can run an experiment, you'll need to make sure you meet all the pre-requisites and set up the Performance Testing Framework. 

See Atlassian Performance Testing Framework for how to get started. 

Defining an experiment

Experiments are described in JSON files that define:

• the independent experiment thread(s) to run (in parallel),
• the type of instance under test,
• the size and shape of the instance(s) under test,
• the shape of the data (projects, repositories, users, etc) in the instance(s),
• the size and shape of the client machine(s) that put load on the instance(s), and
• the mix of operations in the workload, and
• the amount of load to apply in each stage of the experiment (the number of client threads).

There is a library of pre-defined experiments under e3-home/experiments, or you can define your own.

The workloads referenced in the experiment JSON file are defined in  e3-home/workloads.

Experiment phases

Performance experiments have a number of phases, which correspond to the following scripts:

1. Provision.py which spins up product instance(s), cluster(s) of worker machines, and other associated infrastructure in AWS. Will automatically create a 'run file' in  e3-home/runs  from your experiment definition.
2. Run.py which actually runs the client workload from the worker machine(s) (and also resets the product instance(s) to a known good state in between stages).
3. Gather.py  which gathers data from an experiment run (either running or finished) off all the machines.
4. Analyze.py  which analyzes the data from an experiment run into charts that help visualize how well the instance(s) performed and all their "vital signs" while under stress.
5. Deprovision.py which deprovisions the test machines and archives the experiment run.

In addition to the global Orchestrate.py scripts, which runs all these phases in order, each phase can also be run individually:

$ cd ./e3
$ ./Provision.py -e <experiment_name>
$ ./Run.py -r <run_name>
$ ./Gather.py -r <run_name>
$ ./Analyze.py -r <run_name>

Workloads

A workload defines the types and distribution of execution scripts run in your experiment. You can only define a single workload per experiment, but each workload can contain multiple scripts. Workload descriptors are stored in e3-home/workloads.

A workload description contains:

• script: the path of a script to be executed during the experiment relative to the e3/execution/grinder directory
• description: the name of the description; used in graphs produced by analysis
• scale: the relative weight to give the experiment in the analysis stage. Can be used to reduce the visual impact of less important or cheaper operations.
• weight: the weight to give to this execution script. All weights should add up to 1.0

Weights

The way the Performance Testing Framework assigns work to client threads and workers means that weights in a workload definition do not have infinite resolution. Here's an example to illustrate this:

{
    ...
    "stages": [
        {
            "clients": 10
        },
        {
            "clients": 20
        }
    ]
}

Because the experiment defined above is running a stage with only 10 clients, the maximum resolution the weights can afford is 1/10 or 0.1. Therefore defining a workload that contains a script with a weight under 0.1 may not be executed.

The framework will also attempt to distribute client threads amongst all worker machines evenly. This means when choosing the number of client threads for each stage, you should aim to ensure the number is divisible by the number of worker machines you have allocated.

Analysis

Performance Testing Framework provides the ability to produce graphs from the data collected from the worker machines as well as the production machines. There are 2 types of analysis scripts included:

• gnuplot
• rrdtool

Calling ./e3/Analyze.py -r <run_name> will produce both types of graphs for a specific run.

To produce only one type of graph:

$ ./e3/analysis/gnuplot/Analyze.py -r <run_name>

or

$ ./e3/analysis/rrdtool/Analyze.py -r <run_name>

Any produced graphs will be stored in the e3-home/runs/<run_name>/analysis  directory.

Note: in order to get graphs from both rrdtool and gnuplot after an experiment run, you must use the Analyze.py script. Orchestrate.py currently only produces graphs using rrdtool. For more information on Analysis and the types of graphs the framework produces, see the dedicated README found at Analysis/README.md.

Confluence datasets

We supply datasets to populate Confluence instances for load testing in experiments differently.

Instances are populated using the file called space-import.zip.xml found at the location specified in a Confluence experiment file. By default, this is the Confluence resources directory. Although you can replace this file with your own space export file, it is not recommended because the existing space contains specific content that is used by the scripts during testing.

Customizing your configuration

In addition to custom datasets, the Performance Testing Framework also supports additional product-specific custom configuration for fine-tuning the shape and size of the product instances and the nature of the load testing.

Anatomy of a basic experiment

The basic experiment shown below consists of two configuration components, Grinder and Confluence. The custom configuration for each is detailed below.

{
    "product": "Confluence",
    "duration": 180000,
    "workload": "confluence-mixed",
    "threads": [
        {
            "name": "1-node",
            "instance": {
                "name": "1-node",
                "version": "6.1.0",
                "template": "ConfluenceDataCenter",
                "properties": [
                    "confluence.data.directory=resources/confluence",
                    "confluence.license.key=<license-key-string>",
                    "confluence.number.of.instances=1",
                    "confluence.number.of.users=60",
                    "confluence.min.users.to.run.experiment=60"
                ]
            },
            "worker": {
                "name": "2-node-worker-cluster",
                "template": "WorkerCluster",
                "parameters": {
                    "ClusterSize": "2"
                }
            },
            "stages": [
                {
                    "clients": 20
                },
                {
                    "clients": 40
                },
                {
                    "clients": 60
                }
            ]
        }
}

Grinder custom configuration

The following properties specify how the load test will be carried out by Grinder:

• duration: how long to run the load test in milliseconds (ms)
• workload: the name of the workload (as defined in e3-home/workloads](./e3-home/workloads) ) to run in each experiment thread
• worker: the client configuration
  • ClusterSize: determines how many nodes will be acting as clients running scripts against the Confluence instance
• stages: how many consecutive times to run an experiment on an instance and how many clients to simulate at each stage

Confluence custom configuration

The following properties can be specified to customize your Confluence instance:

• version: the Confluence product version to spin-up
• properties: a list of properties that Confluence uses during set up
  • confluence.data.directory: path relative to e3-home where Confluence set up data can be found, including the space import and custom mail server configuration
  • confluence.license.key: license needed to set up the Confluence instance (note: slashes need to be escaped in json)
  • confluence.number.of.instances: the size of the Confluence cluster to set up
  • confluence.number.of.users: the maximum number of clients simulated in an experiment thread over all stages
  • confluence.min.users.to.run.experiment: in case of some setup error, the minimum number of created users required to run the experiment; default value is 0 -- the experiment will try to run even if no users are successfully created.
  • mailtrap.api.token: optional parameter specifying a MailTrap server to use for testing email notification load. If a token is not provided, it will default to the custom mail configuration file mail-server.properties provided.

Bring your own infrastructure 

The Performance Testing Framework allows you to run tests against your own infrastructure if you don't want to use AWS. Currently it only supports Linux systems. In order to do so you will need to:

1. Set up and configure a product instance and some additional software used to collect performance metrics.
2. Write a configuration file describing your product instance(s).
3. Configure a worker cluster that will be responsible for generating load and write a configuration describing it.
4. Write an experiment file describing the performance test.

Setting up your Confluence instance

This section describes the steps required to set up Confluence so that it can be used to serve experiment requests while also reporting performance metrics.

Note: As part of the set up, you will need to create new test users, as well as provide authentication credentials for the framework to remove / import / modify test data. If you are using the Performance Testing Framework on an instance with important data, it is highly recommended that you back up Confluence first.

1. Create a public/private key pair that will be used to authenticate against each node in your cluster. The same key pair must be used across all cluster nodes.
2. Set up an e3-user with the keypair setup above.
3. Ensure the SSH daemon is running on each node serving the Confluence cluster instances(s)
4. Enable JMX monitoring for your Confluence instance (in most Confluence versions this is enabled by default).
5. Configure your Confluence instance so that secure administrator sessions (WebSudo) are disabled, the Remote API is enabled, and the onboarding plugin is disabled. Optionally, set up a mail server. 

6. Set up test users on your Confluence instance by running ./e3/util/AddUsers.py from the PTF machine. You'll need to pass in the Confluence base URL and the max number of clients you intend to spin up during the experiment, as well as authentication credentials with the appropriate user creation privileges. Make sure your license can support the number of test users to be created.
   Example usage:
   
   

   $ ./e3/util/AddUsers/py --product Confluence --url http://localhost:8090/confluence --users 100

7. Install the collectd daemon (version >= 5.2) and configure it with the example confluence-collectd.conf configuration file contained within this repository. You may need to make some adjustments depending on your Linux distribution / collectd installation. See the dedicated config/README.md for more details. Make sure the daemon is running when the test begins. 
8. Write a json file describing your Confluence instance containing the following fields: List of ClusterNodes connection strings A URL for Confluence server 
9. Copy both the json file describing the instance and the private key file for the instance to your e3-home/instances directory. Both files should have identical names with different extensions .json and .pem respectively.

{ 
  "ClusterNodes": [ 
  "e3-user@confluence-server-node-1", 
  "e3-user@confluence-server-node-2" 
  ], 
  "URL": "http://confluence-server" 
}

Configuring worker nodes

This section describes the steps needed to set up a worker cluster that will execute load against your product instance while also collecting metrics that will be useful in understanding your experiment.

Create a public/private key pair that will be used to authenticate against each node in your cluster. The same key pair must be used across all worker nodes.

For each node in your worker cluster:

1. Install the following:
   • git
   • Java
   • python 2.7
   • collectd (version 5.2 or later)
2. Setup collectd using the worker-collectd.conf configuration file contained within this repository as a template. You may need to make some adjustments depending on your Linux distribution / collectd installation. See the dedicated config/README.md for more details. Make sure the daemon is running when the test begins. 
3. Ensure the SSH daemon is running on the worker node. 
4. Create an e3-user with sudo permissions that can be authenticated using the public/private key pair you created for this cluster. There should not be a password prompt when executing sudo commands. You may have to edit your /etc/sudoers config to include something like the following: e3-user ALL=(ALL) NOPASSWD:ALL 
5. Create a /media/data directory with the e3-user as an owner 
6. Write a json file describing your worker cluster. 
7. Copy the json file that describes the worker cluster and the private key file for the instance into your e3-home/instances directory. Both files should have identical names with different extensions .json and .pem respectively.

Example worker cluster configuration file.

{ 
  "ClusterNodes": [ 
  "e3-user@worker-node-1", 
  "e3-user@worker-node-2", 
  "e3-user@worker-node-3", 
  "e3-user@worker-node-4" 
  ] 
}

Example experiment file

Describe your experiment making reference to the infrastructure configuration files you created above. Because you are not intending to provision your instances with the Performance Testing Framework, you may not need all the information described in the Experiments section above. Below is an example experiment file containing the minimum amount of information needed to run experiments on your own infrastructure.

{
  "product": "Bitbucket",
  "duration": 10000,
  "workload": "bitbucket-mixed",
  "threads": [
    {
      "name": "2-node",
      "instance": {
        "stack": {
          "Name": "two-node-cluster"
        }
      },
      "stages": [
        {
          "clients": 40
        }
      ],
      "worker": {
        "stack": {
          "Name": "four-node-worker"
        }
      }
    }
  ]
}

The Name field shown above refers to the names of the json files containing your cluster and worker configurations.

Pulling it all together with a fully worked example

This example shows how to run an experiment comparing a two node and four node cluster under increasing load levels.

First define the product instance configuration files.

{ 
  "ClusterNodes": [ 
    "e3-user@confluence-server-node-1", 
    "e3-user@confluence-server-node-2" 
  ], 
  "URL": "http://confluence-server-with-2-nodes",   
}

{ 
  "ClusterNodes": [ 
    "e3-user@confluence-server-node-1", 
    "e3-user@confluence-server-node-2", 
    "e3-user@confluence-server-node-3", 
    "e3-user@confluence-server-node-4" 
    ], 
  "URL": "http://confluence-server-with-4-nodes", 
}

Next define the worker nodes.

{ 
  "ClusterNodes": [ 
    "e3-user@worker-node-1", 
    "e3-user@worker-node-2", 
    "e3-user@worker-node-3", 
    "e3-user@worker-node-4" 
  ] 
}

and the identical four-node-worker.json

Copy these four files along with their respective private ssh keys to ./e3-home/instances so that the directory now contains.

• two-node.json
• two-node.pem
• four-node.json
• four-node.pem
• two-node-worker.json
• two-node-worker.pem
• four-node-worker.json
• four-node-worker.pem

Define your experiment in a file named my-experiment.json. In this case, we go from 40 to 200 concurrent connections in 5 stages, each lasting 5 minutes and executing a confluence-mixed workload.

{
  "product": "Confluence",
  "duration": 300000,
  "workload": "confluence-mixed",
  "threads": [
    {
      "name": "2-node",
      "instance": {
        "stack": {
          "Name": "two-node"
        }
      },
      "stages": [
        { "clients": 40 },
        { "clients": 80 },
        { "clients": 120 },
        { "clients": 160 },
        { "clients": 200 }
      ],
      "worker": {
        "stack": {
          "Name": "two-node-worker"
        }
      }
    },
    {
      "name": "4-node",
      "instance": {
        "stack": {
          "Name": "four-node"
        }
      },
      "stages": [
        { "clients": 40 },
        { "clients": 80 },
        { "clients": 120 },
        { "clients": 160 },
        { "clients": 200 }
      ],
      "worker": {
        "stack": {
          "Name": "four-node-worker"
        }
      }
    }
  ]

}

You will need to create a new folder with the name of your experiment, in this case my-experiment, in the ./e3-home/runs directory. If the ./e3-home/runs directory does not exist, create it. Copy your experiment file to ./e3-home/runs/my-experiment/.

Running your experiment

You will need to pass in your admin user credentials to allow the Performance Testing Framework to reset Confluence between stages. Go to the e3 directory and run:

$ ./Run.py -r my-experiment -u <admin_username> -p <admin_password>

Analyzing your experiment data

Once the experiment has completed its run, you'll want to look at the data gathered by the Performance Testing Framework. Gathering and analyzing the data from your instance and worker nodes is as simple as running the following commands from the e3 directory:

$ ./Gather -r my-experiment
$ ./Analyze -r my-experiment

The data files and resulting graphs will be found in the experiment run folder created earlier, such as ./e3-home/runs/my-experiment.

Support

The Performance Testing Framework is provided as is, and is not covered by Atlassian support. Our support team won't be able to assist you with interpreting the test results.

Head to the performance-testing-framework topic on Atlassian Community to see how others have been using the framework, share your experiences, and ask for help from the community.