Summary

When you are doing a Jira audit log export from UI it fails with a 504 gateway timeout.

Environment

[Jira Software Cloud]

Cause

Currently, when Jira admins are trying to export the audit events, the exporting of the Jira audit log fails with a 504 Gateway Timeout Error as audit logs are huge in size. This is due to the public bug - Exporting Jira audit logs returns a 504 Gateway Time-out Error when audit events are huge in size.

Solution

To download the Jira audit logs, we have developed a Python script that we can use to extract the logs using the Python script.

The maximum number of audit events fetched by the Python Script in one run will be 10K. So, if there are more than 10K records, you have to run the script in batches of 10K.

Steps to manually download the logs using the Python script.

Step 1

Navigate to your Jira audit log section after logging in to your Jira site (Settings >Audit log). 

Step 2

To check the APIs that are getting called, please follow these steps:

1. Right-click on the page.

2. Choose the 'Inspect' option.

3. Open the 'Network' tab.

Here, you'll be able to see the APIs that are being invoked. You should find an API view that includes 'count' as a request parameter. By clicking on this, you can navigate to the 'Response' section to verify if it displays the same total number of rows as the User Interface (UI) shows.

As you scroll down to the bottom, you can confirm whether this API provides the total data you require. In this instance, the API should be delivering a total of 672 rows.

Step 3

The Python script provided will be utilized to download the data in a CSV format. Save this script as 'audit_log_csv_exporter.py'

import requests
import pandas as pd

def get_json_and_convert_to_csv(url):
    # Define your authentication cookies
    cookies = {
        'ajs_anonymous_id': '<ajs_anonymous_id>',
        'atlassian.account.xsrf.token': '<atlassian.account.xsrf.token>',
        'atlassian.xsrf.token': '<atlassian.xsrf.token>',
        'tenant.session.token': '<tenant.session.token>',
        'JSESSIONID': '<JSESSIONID>'
    }

    # Make a GET request to the API with your cookies
    response = requests.get(url, cookies=cookies)

    print(response)
    # Convert the response to JSON
    json_data = response.json()

    # Flatten each JSON object in the list
    data = []

    for item in json_data['records']:
        # Extract the data for each column
        opDetails = ''
        for changedValue in item.get('changedValues', []):
            to = changedValue.get('to', '')
            opDetails = opDetails + '[' + changedValue['name'] + '] was changed to [' + to + ']' + '\n'
        associatedItems = ''
        for associatedItem in item.get('associatedItems', []):
            if 'objectName' in associatedItem:
                associatedItems = '[' + associatedItem['objectName'] + ']'
            if 'objectType' in associatedItem:
                associatedItems = associatedItems + '[' + associatedItem['objectType'] + ']'
            if 'parentName' in associatedItem:
                associatedItems = associatedItems + '[' + associatedItem['parentName'] + ']'
            associatedItems = associatedItems + '\n'


        row = {
           'ID': item.get('id'),
           'Date': item.get('created'),
           'Time Zone': item.get('timezone'),
           'Event category': item.get('category'),
           'Remote address': item.get('remoteAddr'),
           'Change summary': item.get('summary'),
           'Event source': item.get('eventSource'),
           'Description':'',
           'Author': item.get('author', {}).get('username'),
           'Author name': item.get('author', {}).get('fullName'),
           'Changed object': item.get('objectItem', {}).get('objectName'),
           'Changed object type': item.get('objectItem', {}).get('objectName'),
           'Parent object': item.get('objectItem', {}).get('parentName'),
           'Operation details': opDetails,
           'Associated items': associatedItems
           }
        data.append(row)

        # Convert the processed data to a DataFrame
        df = pd.DataFrame(data)

        # Write the DataFrame to a CSV file
        df.to_csv('output.csv', index=False)

# Replace with your actual API endpoint
get_json_and_convert_to_csv('<url from header section of the API under the Network Tab. For example - from the snippet below>')

3.1 Before running the script, kindly tweak the script as per the required details:-

Here is the modification order that is needed in the script:-

• Cookies section of the script
• URL in the script i.e in the last line of the code, only in the parentheses ('<url from header section of the API under the Network Tab. For example - from the snippet below>').

1. 1. In the Script, you'll need to substitute the current URL with this header url(snippet below for reference). Additionally, change 'count=100' to reflect the total number of rows displayed on the User Interface (UI).
   2. 
   3. In this specific scenario, you would adjust it to 'count=672'(Step 2, Second Image). For example, In this case, the url that will be provided to get_json_and_convert_to_csv in the above script will be - 

https://yoursite.atlassian.com/rest/jira-auditing-plugin/1/view?count=672&maxId=18260&page=1&filter=&timeUnit=&timeUnitValue=&fromDate=&toDate=&currentPage=1&topId=18260&_=1705059214642

• For example - The last line of the code will look like as below:-

Replace - https://yoursite.atlassian.com with your actual Jira site url in the above url

Please copy the cookie details and paste them into any text editor for reference. Then, take each field value and substitute them into the 'cookies' section of the script. Please note, each field value in the cookie is separated by a semicolon(;).

For example - in the below screenshot, From the cookie section, you'll get below details that you need to substitute in the above script(Step 3).

1. ajs_anonymous_id
2. atlassian.account.xsrf.token
3. atlassian.xsrf.token
4. tenant.session.token
5. JSESSIONID

Step 4

To execute your Python script, run the following command on the terminal: 'python3 audit_log_csv_exporter.py'. Upon successful execution, the file 'output.csv' will be downloaded and saved in your current working directory.

If there are more than 10K audit events that has to be exported, follow the below steps before running the script again.

1. Apply filter on the UI for the data to be extracted(date range) and then change the URL in the script as mentioned under step 3.1.
2. Rename the file  'output.csv' which was downloaded in the step 4 to any name of your choice before running the script, otherwise the changes will override with the new run of the script.