Generating HAR files and analyzing web requests
Purpose
HAR is the short form for HTTP Archive format, which tracks all the logging of a web browser's interaction with a site.
HAR files can be a requirement for troubleshooting issues specifically for problems listed below:
- Performance Issue: slow page load, timeouts when performing a specific task
- Page rendering: incorrect page format, missing information
First-line troubleshooting can be conducted by following this guide. Providing this information to the support team will help speed up troubleshooting.
Before you begin
Please be sure to take note of the Supported Platforms page for the supported browser types.
It is highly recommended that multiple HAR files be generated for comparison. Below are guidelines for effective information gathering:
- Generate a HAR file for an unaffected page (without performance or page rendering issues). Example. Dashboard, Issue View, Issue Search, and Project page.
- Generate a HAR file for an affected page. Generate multiple times to get a better average and capture consistent timing
Solution
Below demonstrates how HAR files are generated depending on your browser variant.
Chrome
Using Chrome to generate HAR files is recommended because it natively provides the Developer Tools to facilitate this. You can visit this link to learn more about Chrome's developer tools.
Toolset
Chrome is bundled with a set of Developer Tools, as covered in the above link.
JavaScript Errors
Look in the Chrome Console for any JavaScript errors.
Network Issues
To diagnose support problems, the network tab can be used:
- Open up Chrome and enable the Developer Tools.
- Select the network tab.
- Enable preserve log
- Load the page with the problem.
- Wait until it has finished loading.
- Review it for any errors, referencing HTTP/1.1 Status Codes for information on what those errors mean.
Atlassian Support may also request a browser debug log. If this information has not been requested, skip to "Providing Information to Support."
To enable the extra browser log:
- Start up the browser with the parameters "--enable-logging --v=1" (This article provides instructions for major platforms)
- A file called "chrome_debug.log" will be generated in the Chrome user default location.
Providing Information to Support
- Bring up the developer tools using one of these methods:
- Using Keyboard Shortcut (⌘⌥I on OS X, Ctrl + Shift + i on Linux, F12 on Windows)
- From Chrome Menu
At the top-right of your browser window, select Tools > Developer Tools.
- Navigate to the Network tab on the Development Tool
- Check the Disable Cache option to prevent caching of resources for this specific page.
- Press the F1 key, then check the option Allow to generate HAR with sensitive data.
Without this option set, Chrome will not include the content of HTTP responses in the file.
Refresh the page to start capturing the browser and server traffic.
Please capture a full page load so we can see the requests made before the problem we're analyzing.
- Complete the steps that trigger or demonstrate your issue. The Developer Tools won't open by default if the error happens on a pop-up. We need to go into the Developer Tools configurations Preference tab and enable "Auto-open DevTools for popups" in the Global section
- Right-click any request on the leftmost panel, then select Copy > Copy all as HAR (with sensitive data).
- Paste the contents in a plain text editor (i.e. Notepad), and save the file giving it the .har extension.
- Before sending the HAR file to Atlassian, ensure that any sensitive information is removed/censored using a text editor (that is, remove passwords, secrets, etc.).
Firefox
Toolset
Firefox comes with an embedded set of developer tools that you can use for troubleshooting.
JavaScript Errors
Look in the console for any JavaScript errors.
Network Issues
Access the Network tab within the Firefox Developer Tools to check for any network errors. For further info on HTTP codes, please look through HTTP/1.1 Status Codes.
Providing Information to Support
- Go to Tools > Web Developer > Network
- Click the cog icon, then Persist logs
- Refresh the page to start capturing the browser and server traffic.
- Note: Please capture a full page load so we can see the requests made before the problem we're analyzing.
- Complete the steps that trigger or demonstrate your issue.
- Click on Export followed by Save As... to save the HAR file
- Before sending the HAR file to Atlassian, ensure that any sensitive information is removed/censored using a text editor (that is, remove passwords, secrets, etc.).
It can also be helpful to obtain an export of the console.
Analyzing Web Requests
The steps taken to analyze the Web Requests captured by the HAR file vary depending on the troubleshooting Performance or Page Rendering issue.
A common tool to view the generated HAR files is HAR Viewer, available as a web application.
Troubleshooting Performance Issues
The relevant information for performance issues is the load time and which request is causing the delay in the browser serving the content to the user. So, an understanding of the definitions used for Web Request is required for effective troubleshooting; see below:
The below is extracted from a HAR file loaded in HAR Viewer or Google's tool, HAR Analyzer.
Highlighting any web request after loading the HAR file will reveal the information below:
Request start time since the beginning
The highlighted request is called after how long from the initial request. Example: Like below, +6.32s means the current request is being called 6.32 seconds after the initial request (Most of the time, the HTML request is the initial one)
Waiting
Amount of time waiting for the server to respond. If this value is high, it could mean:
- If the time waiting is low locally, then the network between your client and the server is the problem. Any number of things could hinder the network traversal. There are a lot of points between clients and servers; each one has its own connection limitations and could cause a problem. The simplest method to test reducing this is to put your application on another host and see if the time waiting improves.
- That the server is busy or suffering a performance issue. Below, we can see that there is around 2 second wait time from the server; this scenario was due to a complex JQL query:
If this is always seen during a specific time of the day, record the time of occurrence and create a support ticket at support.atlassian.com for assistance in identifying the root cause.
Receiving
This is the time the server uses to transfer the required information to the client. Typically, this is where we detect a network issue. See below for an example:
This example above has a 1.6-second wait time, which caused most of the delay in completing the request.
DOM Loaded
DOM loaded means that the browser has received and parsed all the HTML into the DOM tree, which can now be manipulated.
It occurs before the page has been fully rendered (as external resources may still need to be fully downloaded, including images, CSS, JavaScript, and any other linked resources).
Don't confuse with onload event:
The onload event occurs when an object has been loaded. onload is often used within the <body> element to execute a script once a web page has completely loaded all content (including images, script files, CSS files, etc.).
The onload event can be used to check the visitor's browser type and browser version and load the proper version of the web page based on the information.
Ref: http://www.w3schools.com/jsref/event_onload.asp
Page Loaded
Total time taken for the page to be fully loaded. (Inclusive of AJAX call or any REST call from javascript to populate data on external server: Example, Google Analytics)
It does not mean that the page is white without any content until it is fully loaded; the page usually will show after the onload event, and then it will populate the information from the external call. Such as Dashboard Gadget when loading:
If you have external resources to populate the page (Gadgets, external links), It could take a longer time to load the page completely, however, it does not depend on the performance of the server but rather the speed to handle the request from other site/server.
Size
The size of the request being served plays a role in performance issues as well; for a better understanding of how it contributes to delay, see below:
How long will it take to serve the 2.4MB data? A calculation is shown below:
3Mbps = Approximate 0.375MBps (note: B = byte, b = bit)
To get 2.4MB, you will need:
2.4MB/0.375MBps = 6.4 seconds
This depends on the throughput available. Generally, you can run a Speed Test and check the throughput of the server nearest to the hosting datacenter.
A similar view is obtained by viewing the Network tab on the respective browser types listed above.
The steps to analyze are very straightforward if the above definition is clearly understood:
- Search for the delayed responding request (typically the longest bar viewing the total web requests)
- Identify what is the longest waiting time and how long it waited
- Check what is the main contributor towards the delay (Blocking, Waiting or Receiving)
- Run consistency checking by reloading the page multiple time
- Seek assistance from the Support team with the information captured if identifying the server contributes to the delay. For other issues, such as network, reaching out directly to the ISP or internal hosting team for assistance is faster. The information gathered should be sufficient to show the observation.
Troubleshooting Rendering Issue
In common cases, when a page fails to render correctly, the cause should be logged in the Console tab from the Developer Tools. If no reason is logged, try disabling the browser cache by checking the box next to Disable Cache. This will force the browser to render the page from scratch instead of using cached data. Similarly the status code from the HAR file can also help to identify which request could have caused the issue.
Example of an error message seen in the Console tab accompanying the code 404 Bad Request:
{"errorMessages":["Error in JQL Query: Expecting either a value, list or function but got 'IN'. You must surround 'IN' in quotation marks to use it as a value. (line 1, character 11)"],"errors":{}}This is found in the bug JRA-42216 - Getting issue details... STATUS which is also a page rendering issue
It is essential to be able to identify the status code from the HAR files generated as well. Below are a few of typical examples:
- 200 - Success
- 404 - Page not found / Bad Request
- 401 - Unauthorized
- 403 - Forbidden
- 304 - Not Modified (Content is cached)
- 500 - Internal Server Error
For the definition of the error, refer to the HTTP/1.1 Status Code Definition page for details.
After capturing the consistent behavior, provide this information to the support team together with the steps taken for the Support team to work on the potential cause behind the observation.