1. Support zip or logs

If you are running the product in a container (such as Docker, Kubernetes, etc) and encounter issues with logs being lost after each restart, you have a few options:

1. Create a Support Zip through the UI, if it is accessible.

2. If the UI is not available, you can create a Support Zip using alternative methods, such as:

   • How to create Jira Support Zip via command line

   • How to create a support zip via command line when Confluence is down

3. If neither of the above options is feasible, you can manually zip the contents of the following folders:

   • <application-local-home>/log(s)

   • <application-installation-directory>/log(s)

Once you have created the Support Zip or zipped the log(s) folders, be sure to attach them to your support ticket upon creation or immediately afterward. Additionally, inform the support team if you were able to collect the data before the restart.

2. Review the steps from other topics

When restarting the product because of a different symptom (described below), it is essential to gather data related to that specific symptom prior to restarting the node. This ensures that all relevant information is collected for effective troubleshooting and resolution.

Slowness can be a symptom with various underlying causes. Troubleshooting typically begins by determining whether the issue originates from the "server side," "client side," or the "network."

1. Generating HAR files

Is your operation taking a bit longer to complete in the UI, or is a page loading slowly? If so, capturing a HAR file while you perform the action can be incredibly helpful. To ensure the best results, it's recommended to conduct all browser data captures in an "incognito window" or after clearing all caches, cookies, and site data, followed by restarting the Browser.

• Generating HAR files and analyzing web requests

The HAR file will provide valuable insights, helping us understand where to direct our attention next: whether it's on the server, network, or client side.

2. Generating thread dumps

Thread dumps are signals emitted to the JVM (Java Virtual Machine), prompting it to "write down" what every thread is doing at that precise moment. This data is invaluable when addressing issues related to "slowness on the server," as it provides insight into what might be causing delays.

• Generating a thread dump

It's important to note that thread dumps are most effective when captured at regular intervals, such as "20 times every 3 seconds" (one minute). By examining threads that appear to be "doing the same thing" across multiple "snapshots", we can identify potential slowdowns or bottlenecks in specific areas of execution.

As highlighted in the article, jstack is the preferred method for generating these dumps. If you find yourself without it and need to use kill -3, please rest assured that this action won't stop the product. The kill -3 command simply signals the JVM to produce the thread dumps, and it is not the same as a regular "kill" or "kill -9" command.

Should you opt for kill -3, it will also be necessary to provide a Support Zip that includes "all files" with "no file size limit" selected, or alternatively, you can send the <application-installation-directory>/log(s)/catalina.out file, as this is where the thread dumps are recorded.

3. Generating s heap dump when there are Full GC cycles or OutOfMemoryError (OOM)

If the product is unresponsive or slow due to memory pressure, obtaining a heap dump will be crucial in understanding the root cause.

To investigate potential Full GCs, you can look for the term "Full" in the GC logs, such as <application-local-home>/logs/atlassian-application-gc-*.log. It's helpful to check for any recent timestamps at the beginning of each line, as this can provide valuable context regarding the performance concerns you're experiencing:

grep -a -E -h 'Full.*->' <application-local-home>/logs/*-gc-* -h | sort -nr | uniq | head -100

Or if there are recent OOM (OutOfMemoryError) in the logs, it might be helpful to take a closer look at any recent time entries (beginning of each line that has it):

grep -a -h "java\.lang\.OutOfMemoryError" <application-local-home>/log/atlassian-*.log* -C3

If so, it would be very helpful to generate a heap dump while the issue is occurring:

• Analyze OutofMemory errors in Jira server with heap dumps
• Monitor Memory usage and Garbage Collection in Bamboo

If you already have the options -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=... set for the JVM, please take a moment to check if there’s any recent (and corresponding with the incident time) .hprof file in the folder "HeapDumpPath". If you find one, uploading it would be very beneficial, as these files are generated specifically by the OutOfMemory event and can provide valuable insights.

4. Support zip or logs

1. Create a Support Zip through the UI, if it is accessible.

2. If the UI is not available, you can create a Support Zip using alternative methods, such as:

   • How to create Jira Support Zip via command line

   • How to create a support zip via command line when Confluence is down

3. If neither of the above options is feasible, you can manually zip the contents of the following folders:

   • <application-local-home>/log(s)

   • <application-installation-directory>/log(s)

Unresponsiveness may suggest that the product is either down or operating at such a slow speed that it becomes impractical for use.

1. Confirm if the product is running

On each node of the cluster, check whether the product task or process is currently running. This step is crucial as it will help us understand if it's really down or just terribly slow.

In Linux, for example, this command can be used if executed as root or sudo:

ps -ef | grep -i java

If the product is not running, please start it up and generate a support zip from the affected nodes. Once created, upload these zips to the support request. The support zips will contain historical log data that may be helpful in diagnosing the issue.

Additionally, inform the Support team about the product's downtime and the exact time you restarted it.

2. Refer to the steps from "Product is slow" above

If the product process is currently running, this could indicate a "Product is slow" scenario. Therefore, please refer to the steps outlined in that section instead.

This section serves as a comprehensive overview for any operational failures encountered within the product. This includes issues such as difficulties in opening a page, clicking on links, or attempting to edit content.

1. Screenshots of errors and timing

Screenshots can be incredibly helpful for effective troubleshooting. Key elements to capture include the screen being displayed, the error message, and the time of occurrence.

To maximize the utility of your screenshot, ensure it captures the full URL in the browser, the entire product page, and the exact time the issue occurred, along with the corresponding timezone. If necessary, sensitive information can be obscured using an image editor. This comprehensive approach is essential, as a partial view of the page may not provide enough context, while the complete URL and full page will significantly aid in identifying the problem.

2. Generating HAR files

The HAR file contains key information about the requests made and the responses received. The response status codes are important, along with their content when available. To ensure the best results, it's recommended to conduct all browser data captures in an "incognito window" or after clearing all caches, cookies, and site data, followed by restarting the Browser.

• Generating HAR files and analyzing web requests

Depending on the specific issue, Support may also request a copy of the "Console log" from the browser; however, the HAR file serves as an excellent starting point for troubleshooting.

3. Support zips

There's not much to collect at this initial moment aside from the above and a support zip from all nodes customized for today, no file limit, and all files:

• Create a Support Zip

Support may request additional data depending on what the evidence above reveals.

Product specific

https://JIRA-URL/rest/api/latest/issue/NOT_OK_ISSUE-99999?expand=changelog,names

https://JIRA-URL/rest/api/latest/issue/OK_ISSUE-99999?expand=changelog,names

The issues we encounter are similar to those seen in the web browser; however, we require slightly different data for resolution.

This article serves as a valuable resource for self-guided troubleshooting. While the title indicates a focus on requests from Jira, it is equally beneficial for addressing requests to Jira as well as to other products.

• WebHooks or Web Requests fail with HTTP Status Code 400, 401 or 403 in Jira

1. Request attempted, response received, and timestamp

Provide the following information if possible:

• Request Details: Include the complete URL, parameters, and any payload data associated with the request. Sensitive information such as passwords or authentication tokens can be redacted for security.

• Response Details: Share both the HTTP Response Status Code and the content of the response. If there are any response headers, please include all of them as well.

• Timestamp: Indicate the time (and timezone) when the error occurred. This will help us narrow down the logs more effectively.

It is very helpful to replicate the request using tools like Postman or cURL. Remember to include the -D- parameter to print all headers and content, and share that information with Support.

2. Support zips

There's not much to collect at this initial moment aside from the above and a support zip from all nodes customized for today, no file limit, and all files:

• Create a Support Zip

Support may request additional data depending on what the evidence above reveals.

Product specific

https://JIRA-URL/rest/api/latest/issue/NOT_OK_ISSUE-99999?expand=changelog,names

https://JIRA-URL/rest/api/latest/issue/OK_ISSUE-99999?expand=changelog,names

It is not uncommon for Jira to experience performance issues, such as delays in sending or receiving emails. The underlying causes can vary widely, ranging from internal bottlenecks within Jira to network latency or issues related to mailboxes.

1. Screenshots and types of emails

Jira Software and Jira Service Management use different mail handlers for both incoming and outgoing emails. Therefore, it is important to inform Support about the specific types of emails that are affected.

Please provide screenshots of the relevant cases:

• Emails that have ceased to be sent (including one that was successfully sent shortly before the issue occurred).

• An example ticket that was no longer created through incoming email (including one that was successfully created shortly before the issue arose).

Additionally, include a screenshot of the mail configuration page:

• Configuring email notifications

It is crucial to determine whether the batching notification setting is enabled or disabled.

2. Enabling DEBUG logs

Before you proceed with creating a support zip or gathering logs, it’s important to first enable mail logging and DEBUG as outlined in this helpful article:

• Enable mail debugging mode in Jira Data Center

Once you’ve done that, please allow Jira to run for a few minutes. During this time, try to perform some actions that would typically trigger the email features in Jira, whether that’s sending or fetching emails. When you reach out to Support, share the specific tests you conducted, such as which issue you created or updated. This information will assist us in narrowing down the scope in the logs more efficiently.

3. Support zips

After enabling mail logs and DEBUG mode for a few minutes, and performing actions that may activate the (faulty or slow) e-mail functionalities, you can generate a support zip file. For this, select the parameters: 3 days, 100 MB file size limit, and include all files.

• Create a Support Zip

You may want to disable the DEBUG mode after you're done testing (perhaps by adjusting it to WARN), which will help prevent the application logs from rolling off too quickly. However, you can keep the e-mail logs enabled for ongoing monitoring. It's also important to inform the Support team about your intention to disable DEBUG logs, as they can offer additional assistance if required.

If Jira indicates that there are Index or Cache replication failures, or if you notice discrepancies in the data based on the node to which you are connected, it is advisable to gather the following information for further analysis.

1. Index summary from each node

With the introduction of Document Based Replication (DBR) in Jira 8.12, the relevance of this index summary endpoint has diminished. However, it still holds value in certain situations.

If you can connect to each node, please visit this URL on each of them and copy the output to the support request:

https://node-specific-url/rest/api/2/index/summary

2. Generating thread dumps from each node

Index replication issues frequently arise when replication threads become stuck on certain nodes. To identify these bottlenecks, performing a thread dump on all affected nodes is highly beneficial:

• Generating a thread dump

Ensure that you compress the files and name them according to each respective node. Unfortunately, the files do not contain any data that can assist support in correlating them to their originating nodes.

3. Support zips

Lastly, capture a support zip from all nodes and upload them to the support request. This will be incredibly helpful as it contains the relevant logs and configuration data that Support needs to effectively correlate with the thread dump data.