Automation For Jira (A4J) Data Center - The ultimate troubleshooting guide

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

 

Summary

Configuring and troubleshooting automation rules using Automation For Jira (aka A4J) can be a challenge. The purpose of this page is to list all the useful resource and links that will help you depending on the type of challenge you are facing.

Configuring an automation rule based on a complex use case

(question) You have a complex use case, and you are not sure what is the best way to turn it into an automation rule?

(lightbulb) In this case, we advise that you open the article below, since it includes more than 50 rule templates, and it is actively updated with new examples. It is a great starting point, as you might find a template that is close to what you are trying to achieve. 

Automation Rule Templates

Investigating automation rule execution issues

(question) You already configured you automation rule but it is not working, meaning that it did not run, failed to run, or did not lead to the expected result? You already went through the basic steps listed in Troubleshoot Automation Rules, but they did not help?

(lightbulb)In this case, we need to first identify the type of problem, and then move on to the relevant section of this page based on the problem:

  • Go to the Audit Logs page of the rule:
  • Look for an entry at the time when the rule was supposed to be executed:
    • If you do not find any entry at the time when you expected the rule to be triggered, move on to the Symptom 1 section
    • If you are able to find an entry in the audit logs, check the status of the rule execution and move on to the section that addresses this status

Symptom 1 - The rule was not triggered

For this type of issue, please check the KB article linked below, which lists all the known root causes so far:

Troubleshooting guide for situations where automation rules are not triggered

If this article does not help, we recommend reaching out to Atlassian Support and provide us with the necessary data as explained in Providing Data to Atlassian Support.

Symptom 2 - The rule failed to complete with the SOME ERRORS status

If the rule was triggered but failed with the status SOME ERRORS, look for the error you found in the audit logs in the list below and open the corresponding article or bug report to resolve it:

List of all known errors and their corresponding articles

If these articles don't help, or if the error you see is not listed, we recommend reaching out to Atlassian Support and provide us with the necessary data as explained in Providing Data to Atlassian Support.

Symptom 3 - The rule failed to complete with the FAILURE status

There can be various reasons why a rule execution failed. Please check the articles/bug tickets linked below, depending on the error found in the rule audit logs if any:

Symptom 4 - The rule failed to complete with the IN PROGRESS status

There can be various reasons why a rule is showing as stuck IN PROGRESS. The 2 most common root causes are documented in the links below:

Symptom 5 - The rule completed with the SUCCESS status but did not provide the expected outcome 

Even if the automation rule shows the SUCCESS status in the audit logs, it does not necessarily mean that the automation rule provided the outcome that you were expecting.

Expand the section below to check how to investigate this type of situation:

How to troubleshoot "success" rules that did not provide the expected results

We recommend checking the KB articles listed below depending on the type of action/operation that the rule was executing:

If these articles don't help, we recommend reaching out to Atlassian Support and provide us with the necessary data as explained in Providing Data to Atlassian Support.

Symptom 6 - The rule completed with the SUCCESS status, but was triggered twice instead of just once

As for now, there is only one known scenario which is known to cause a duplicate execution of a rule, which is explained in the article below:

Symptom 7 - The rule completed with the SUCCESS status, but was triggered with a long delay (and the automation queue is piling up)

Expand the section below to check how to investigate this type of symptom:

How to troubleshoot delayed rules, queue size issue, etc...

This type of issue can be complex to investigate. As a first step, we recommend checking the links below which provide ways to check which automation rules are the most resource intensive and might prevent other rules from being triggered on time:

If you see rules in the A4J Performance Insights page that is being executed a very high number of times (compared to other rules), has a very high total execution time, or is being often throttled, then it might be a good idea to look into these rules and:

  • Optimize them by:
    • Making sure that their scope is not global, but limited to specific projects
    • Making sure that there is some IF conditions using JQL queries inside the rule, so that it is only running for a small set of issues
  • Or temporarily disable them if possible

Another point that is worth checking is the state of the A4J queue, which collects all the events/items that need to be processed by automation rules:

  • Query 1

    • The SQL query below will show the size of the A4J queue. Running this query at constant intervals (for example, every 10min) will give you an idea if the queue is constantly growing (to tens of thousands of items) or if its size is under control:

      select count(*) from "AO_589059_AUTOMATION_QUEUE";
  • Query 2

    • The SQL query below will group the queue items by automation rule IDs. If you see that 1 specific Rule ID contains a lot of items in the queue, then it is worth focusing on this rule. It is possible that this rule is configured to be triggered too often, or for too many issues:

      select "RULE_ID",count (*) from "AO_589059_AUTOMATION_QUEUE" GROUP BY "RULE_ID" ORDER BY count(*) desc;
  • Query 3

    • The SQL query below will group the queue items by event types, and should give us an idea of what type of Jira issue activity is contributing to the queue:

      SELECT 
    (aaq."PAYLOAD"::json -> 'event' ->> 'type') AS eventType,
    COUNT(*) AS event_type_count
FROM 
    "AO_589059_AUTOMATION_QUEUE" aaq 
where
    "PAYLOAD" is not null 
GROUP BY 
    eventType
order BY 
    event_type_count DESC;

Lastly, generating Thread Dumps from the Jira application is also a good way to investigate such issue, so that we can see that the A4J theads are doings. For example: are they all stuck running the same class, are they blocked or waiting for some resource to be available, etc...?

If these preliminary steps don't help, we recommend reaching out to Atlassian Support and provide us with the necessary data as explained in Providing Data to Atlassian Support.

Investigating issues related to the A4J upgrade

When upgrading A4J from 8.x to 9.x, or when upgrading to Jira to 9.11.x (which is bundled with A4J 9.x), you might run into various issues, such as the inability to edit or publish automation rules.

If you face any of these problems, we recommend checking the 2 articles below which address these symptoms and how to fix them:

If these articles don't help, we recommend reaching out to Atlassian Support and provide us with the necessary data as explained in Providing Data to Atlassian Support.

Miscellaneous issues

If the issue you are facing is not covered by any of the scenario and article listed above, you might want to take expand the section below and look at the issues listed there:

Miscellaneous issues

If these articles don't help, we recommend reaching out to Atlassian Support and provide us with the necessary data as explained in Providing Data to Atlassian Support.

Useful troubleshooting data

Database tables and SQL queries

The A4J (Automation For Jira) DB tables are stored in table which name starts with AO_589059_.

  • Tables starting with AO_589059_RULE_ are mostly related to rule configuration
  • Tables starting with AO_589059_AUDIT_ are related to the rule audit logs
  • Tables starting with AO_589059_RULE_STAT_ are related to rule execution statistics
  • The table AO_589059_AUTOMATION_QUEUE contains the items/events added to the automation queue

You can find below the list of KB article which provides examples on how to query these tables:

Automation Threads

Below is the list of all the automation threads which come by default with A4J (Automation For Jira):

automation-event-serializer:thread-1
automation-event-serializer:thread-2
automation-queue-claimer:thread-1
automation-rule-executor:thread-1
automation-rule-executor:thread-2
automation-rule-executor:thread-3
automation-rule-executor:thread-4
automation-rule-executor:thread-5
automation-rule-executor:thread-6
automation-TaskTimer:thread-1

(info) Please note that the number of threads might differ from the list below, since this number can be modified as per Automation Service Limits.

The purpose of each main category of A4J threads is described below:

  • Event Serializer threads (automation-event-serializer:thread-X)
    • They listen to Jira issue events, serialize them and store them in the A4J queue in the table AO_589059_AUTOMATION_QUEUE

  • Queue Claimer thread (automation-queue-claimer:thread-1)

    • It reads the events queued in the database

    • Then it passes the events to the executor threads

  • Event executor threads (automation-rule-executor:thread-X)

    • They receive the event from the queue claimer thread

    • They execute the Automation rule(s) that should be triggered from that event

    • They add add an entry in the audit log

Debug packages

In case you are trying to troubleshoot situations where automation rules are either not triggered or triggered with a delay, enabling the packages below with the DEBUB level (via the page ⚙ > System > Logging and Profiling) might help:

com.codebarrel.automation
com.codebarrel.jira.plugin.automation

(warning) Note that, due to the verbosity of such DEBUG packages, it is recommended to only enable them for a short period of time. Alternatively, you might consider enabling the less verbose packages listed below:

com.codebarrel.automation.api.service
com.codebarrel.automation.rulecomponent.jira
com.codebarrel.jira.plugin.automation.event 
com.codebarrel.jira.plugin.automation.queue


Providing data to Atlassian Support

If none of the article listed below helped investigate the issue you are facing, we recommend reaching out to Atlassian Support and provide the information below:

Providing data to Atlassian Support
  • Most types of rule issues (the rule fails with an error, or was not triggered, or did not provide the expected outcome...):
    • Screenshots showing the configuration of the problematic rules (please include the full configuration of these rules, including their trigger/conditions/actions...)
    • Screenshots showing the audit logs from these rules
    • Screenshots showing an example of Jira issue that triggered these rules or was supposed to trigger them
  • In case the issue is related to automation rules not being triggered, or triggered with a delay:
    • Generate a support zip from the Jira application, by following the steps below:
      • Go to ⚙ > System > Logging and Profiling
        • Click Configure logging level for another package 
        • Use com.codebarrel.automation as the package name, and select "DEBUG" for the "Logging Level"
        • Click Configure logging level for another package 
        • Use com.codebarrel.jira.plugin.automation as the package name, and select "DEBUG" for the "Logging Level"
      • Replicate the issue
      • Go back to ⚙ > System > Logging and Profiling
        • Search for the package com.codebarrel.automation and click on the OFF link next to it
        • Search for the package com.codebarrel.automation and click on the OFF link next to it
      • Go to ⚙ > System > Troubleshooting and support tools > Create Support zip and generate a support zip
    • Generate Thread Dumps from the Jira application




Last modified on Dec 17, 2024

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.