Automation for Jira - Troubleshooting guide for situations where automation rules are not triggered
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Rules configured with the Automation for Jira (A4J) application rely on events (issue created, issue updated, new comment added...) to be triggered. When rules are working properly, they should listen to the events they were configured with, and add a new entry to their audit logs, regardless if an action was executed or not. However, under some circumstances, automation rules might "miss" or "skip" some events.
This article provides the list of the most common reasons why a rule was not triggered as expected, and how to identify them.
What does "rules not triggered" mean?
First, let's clarify what "a rule is not triggered" means in the context of A4J:
- Let's assume that we have an A4J rule configured to be triggered when new issues are created
- Let's assume that 2 issue were created after the rule was configured, with keys SCRUM-73 and SCRUM-74. When checking the rule audit logs, is no entry is showing at all for an issue (issue key SCRUM-73 in the example below), then it means that the rule did not get triggered for this issue:
If you expected an automation rule to be triggered upon a certain event, but no entry was recorded in the audit logs, then one of the root cause listed in this KB article might be relevant.
Environment
- Jira Server/Data Center 8.0.0 or above
- Automation for Jira 7.3.0 or above
Diagnosis
To identify which root cause is relevant, you can check the following points below:
- First, make sure that the Automation rule that was not triggered is enabled. If it is disabled as shown in the screenshot below, then Root Cause 1 is relevant
- Look for the option Allow rule trigger in the automation rule (in the Rule Details page). If you see that the automation rule only occasionally does not get triggered, and if the events that were missed where caused by other automation rules, then Root Cause 2 is relevant
- If the rule is configured to be triggered on the "issue created" event:
- if the rule does not get triggered by issues created in a specific project or with a specific issue type, then Root Cause 5 might be relevant
- if the rule does not get triggered by issues imported from a CSV file, then Root Cause 6 might be relevant
- Check the Audit logs of an example of impacted automation rule. If you see that the automation rule is offered triggered, but intermittently skips some issues (as shown in the example below), then Root Cause 3 or Root Cause 4 might be relevant:
- Check if A4J is enabled in the Jira application via the page ⚙ > Manage Apps > Manage Apps
- In case of a cluster of nodes in Jira Data Center, it is important that you confirm that A4J is enabled on each Jira node since we have seen situations where this app might be disabled on a specific node. If A4J is disabled on at least one node, then Root Cause 3 is relevant.
- You can check the state of A4J by:
- either by logging directly into each node (by bypassing the Load Balancer) and going to the page ⚙ > Manage Apps > Manage Apps
or by generating a support zip from each node, and searching for "Automation for Jira" in the application.xml file. If A4J is disabled on a node, you will see something like this:
<plugin> <key>com.codebarrel.addons.automation</key> <name>Automation for Jira</name> <version>8.0.0</version> <vendor>Atlassian</vendor> <status>DISABLED</status> <vendor-url>https://atlassian.com/</vendor-url> <framework-version>2</framework-version> <bundled>User installed</bundled> </plugin>
- If A4J is enabled on all the nodes of the Jira Clusters, but automation rules are randomly skipping some issues, then the next steps is to generate thread dumps and search for the automation threads. If you see that most A4J threads are missing in the thread dumps on at least 1 node, then this node is "not healthy" and Root Cause 4 is relevant:
- Just for comparison purpose, when a Jira node is healthy, you should be able to find the following threads:
- Just for comparison purpose, when a Jira node is healthy, you should be able to find the following threads:
- When analyzing the thread dumps, if you find that all the A4J threads automation-rule-executor:thread-X are in the BLOCKED state (as shown in the example below) or in the TIMED_WAITING state, then Root Cause 7 might be relevant:
Cause
Root Cause 1 - The rule is disabled
If a rule is disabled, it will stop listening to the event it was configured with, and it will no longer be triggered.
Root Cause 2 - The allow rule trigger option is disabled
By default, any new rule has the option Allow rule trigger disabled. In such case, if the event that the rule is listening was fired as a result of another automation rule, this rule will ignore this event.
To allow other automation rules to trigger this rule, you will need to tick this option in the Rule Details page:
You can find more information about this scenario and its resolution in the KB article Automation for Jira - Rules don't get triggered if the trigger came from another rule.
Root Cause 3 - The A4J app is disabled on one node in the Jira Data Center cluster
If you are using a cluster of Jira nodes in a Data Center environment, and if a node did not startup properly, the A4J app might be disabled on that specific node.
In this case:
- whenever an issue is created/updated on that node, the event fired by this issue will be ignored by A4J since it is disabled on that node
- the common symptom that is observed is that automation rules are triggered for most issues, but they are also skipping issues
You can find more information about this scenario and its resolution in the KB article Automation for Jira - Rules are not triggered for some Jira issues due the application being disabled.
Root Cause 4 - The A4J threads are not running on one node in the Jira Data Center cluster
If you are using a cluster of Jira nodes in a Data Center environment, there might be a situation where A4J is enabled on all the nodes but for some reason it failed to initialize its own threads during the startup. The reason why A4J might fail to initiliaze its thread is unclear, but it might happen as a rare corner case.
In this case:
- whenever an issue is created/updated on that node, the event fired by this issue will be ignored by A4J since it is disabled on that node
- the common symptom that is observed is that automation rules are triggered for most issues, but they are also skipping issues
You can find more information about this scenario and its resolution in the KB article Automation for Jira - Rules are not triggered for some Jira issues due to missing threads.
Root Cause 5 - The rule should be triggered on the "issue created" event, but the wrong event is fired in the workflow
Rules that are triggered when new issues are created rely on the "issue created" event to be fired by the Jira application. By default, any workflow is configured to fire the "issue created" event using the issue creation, so that any functionality that relies on this event (Jira notifications, automation rules...) can be triggered.
If the post-functions of the Create transition of a workflow were edited to fire a different type of event, then automation rules that rely on this event will no longer be triggered.
To verify whether a workflow is firing the right event at issue creation:
- Go to the workflow associated to the type of the created issue for which you expected the rule to be triggered
- Click on the Create transition and then the post functions link
- Check the event that is fired in the post functions, and verify that it is either:
- Firing the event of ID 1 (which is the "issue created" event)
- Or explicitly firing the "issue created" event:
- Firing the event of ID 1 (which is the "issue created" event)
Root Cause 6 - The rule should be triggered on the "issue created" event, and the issue was created from a CSV import
Jira comes out-of-the-box with 2 types of CSV import:
- Bulk Create, accessible from the page Issues > Import issues from CSV
- CSV External System Import, accessible from the page ⚙ > System > External system import > CSV
While the 1st type of import is available to users who are granted the Create Issues project permission and the Bulk Change global permission for the project, the 2nd type of import is only available to Jira System Admin users.
Something that is important to note is that the 2nd type of import does not trigger the Issue Created event when issues are created via this import. As a result, any Jira (or 3rd party add-on) functionality that rely on this event to be triggered will not be triggered. Such functionalities include:
- Jira's native Webhook
- Automation for Jira's "Issue Created" Trigger
- Email Notifications relying on the "Issue Created" event
This behavior is tracked in the feature request Issue Created or Issue Updated events are not trapped when importing issues using CSV External System Import.
If the automation rule that did not get triggered is configured with the "Issue Created" event, and if the issues were created using the CSV External System Import, then the behavior is expected and this root cause is relevant. Instead, the Bulk Create CSV import should be used to ensure that the rule will be triggered for the imported issue.
Root Cause 7 - All A4J execution threads are stuck due to a 3rd party add-on
The A4J execution threads automation-rule-executor:thread-X are responsible to process events that are accumulating in the A4J queue. We have seen a situation where all these threads might get into a "BLOCKED" or "TIMED_WAITING" state and stop running because of a 3rd party add-on.
To check if this root cause might be relevant, please refer to the KB article Automation for Jira - The automation executor threads are stuck because of the usage of a 3rd party add-on.
Root Cause 8 - Rules configured with the "Field value changed" trigger are not triggered due to the "Field Security Plugin for Jira" add-on
As explained in the KB article Automation For Jira - Some automation rules using the "Field value changed" trigger are not triggered, the 3rd party add-on Field Security Plugin for Jira might prevent automation rules using the Field value changed trigger from being triggered when the field they are monitoring is updated. To check if this root cause is relevant, please refer to the linked KB article.
Root Cause 9 - Rules configured with the "Multiple issue events" trigger are not triggered if the event was published by another rule
To verify if this root cause is relevant, check the following points:
- The only type of impacted rules are the ones configured with the Multiple issue events trigger
- The impacted rules are already configured with the Allow Trigger option ticked (allowing other rules to trigger these rules)
- The events that these rules should be triggered from are published by other automation rules
- The rules that are publishing the event do not have the option Publish newer style 'IssueEventBundle' ticked in the Publish Event action
If all the points above are verified, then this root cause is relevant, and we are facing the bug JIRAAUTOSERVER-910 - Getting issue details... STATUS
To fix this issue, tick the option Publish newer style 'IssueEventBundle' Publish Event action of all the rules that are publishing the events.
Root Cause 10 - Multiple rules are triggered by the same event and one of these rules is taking more than 5 minutes to complete
In the case where multiple rules are triggered by the same event, Automation For Jira loops over each rule 1 by 1 to process this event. If one of the rule that is processing the event takes more than 5 min to complete, the following happens:
- The event gets cleared from the cache (that's because Automation For Jira automatically clears events from the cache that have not been access for more than 5 min)
The exception below is thrown in the Jira logs
2023-11-07 09:59:18,905+0100 automation-rule-executor:thread-3 ERROR some_user [c.c.j.p.a.service.execution.JiraThreadLocalExecutor] Unexpected error in thread local executor with actorKey 'some_user'. java.lang.IllegalStateException: There is no execution wih id=TenantExecution{tenantExecution=TenantExecution{tenantContext=TenantContext{environment=prod, clientKey='com.codebarrel.tenant.global', tenantId=TenantId{id='00000000-0000-0000-0000-000000bbbbbb'}}, executionUuid='c463420f-d3ae-41cb-9f4c-7d0122769999'}, auditItemId='10360381'} at com.codebarrel.automation.api.execution.DefaultExecutionFinishSynchronizer.lambda$gerRuleInfo$2(DefaultExecutionFinishSynchronizer.java:108)
- Automation For Jira exits the loop even though there were still some rules left to process the event
- Ultimately, some rules miss the event and don't get triggered
This behavior was identified as a bug and reported in JIRAAUTOSERVER-946 - Getting issue details... STATUS
To verify if this this root cause is relevant:
- Check if the event that was missed by some automation rule(s) was processed by at least another automation rule that took more than 5 min to complete (automation rule execution times can be seen in the "Duration" column of the audit logs)
- Check if you can find the exception mentioned above in the Jira application logs
There is unfortunately no ideal/easy workaround for this bug, other than doing the following:
- Identify which rule is taking more than 5 min to process the event that was missed by other rules
- Then, either disable this rule, or optimize it so that it takes less than 5 min to complete
Root Cause 11 - The rule is configured using the trigger "Multiple issue events" and the wrong event was selected
If a rule is configured with the trigger "Multiple issue events", it is important to select the right event that this rule should be triggered on. For example, if this rule is configured to react to the event "Issue Updated", and you expect the rule to be trigger when an issue is transition to a new status, then the event figured by that transition needs to be the same event the rule is configured with.
To check if this root cause is relevant:
- Go to the Rule configuration and check which event is configured with the trigger "Multiple issue events"
- Go to the workflow used by the Jira issue that failed to trigger this rule, check the transition that was used for this Jira issue, and look at the list of post-function configured with this transition
- Look at the event that is fired by this transition
- If the event fired by the transition does not match the event the rule is listening, then this root cause is relevant
- In such case, either edit the workflow transition or the automation rule trigger so that both events match