Summary

The purpose of this knowledge article is to describe a couple scenarios where automation rules suddenly disappeared/vanished without any Jira user deleting them from the UI.

Environment

• Jira Server/Data Center on any version from 8.20.0
• Automation For Jira on any version below 9.0

Diagnosis

Diagnosis for any root cause

• The impacted rules all had an action that is sending a Webhook or Message to another application, such as the actions listed below:
  • Send HipChat message
  • Send Microsoft Teams message
  • Send Slack message
  • Send Stride message
  • Send Twilio message
  • Send web request
• Go to the global automation administration page (in ⚙ > System > Automation rules)
  • Search for the the name of a rule that disappeared, by using the filter field, and verify that it is missing:
    
  • Go to ... > Show audit logs, filter with the Config Changes status, and note that there is no entry showing that the rule was deleted, but only entries showing that the rule was created (and edited in some cases)
    
• Log into the Jira DB

  • Run the SQL query below, which is meant to query the audit logs to search for rules that were deleted. If the missing rule is not listed in the result, or if there is no result at all, then it's another proof that the rule was not deleted:

    SELECT 
    ai."CATEGORY" AS "Rule Result", 
    ai."CREATED" AS "Deletion Time", 
    ai."OBJECT_ITEM_NAME" AS "Rule Name", 
    ai."SUMMARY" AS "Rule Summary", 
    aici."FIELD_NAME" AS "Action Type"
    FROM public."AO_589059_AUDIT_ITEM" ai 
    INNER JOIN  public."AO_589059_AUDIT_ITEM_CGE_ITEM" aici
    ON ai."ID" = aici."AUDIT_ITEM_ID"
    WHERE aici."FIELD_NAME" in ('com.codebarrel.automation.audit.rule.deleted');

  • Run the SQL query below after replacing <RULE_NAME> with the name of the missing rule. If you get a result, it is another indication that the rule is still there and never got deleted:

    select * from "AO_589059_RULE_CONFIG" WHERE "NAME" = '<RULE_NAME>';

    • Example of SQL query

      select * from "AO_589059_RULE_CONFIG" WHERE "NAME" = 'Rule using MS Team Webhook';

    • Example of result

      |ACTOR_KEY|AUTHOR_KEY|CAN_OTHER_RULE_TRIGGER|CLIENT_KEY                  |CREATED                |DESCRIPTION|ID |NAME                      |NOTIFY_ON_ERROR|PROJECT_ID|PROJECT_TYPE_KEY|STATE  |UPDATED                |
      |---------|----------|----------------------|----------------------------|-----------------------|-----------|---|--------------------------|---------------|----------|----------------|-------|-----------------------|
      |admin    |admin     |false                 |com.codebarrel.tenant.global|2024-10-29 09:27:06.570|           |3  |Rule using MS Team Webhook|FIRSTERROR     |          |                |ENABLED|2024-10-29 09:27:06.570|

Diagnosis specific to Root Cause 1

• Automation For Jira is on a version lower than 9.0.1 (it was at some point on version 9.0.1 or above, but it got downgraded)

• Checking the Jira application logs, you might see an error similar to the one below, including the name of one of the rule that disappeared:

  2024-10-31 09:49:01,155+0000 http-nio-8080-exec-3 url: /jira/secure/AutomationGlobalAdminAction!iframe.jspa; user: admin WARN admin 589x364x1 n70q1s 172.29.219.178,172.50.0.2 /secure/AutomationGlobalAdminAction!iframe.jspa [c.c.a.api.store.RuleConfigToBeanTransformer] Error while converting rule config: 3 Rule using MS Team Webhook
  com.codebarrel.api.JsonSupport$JsonDeserializationException: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)
   at [Source: (ByteArrayInputStream); line: 1, column: 15] (through reference chain: com.codebarrel.automation.rulecomponent.jira.action.msteams.MicrosoftTeamsNotificationActionConfig["webhookUrl"])
  	at com.codebarrel.api.JsonSupport.toObject(JsonSupport.java:107)
  	at com.codebarrel.api.JsonSupport.toObject(JsonSupport.java:124)
  
  ...
  
  Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)
   at [Source: (ByteArrayInputStream); line: 1, column: 15] (through reference chain: com.codebarrel.automation.rulecomponent.jira.action.msteams.MicrosoftTeamsNotificationActionConfig["webhookUrl"])
  	at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
  	at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1752)
  	at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1526)
  	at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1431)
  	at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseString(StdDeserializer.java:1424)
  	at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:48)

Diagnosis specific to Root Cause 2

• Automation For Jira is on version 9.0.1 or any higher version (and it was recently upgraded from a version lower than 9.0.1)

• Checking the Jira application logs, you might see an error similar to the one below:

  2024-11-08 16:32:54,246+0000 http-nio-8080-exec-84 WARN admin 992x40711x1 1ew541j 172.29.182.65,10.9.31.44 /secure/AutomationGlobalAdminAction!iframe.jspa [c.c.a.api.store.RuleConfigToBeanTransformer] Error while converting rule config: 3 Slack notification
  com.querydsl.core.QueryException: Caught PSQLException for insert into "public"."AO_589059_RULE_SECRET" ("KEY", "VALUE")
  values (?, ?)
  	at com.querydsl.sql.DefaultSQLExceptionTranslator.translate(DefaultSQLExceptionTranslator.java:50)
  	at com.querydsl.sql.Configuration.translate(Configuration.java:459) 

Cause

Root Cause 1

The reason why some automation rules are missing is because:

• These rules are sending Webhooks/Messages using the Masked Secret key feature, which was introduced with Automation For Jira (A4J) 9.x as per the Automation release notes page
• The Automation For Jira (A4J) version that is currently installed with Jira is on version lower than 9.0, which does not support this feature

This situation might happen if:

• Jira was upgraded at some point to any version from 9.11.0 (which ships with A4J 9.x), or if A4J was upgraded via the Manage Apps page to 9.x or higher
• Then some automation rules using Webhooks with the "Marked Secret Keys" feature were configured
• Later on, the Jira version was rolled back to any version below 9.11.0 (or A4J was downgraded to any version below 9.0)

In such case, we end up with an A4J version that does not recognize the Secret Keys used by these rules. As a result, A4J fails to render these rules, and no longer shows them in the UI, making the users believe that these rules were deleted, while they actually still exist.

Root Cause 2

The reason why some automation rules are missing is because:

• These rules were configured to send Webhooks/Messages while A4J was still on a version lower than 9.0.1 with a Webhook URL which size exceeds 255 characters
• Such character restriction did not exist in A4J on versions below 9.0.1 (before the introduction of the Masked Secret key feature), therefore it was possible to configure Webhook URLs with more than 255 characters
• The 255 character limit that was introduced in A4J 9.0.1 is actually due to a bug tracked in JIRAAUTOSERVER-972 which was caused by the introduction of the Masked Secret key feature

Solution

Solution for Root Cause 1

• Go to the page ⚙ > Manage Apps > Manage Apps
• Look for the Automation For Jira app
• Upgrade it to any version from 9.0

Solution for Root Cause 2

Even though the bug JIRAAUTOSERVER-972 was fixed in A4J 9.0.5 and 9.2.1 (there is no longer a character limit of 255), the rules that vanished can no longer be recovered if A4J was upgraded first to a non fixed version (such as 9.0.1, 9.0.2, ..., 9.2.0, etc...) at some point.

For this reason, it will be necessary to manually re-create the rules that vanished. To identify which rules might have vanished due to this bug and the character size limit, we recommend checking the workaround section of the bug.