Best practices on working with WebHooks in Jira
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
WebHooks have been a feature in Jira for a long time and are very simple to use and configure. This, however, doesn't mean it's the most appropriate feature to achieve the goal of "posting to an external URL when a specific event occurs in Jira".
This article shares some best practices on managing WebHooks and is paired up with Troubleshooting WebHooks in Jira Data Center. Each of the best practices listed here is drawn from issues you can troubleshoot and RCA with the help of that Troubleshooting article.
For a more specific assessment and optimization, you may seek the services of Atlassian Solution Partners or bring your case for discussion on the Community.
Other valuable resources
- Troubleshooting WebHooks in Jira Data Center
- Jira's "Managing WebHooks" documentation
- WebHooks guide on developer.atlassian.com
- How to find the Issues with most data in Jira
Solution
1. Have as few WebHooks as possible
WebHooks are configured to listen to particular events in Jira and their context is always global.
This means that every time an event happens (like issue created, issue updated, comment added...), Jira will execute the filter (JQL) of every WebHook listening to that event in order to know which WebHooks matches (JQL returns a non-empty result) and should be fired. This JQL matchings are performed synchronously: this means Jira executes each JQL sequentially, one-by-one, and on the same Thread responsible for triggering the event (which may be a user request from the Browser).
So if you have 20 WebHooks configured to listen to the Issue Created event, and each have their own JQL specifying each a different Project, all of them will be executed on every Issue creation, on every Project. Jira appends the Issue key triggering the event to each WebHook JQL in order to check if "this Issue, with the WebHook JQL, should fire the WebHook".
For example: a WebHook listening to an Issue Created event with a JQL of "project = JIRA AND component in (Workflows, Dashboards)" will be executed as "project = JIRA AND component in (Workflows, Dashboards) AND key = CONF-12345" when the Issue CONF-12345 is created (notice it's another Project).
You may consider porting some WebHooks to more recently developed event handling technologies or apps such as Automation for Jira, that may have Project context and even more specific triggers such as a particular field update.
2. Have WebHooks filters (JQL) respond as quickly as possible
Since WebHooks have global context, you may have several WebHooks being executed that you know are irrelevant for the Issue at hand — but Jira's implementation of WebHooks has no way of anticipating it before actually executing the filter.
All searches in Jira (WebHooks filters included) are performed on Apache Lucene's index (which is filesystem operation and OS memory/cache) for performance, but you should always keep them as optimized as possible. A too complex or expensive filter will be slow to execute and impact the response time of the operation that triggered the event.
Some JQL functions (native to Jira or provided by 3rd party apps) are very handy and may have a good performance, but when executed for many operations along with other WebHook filters, they may add up to a staggering amount of time (like each filter taking 200ms at most, but if you have 20 WebHooks for that event trigger, that's 4 seconds).
3. Mind the Issues with too much data when posting the full JSON
Another caveat of WebHooks is that they allow to post the whole Issue JSON representation in the WebHook request payload (as the body of the request). This is useful when the external system needs to read some specific data, but can become a point of concern if you have Issues with too much data triggering WebHooks that send the whole JSON: this may lead to long-running Threads, more network load and maybe bottlenecks.
This article on How to find the Issues with most data in Jira may be helpful to match against what Issues are triggering WebHooks.
Issues with too much data may impact other aspects of Jira (like reindex, load time, etc) and if unable to archive, restrict or somehow decommission them for another (newer) Issue, you may want to optimize the WebHooks to either not match them, stop sending the JSON or port them into Automation for Jira or other app that allows only certain fields to be sent in the request payload.
4. Asynchronous WebHooks
Jira (Core) 8.13.25+, 8.20.12+ and 9.2.0+ allow asynchronous WebHooks to be configured (JRASERVER-68174) through the Feature Flag com.atlassian.jira.webhookEventsAsyncProcessing.enabled:
- Manually enable the "com.atlassian.jira.webhookEventsAsyncProcessing.enabled" feature flag with following instruction.
- Restart your Jira instance (or each node in case of DC)
Jira 8.13 important notice!
Jira versions 8.13.21, 8.13.22, 8.13.23 and 8.13.24 were shipped with asynchronous WebHooks by default and not yet the optimal implementation of WebHooks that 8.13.25+ ships.
If you have these versions installed, please consider one of the options below:
- Upgrading Jira to the latest 8.13.* release or the latest 8.20.* release or to any version higher than 9.2.0.
- Disabling async WebHooks through the Feature Flag
com.atlassian.jira.webhookEventsAsyncProcessing.disabled
This makes use of the Web-Hook-Events-Processor Thread pool and queue and perform the JQL matching and data collecting and JSON building outside of the triggering Thread, meaning the user's operation through the Browser won't delay as webHooks are processed.
Keep in mind, though, that each WebHook's filter is still executed in sequence and Issues with too much data may still become a bottleneck — it just won't impact the end-user's request response time.
On large or enterprise level instances — specially with 3rd party SAML/SSO apps — and heavily used, you may need to tweak the WebHookEventsProcessor queue and Thread pool size. Refer to Troubleshooting WebHooks in Jira Data Center for how to assess a queue overflow (and WebHook events being dropped due to a full queue) and adjust the System Properties appropriately (see Setting properties and options on startup):
-Dwebhooks.executor.queue.size=200 -Dwebhooks.executor.thread.pool.size=10
(The above 200 and 10 are the default values)
Increasing the queue size is advised, as the only impact is more allocated memory but the events themselves don't have a noticeable memory footprint. Increasing the Thread pool size though requires closer monitoring in case it starts consuming other resources like DB connections or I/O.
This issue's being tracked by - JRASERVER-74492Getting issue details... STATUS