JIRA fails to post webhooks to external system due to network infrastructure blocking the request.

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community


Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

JIRA is unable to post webhook content to a remote service as per the webhook configuration.

The webhook content is being generated correctly without a problem by JIRA, we can verify this by setting log level to DEBUG for the package: com.atlassian.webhooks.

No errors are being logged.

The actual webhook never reaches its destination.

Diagnosis

Environment

  • JIRA running behind some sort of network infrastructure appliance like a firewall, a content filter, a proxy, etc.

Diagnostic Steps

  • To troubleshoot this issue further, we need to enable DEBUG logging on the package org.apache.http, this will help us see how JIRA sends the webhook request and what kind of response it is getting.

Note: Please note that this logging is very verbose, it should be only enabled for troubleshooting reasons, it should be disabled afterwords.

We will be seeing log messages similar to below in atlassian-jira.log:

2018-07-03 12:20:05,605 Web-Hook-Publisher-0 DEBUG user 1111111x aaaaaax x.x.x.x,x.x.x.x /secure/WorkflowUIDispatcher.jspa [o.a.h.i.client.cache.CacheInvalidator] Invalidating parent cache entry: null
2018-07-03 12:20:05,605 Web-Hook-Publisher-0 DEBUG user 1111111x aaaaaax x.x.x.x,x.x.x.x /secure/WorkflowUIDispatcher.jspa [o.a.h.i.client.cache.CachingHttpAsyncClient] Request is not servable from cache
2018-07-03 12:20:05,605 Web-Hook-Publisher-0 DEBUG user 1111111x aaaaaax x.x.x.x,x.x.x.x /secure/WorkflowUIDispatcher.jspa [o.a.h.i.nio.client.MainClientExec] [exchange: 46] start execution
2018-07-03 12:20:05,605 Web-Hook-Publisher-0 DEBUG user 1111111x aaaaaax x.x.x.x,x.x.x.x /secure/WorkflowUIDispatcher.jspa [o.a.h.client.protocol.RequestAddCookies] CookieSpec selected: ignoreCookies
2018-07-03 12:20:05,605 Web-Hook-Publisher-0 DEBUG user 1111111x aaaaaax x.x.x.x,x.x.x.x /secure/WorkflowUIDispatcher.jspa [o.a.h.client.protocol.RequestAuthCache] Auth cache not set in the context
2018-07-03 12:20:05,605 Web-Hook-Publisher-0 DEBUG user 1111111x aaaaaax x.x.x.x,x.x.x.x /secure/WorkflowUIDispatcher.jspa [o.a.h.i.nio.client.InternalHttpAsyncClient] [exchange: 46] Request connection for {s}->https://webhook.site:443
2018-07-03 12:20:05,613 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.client.InternalHttpAsyncClient] [exchange: 46] Connection allocated: CPoolProxy{http-outgoing-37 [ACTIVE]}
2018-07-03 12:20:05,614 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][r:]: Set attribute http.nio.exchange-handler
2018-07-03 12:20:05,614 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][rw:]: Event set [w]
2018-07-03 12:20:05,614 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][rw:]: Set timeout 0
2018-07-03 12:20:05,614 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][rw:]: Set attribute http.nio.http-exchange-state
2018-07-03 12:20:05,614 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.client.InternalHttpAsyncClient] Start connection routing
2018-07-03 12:20:05,615 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 Upgrade session x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][rw:][ACTIVE][rw][NEED_UNWRAP][0][0][223][0]
2018-07-03 12:20:05,615 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.client.MainClientExec] Connection route established
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.client.MainClientExec] [exchange: 46] Attempt 1 to execute request
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.client.MainClientExec] Target auth state: UNCHALLENGED
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.client.MainClientExec] Proxy auth state: UNCHALLENGED
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][rw:][ACTIVE][rw][NEED_UNWRAP][0][0][223][0]: Set timeout 20000
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> POST /2cdfcf07-2abf-4e19-977a-a37d971d7b73?user_id=user&user_key=user HTTP/1.1
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> Accept: */*
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> Content-Type: application/json; charset=UTF-8
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> Via: 1.1 localhost (Apache-HttpClient/4.4.1 (cache))
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> Content-Length: 18919
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> Host: webhook.site
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> Connection: Keep-Alive
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.apache.http.headers] http-outgoing-37 >> User-Agent: Atlassian HttpClient 0.23.3 / JIRA-7.8.0 (78000) / Default
2018-07-03 12:20:05,616 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][rw:][ACTIVE][rw][NEED_UNWRAP][0][0][223][0]: Event set [w]
2018-07-03 12:20:05,664 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 x.x.x.x:pppp<->188.226.137.35:443[ACTIVE][r:r][ACTIVE][rw][NEED_WRAP][inbound done][][379][0][0][0]: Shutdown
2018-07-03 12:20:05,665 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 0.0.0.0:34266<->188.226.137.35:443[CLOSED][][CLOSED][rw][NEED_WRAP][inbound done][][379][0][0][0]: Shutdown
2018-07-03 12:20:05,665 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.client.InternalHttpAsyncClient] [exchange: 46] connection aborted
2018-07-03 12:20:05,665 I/O dispatcher 8 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-37 0.0.0.0:34266<->188.226.137.35:443[CLOSED][][CLOSED][rw][NEED_WRAP][inbound done][][379][0][0][0]: Shutdown

The exchange part of the log message (exchange: 46) is how you match the Web-Hook-Publisher Thread with the I/O dispatcher thread:

2018-07-03 12:20:05,605 Web-Hook-Publisher-0 DEBUG user 1111111x aaaaaax x.x.x.x,x.x.x.x /secure/WorkflowUIDispatcher.jspa [o.a.h.i.nio.client.InternalHttpAsyncClient] [exchange: 46] Request connection for {s}->https://webhook.site:443

2018-07-03 12:20:05,613 I/O dispatcher 8 DEBUG anonymous [o.a.h.i.nio.client.InternalHttpAsyncClient] [exchange: 46] Connection allocated: CPoolProxy{http-outgoing-37 [ACTIVE]}


What we notice is that the connection gets closed just when JIRA is about to send its webhook payload.

(info) If there are no I/O dispatcher entries in the log, it could be related to the bug  JRASERVER-72995 - Getting issue details... STATUS

After the event Event set [w] logged at 2018-07-03 12:20:05 there should be another log mentioning produce content, instead of that, in our case here we get a Shutdown message and JIRA is not able to send any content on the connection.

  • In order to ensure JIRA is able to post webhook content, let's have JIRA post the webhook to itself on a none existing URI, this way we get a 404 in the apache http client debug log.

We need to configure the webhook to point to a localhost URL, lets say to JIRA itself like this: http://localhost:<JIRA_PORT>/webhooktest 

<JIRA_PORT> can be obtained from the tomcat connector config from JIRA server.xml found at <JIRA_INSTALL_DIR>/conf/server.xml.

This should give us a 404 response to any webhook request, since Jira doesn't have a URI defined as "/webhooktest".

The http client debug log should show messages like below: 

2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-65 127.0.0.1:57201<->127.0.0.1:8080[ACTIVE][r:w]: Event cleared [w]
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.a.h.i.nio.conn.ManagedNHttpClientConnectionImpl] http-outgoing-65 127.0.0.1:57201<->127.0.0.1:8080[ACTIVE][r:r]: 3237 bytes read
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "HTTP/1.1 404 [\r][\n]"
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "X-AREQUESTID: 11111111111x[\r][\n]"
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "X-XSS-Protection: 1; mode=block[\r][\n]"
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "X-Content-Type-Options: nosniff[\r][\n]"
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "X-Frame-Options: SAMEORIGIN[\r][\n]"
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "Content-Security-Policy: frame-ancestors 'self'[\r][\n]"
2018-07-09 11:51:04,038 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "X-ASEN: SEN-L11812363[\r][\n]"
2018-07-09 11:51:04,039 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "Set-Cookie: atlassian.xsrf.token=dddd-dddd-dddd-dddd|111111111111111111111111111111111|lout;path=/;Secure[\r][\n]"
2018-07-09 11:51:04,039 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "X-AUSERNAME: anonymous[\r][\n]"
2018-07-09 11:51:04,039 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "Set-Cookie: JSESSIONID=JSESSIONIDCOOKIE18CHARACTER;path=/;Secure;HttpOnly[\r][\n]"
2018-07-09 11:51:04,039 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "Content-Type: text/html;charset=UTF-8[\r][\n]"
2018-07-09 11:51:04,039 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "Content-Length: 2710[\r][\n]"
2018-07-09 11:51:04,039 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "Date: Mon, 09 Jul 2018 10:51:03 GMT[\r][\n]"
2018-07-09 11:51:04,039 I/O dispatcher 6 DEBUG anonymous     [o.apache.http.wire] http-outgoing-65 << "[\r][\n]"

Seeing the 404 proves that JIRA can actually post content successfully and also is able to receive response (404 in this case).

This hints that the issue is with the external network infrastructure rather than with JIRA itself.

  • Last step is to verify the connectivity between JIRA server and the destination of the webhook.

We can check this either by a browser from the JIRA server or using tools like curl from the command line (curl can be installed on Windows if needed.) 

In case of curl, we need to run the below commands:

curl -v -X POST https://<webhook_host>/<webhook-end-point-id>
and
curl -v -X POST http://<webhook_host>/<webhook-end-point-id>

The commands are trying to verify both http and https connectivity.

In case of using curl, the result can look like either of the 2 examples below:

  • 1st Example: In the example below, we can see that the curl fails to reach the destination network, so the problem is now proved not to be coming from Jira:

    [user@jira_host]# curl -v -X POST http://webhook.site/4444444-0000-1111-22222-333333333
* About to connect() to webhook.site port 80 (#0)
*   Trying 188.226.137.35... Connection timed out
*   Trying 2a03:b0c0:0:1010::230:3001... Failed to connect to 2a03:b0c0:0:1010::230:3001: Network is unreachable
* Success
* couldn't connect to host
* Closing connection #0
curl: (7) Failed to connect to 2a03:b0c0:0:1010::230:3001: Network is unreachable

  • 2nd example: In the example below, we can see that the curl fails because the Jira Server cannot trust the Webhook site it's trying to connect to, since the SSL Certificate from this website is missing from Java's Trust store:

    [user@jira_host]# curl -v -X POST https://webhook.site/4444444-0000-1111-22222-333333333
Trying 188.226.137.35...
Connected to webhook.site (188.226.137.35) port 443 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Server certificate:
subject: CN=webhook.site,OU=...
start date: Oct 07 11:52:26 2020 GMT
expire date: Oct 07 11:52:26 2022 GMT
common name: webhook.site
issuer: CN=Something,DC=Something,DC=net
NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
Peer's Certificate issuer is not recognized.
Closing connection 0
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

  • If a customized java keystore is in use for the Jira instance, we need to check on this as well. This can be done using the httpclienttest-1.0.2.jar file available here. Run the following command

    <jiraJVM>/java -Djavax.net.debug=all -Djavax.net.ssl.trustStore=<path to keystore as used in JIRA config> -jar httpclienttest-1.0.2.jar "https://<webhook_host>/<webhook-end-point-id>"

    The result will show similar responses as the curl command above concerning the same causes.


Cause

  • The root cause of this issue in the 1st example is that Jira is blocked by a network appliance that doesn't allow the request to path through.
  • The root cause of this issue in the 2nd example is that Jira does not trust the Webhook website over SSL. This can happen if the Webhook is using a self-signed certificate which has not been imported into Java trust store

Resolution



Last modified on Aug 23, 2023

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.