Unable to create Application links due to "PKIX Path Building Failed" error when fisheye is configured with custom truststore on config.xml
Summary
Fisheye is configured with a custom on $FISHEYE_INST/config.xml, and the Application Link creation from Fisheye to other Atlassian applications(Jira, Bitbucket) is failing with PKIX path building failed
error.
Environment
4.x
Diagnosis
<web-server site-url="https://fisheye.instenv.com">
<ssl keystore-password="Sanitized by Support Utility" bind=":8443" truststore-password="Sanitized by Support Utility" truststore="/var/atlassian/application-data/fecru/ssl-keystore.p12" keystore="/var/atlassian/application-data/fecru/ssl-keystore.p12"><excludeProtocols><protocol>SSLv3</protocol></excludeProtocols></ssl>
<http bind=":8060" proxy-host="fisheye.instenv.com" proxy-port="443" proxy-scheme="https"/>
</web-server>
2022-12-10 10:00:44,408 ERROR [qtp1871612052-170 ] com.atlassian.applinks.core.rest.ui.CreateApplicationLinkUIResource CreateApplicationLinkUIResource-tryToFetchManifest - ManifestNotFoundException thrown while retrieving manifest
com.atlassian.applinks.spi.manifest.ManifestNotFoundException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.doDownload(AppLinksManifestDownloader.java:207) [applinks-plugin-5.4.28_1655717282000.jar:?]
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.access$000(AppLinksManifestDownloader.java:52) [applinks-plugin-5.4.28_1655717282000.jar:?]
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader$1$1.<init>(AppLinksManifestDownloader.java:129) [applinks-plugin-5.4.28_1655717282000.jar:?]
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader$1.load(AppLinksManifestDownloader.java:123) [applinks-plugin-5.4.28_1655717282000.jar:?]
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader$1.load(AppLinksManifestDownloader.java:120) [applinks-plugin-5.4.28_1655717282000.jar:?]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) [guava-18.0.jar:?]
.......
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alert.createSSLException(Alert.java:131) [?:1.8.0_332]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) [?:1.8.0_332]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) [?:1.8.0_332]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) [?:1.8.0_332]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) [?:1.8.0_332]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) [?:1.8.0_332]
......
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) [httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) [httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) [httpclient-4.5.13.jar:4.5.13]
at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:105) [?:?]
at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:131) [atlassian-rest-module-3.4.16_1655717282000.jar:?]
at com.atlassian.plugins.rest.module.jersey.JerseyRequest.execute(JerseyRequest.java:113) [atlassian-rest-module-3.4.16_1655717282000.jar:?]
at com.atlassian.applinks.core.manifest.AppLinksManifestDownloader.doDownload(AppLinksManifestDownloader.java:174) [applinks-plugin-5.4.28_1655717282000.jar:?]
... 214 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456) [?:1.8.0_332]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) [?:1.8.0_332]
at sun.security.validator.Validator.validate(Validator.java:271) [?:1.8.0_332]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) [?:1.8.0_332]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223) [?:1.8.0_332]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) [?:1.8.0_332]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) [?:1.8.0_332]
... 239 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [?:1.8.0_332]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [?:1.8.0_332]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [?:1.8.0_332]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451) [?:1.8.0_332]
... 245 more
Cause
- The Fisheye is configured with a custom SSL Truststore on the
$FISHEYE_INST/config.xml
file - While creating the Application link from Fisheye to other Atlassian applications(Jira, Bitbucket) the Java used by Fisheye is looking to verify the target application SSL cert on the default truststore location
$JAVA_HOME/jre/lib/security/cacerts
and not looking for the trusted cert entry on the SSL Truststore defined on$FISHEYE_INST/config.xml
this is due to the bug FE-7531 - SSL Truststore configuration in config.xml does not work as expected. - So if the target SSL certificate is not added on the default Java Truststore location and only added to the SSL Truststore on the
$FISHEYE_INST/config.xml
file then the Application link creation would throw thePKIX path building failed
error.
Solution
The application link creation request when initiated from the Fisheye is looking to verify the target application URL on the default Java Truststore location $JAVA_HOME/jre/lib/security/cacerts
when no custom Truststore is set on the JVM argument, defining that on the SSL Truststore on the $FISHEYE_INST/config.xml
doesn't get considered due to the bug FE-7531 - SSL Truststore configuration in config.xml does not work as expected.
Resolution 1
- Add the self-signed certificate of the target application to Java's system-wide truststore:
- Java 8:
$JAVA_HOME/jre/lib/security/cacerts
- Java 8:
Resolution 2
It is also possible to use a different truststore by specifying a JVM parameter on the
FISHEYE_OPTS
, -Djavax.net.ssl.trustStore=/path/to/truststore
, where '/path/to/truststore
' is the absolute file path of the alternative truststore. Information on how to configureFISHEYE_OPTS
startup variables can be found here.However, setting this is not recommended because if Java is told to use a custom truststore (eg. containing only a self-signed certificate), then Java will not have access to the root certificates of signing authorities found in
$JAVA_HOME/jre/lib/security/cacerts
, and accessing most CA-signed SSL sites will fail. It is better to add new certificates (eg. self-signed) to the system-wide truststore ($JAVA_HOME/jre/lib/security/cacerts
).