Synchronization with JIRA User Server fails - An invalid XML character was found in the element content of the document
Symptoms
Fisheye/Crucible fails to synchronize users when using JIRA as a user server and JIRA is connected to an external user directory:
The following appears in the atlassian-fisheye-YYYY-MM-DD.log
:
2014-05-07 00:05:42,960 ERROR [qtp1962215609-30641 ] fisheye DefaultUserManager-createTrustedUserLoginImpl - Communication error with crowd
com.cenqua.fisheye.user.AuthenticationException: Problem communicating with Crowd
at com.cenqua.fisheye.user.crowd.CrowdAuth.recreateAuth(CrowdAuth.java:316)
at com.cenqua.fisheye.user.DefaultUserManager.createTrustedUserLoginImpl(DefaultUserManager.java:871)
at com.cenqua.fisheye.user.DefaultUserManager.createTrustedUserLogin(DefaultUserManager.java:847)
at com.cenqua.fisheye.user.DefaultUserManager.createTrustedUserLogin(DefaultUserManager.java:856)
at sun.reflect.GeneratedMethodAccessor474.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
at com.sun.proxy.$Proxy34.createTrustedUserLogin(Unknown Source)
at com.atlassian.fisheye.trustedapplications.FisheyeUserResolver.resolve(FisheyeUserResolver.java:17)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationFilterAuthenticator.authenticate(TrustedApplicationFilterAuthenticator.java:190)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.authenticate(TrustedApplicationsFilter.java:115)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:97)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at com.cenqua.fisheye.web.filters.CustomIncludableGzipFilter.doFilter(CustomIncludableGzipFilter.java:27)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.cenqua.fisheye.web.filters.ProductInfoFilter.doFilter(ProductInfoFilter.java:44)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
at com.atlassian.streams.fisheye.RequestFilter.doFilter(RequestFilter.java:21)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.cenqua.fisheye.web.filters.UpfrontFilter.doFilter(UpfrontFilter.java:59)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.atlassian.fecru.profiling.ProfilingServletFilter.doFilter(ProfilingServletFilter.java:62)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.xml.bind.DataBindingException: javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 495; An invalid XML character (Unicode: 0x6) was found in the element content of the document.]
at javax.xml.bind.JAXB.unmarshal(Unknown Source)
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:349)
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getUser(RestCrowdClient.java:87)
at com.cenqua.fisheye.user.crowd.CrowdAuth$4.call(CrowdAuth.java:303)
at com.cenqua.fisheye.user.crowd.CrowdAuth$4.call(CrowdAuth.java:283)
at com.cenqua.fisheye.user.crowd.CrowdAuth.withTimer(CrowdAuth.java:125)
at com.cenqua.fisheye.user.crowd.CrowdAuth.recreateAuth(CrowdAuth.java:283)
... 80 more
Caused by: javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 495; An invalid XML character (Unicode: 0x6) was found in the element content of the document.]
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(Unknown Source)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(UnmarshallerImpl.java:514)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:215)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:190)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:241)
... 87 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 495; An invalid XML character (Unicode: 0x6) was found in the element content of the document.
at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:211)
... 89 more
Diagnosis
Make the same call that Fisheye/Crucible does when attempting to sync with JIRA (modified to expand all the data) and redirect the output to a file to see the bad data:
curl -u <user>:<password> -s "https://jira.atlassian.com/rest/usermanagement/1/search?entity-type=user&start-index=0&max-results=-1&expand=user" > expanded_results.xml
expanded_results.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><users expand="user"><user name="jadams"><link href="https://jira.atlassian.com/rest/usermanagement/1/user?username=jadams" rel="self"/><first-name></first-name><last-name></last-name><display-name>John Adams</display-name><email>john.adams@atlassian.com</email><password><link href="https://jira.atlassian.com/rest/usermanagement/1/user/password?username=jadams" rel="edit"/></password><key>1:aaa21e74-8975-4639-96be-a7ea7d859ef2</key><active>true</active><attributes><link href="https://jira.atlassian.com/rest/usermanagement/1/user/attribute?username=jadams" rel="self"/></attributes></user><user name="badams"><link href="https://jira.atlassian.com/rest/usermanagement/1/user?username=badams" rel="self"/><first-name>Bob</first-name><last-name>Adams</last-name><display-name>Bob Adams</display-name><email>badams@atlassian.com</email><password><link href="https://jira.atlassian.com/rest/usermanagement/1/user/password?username=badams" rel="edit"/></password><key>3:2d�c��F��A���O</key><active>true</active><attributes><link href="https://jira.atlassian.com/rest/usermanagement/1/user/attribute?username=aamin-tehrani" rel="self"/></attributes></user>
Cause
A bug in Crowd permits the storage of bad XML: CWD-3748 - Crowd does not check if the LDAP externalId attribute value can be represented in XML
JIRA functions properly for the most part with this bad data although user rename fails. The corresponding bug in JIRA: JRASERVER-36220 - JIRA Connecting to Open LDAP which proxies to Active Directory Returns Invalid XML Document When an Embedded Crowd Auth Request Is Made
Workaround
Connect to the external user directory from Fisheye/Crucible.
Resolution
Upgrade to JIRA 6.2.3.