Please be sure to document the new syntax for all of your macros, namely:

• The full content model for all your <ri:blah> and <ac:blah> elements: what sub-elements are required versus optional (if there are any sub-elements).
• The full set of attributes for each of your <ri:blah> elements: what's required, what's optional, what the allowed values for each attribute are, and most importantly, what exactly each attribute value will do.
  
  

Honestly, I'm not sure a simple table format like you're starting with here will accomplish this, at least not for your macros. Rember, the people who will need to edit the storage format directly will need to understand all of it. We're talking a full schema/DTD description in plain language.

Perhaps one option is to document the high-level stuff in the table but for each element provide a link to a page (or subsection of a large page) that lists the full element declaration from your schema/DTD (personally I think DTD-style element declarations are more human-readable), along with the details? That way at least, people who are familiar with how to read SGML/XML schemas/DTDs can figure out what they can and cannot do.

And to help you walk the line between expressing content models in an XSD manner or a DTD manner, here's the approach the OASIS technical committee that created the DITA specification used to specify content models in a neutral manner that enables people who prefer XSD declarations versus people who prefer DTD declarations to get the basic information they need:

DITA language reference example for the FIGGROUP element

Anecdotally, I was on the OASIS technical committee that produced the DITA 1.1 specification, and I've done a fair bit of DITA specialization work in the past. All the IBM luminaries who developed DITA and trained others taught by using DTDs and DTD declarations, and when I worked on DITA implementations we all worked with DTDs instead of schema to produce our specializations. The reason is simple; DTD-style element declarations are much easier for humans to read and parse. So if you're thinking of taking a shortcut and just pasting in your XSD declarations to explain the content models for your more complex macros, I'd recommend against that. If you guys prefer working with XSD internally, that's great, but either translate those to DTD models for posting in your language reference here, or create a neutral language reference as demonstrated in the provided link.

It can be helpful even for your elements based on XHTML to clearly spell out any restrictions about which Confluence-unique elements can be placed inside them (or not). This is what I mean by specifying the full content model for every element in your XML schema. For example, can I put literally any of your <ac:blah> elements inside of an <li>? How about inside of a <tr>? Believe me, people will have these questions.

Please publish (make available for download) a schema (XSD).

Annotate the schema with human-readable documentation (in xs:documentation elements).

Use an XSLT stylesheet to transform the schema into XHTML-source reference documentation.

Publish the resulting reference documentation on the web.

For examples of human-readable schema documentation, see:

http://www.schemacentral.com/

I thought I'd revisit this page to see if you had removed "(Still considering)" from the "Provide an XSD" to-do item. Nope. It's still there.

Some further thoughts, while I'm here...

Academic discussion: XHTML as your XML source format

There are strong arguments (that I might make myself) against creating XHTML that is valid but subverts the semantics of XHTML, and what I'm about to discuss (I'm deliberately not using the word "suggest") might come close to, or perhaps cross, that line, but I can't help noticing: I've yet to see any Confluence 4.x source documented in the body of this page that I think could not have been coded in XHTML 1.0 Strict (I'm deliberately not referring to (X)HTML5... yet).

For example, instead of coining the ac:macro (and ac:link) and ac:parameter elements, you might have used the XHTML elements object and param.

I realize that this discussion is academic. This - using XHTML as-is, or coining your own XML vocabulary - would have been the kind of early design decision that you made after careful deliberation. And I am not saying that you made the wrong decision. Neither am I saying that this XHTML - with, for example, XHTML object elements in place of ac:macro - would have been what you actually presented in browsers. No, this XHTML would have been your XML storage format. To produce "presentation-level" XHTML, you'd convert those object elements into (probably) object-element-free XHTML.

So why even discuss it? You'd have saved yourself the trouble of having to customize the XHTML schema, because your XML storage format would have been valid XHTML, and I/you/we would have been able to use validating XML editors with the (unchanged) XHTML schema.

I understand that coining your own Confluence-specific XML vocabulary frees you from the constraints of the XHTML vocabulary and any taint of subverting the XHTML semantics to your own proprietary ends. Coining your own vocabulary also offers the potential for tighter validation: you can require certain Confluence-specific markup, rather than expecting, but not being able to use the xHTML schema to validate, certain combinations of XHTML elements (expected by macros.) Having said that (as has been pointed out by Shannon), the W3C XML Schema 1.0 language ("XSD") has some limitations here (DTDs have limitations, too: http://www.w3.org/TR/xhtml1-schema/#why).

Style attributes, but not class attributes?

I note that, in the "XHTML portions" of your source format, you've (so far) avoided any use of class attributes: you've opted for style attributes instead. I'm a little curious about that.

I also wonder why you've wrapped your "square bullet" lists in a <div> (but perhaps that's a typo).

(And I wish that you'd used the default - as in, default CSS implemented by most browsers - vertical spacing for ordered and unordered lists, rather than bringing forward the Confluence 3.x "compact" style, but that's getting off-topic.)

How are you validating your XML source format?

I've asked this question in previous comments, so far, without a response: how are you currently validating your XML source format?

So: you've released the "beta" V1 advanced editor now. Do you realize that you have NOT given us enough information above to support use of this editor? Where are the content models? The specific content models for every possible element.

C'mon Atlassian, this is XML 101 and you are FAILING. Yes, I'm pissed. Just document the XSD or DTD already, in a standard XML language reference spec format. Sheesh, I could have done this in my sleep in 20 person hours or less if you would just show me the DTD/XSD.

BTW, Graham has already given you the keys to the kingdom above. You don't even have to commit your own tech writing resources to the effort. Some other third party volunteered to do a proper language reference for you.

I feel that I am one of several users who have tried but so far failed to encourage you (Atlassian) to publish a schema for your XML storage format.

In your own words, a schema (or DTD) is something that you're "still considering"; at best, most recently, following (but I realize not necessarily prompted by) encouragement from myself and others, "something we want to look into generating at some point". (How's that for a definite commitment? "want to... look into... at some point"!) Not a requirement; not a "given".

I have voted for CONF-24884 - Getting issue details... STATUS but frankly, I'm tired of trying to convince you.

One more attempt to talk sense into the Atlassian PMs and architects regarding your perception that a fully documented language reference for your XML schema isn't desired by enough customers to be worth the development effort.

Premise A: Not many of your existing wiki users have experience with XML-based documentation authoring environments. XML is still a 'big player' or 'bleeding edge' technology in the general world of tech writing and documentation. Because not many of your existing wiki users have experience, they are not familiar with the pain points of working in XML syntax.

Premise B: The four people in this conversation thread (and related CONF-24884) who obviously have significant XML-based authoring experience are all telling you the same exact thing: you must provide a proper language reference and you must provide an XSD/DTD. That's me, Graham, Andreas, and Kathleen.

So: everyone who knows the pain points is telling you that this is mandatory to provide. Everyone else that you are expecting to convince you of the need via "voting" are (no insult intended) ignorant of the pain points and therefore won't be motivated to vote.

In short, you are going about this entirely the wrong way.

I was reluctant to publish the details of the storage format at all. Publishing something makes it part of the API, and as has been exhaustively detailed here, the format is not yet "API-worthy". We wavered for some time between publishing, or keeping it internal until it was "ready", and ultimately decided based on a deluge of feedback that a flawed, ad-hoc documented format beats an unpublished format that might become pure enough to publish in the future.

Software development is a trade-off between competing priorities. The priority for Confluence 4.0 was concentrating on the editor itself and treating the storage format as an implementation detail. Would it have been nice to clean up the flagrant abuse of style attributes before release? Absolutely. Would we have liked to have a published schema on day zero? Absolutely. Would it have been worth the dozen or so major editor bugs that wouldn't have been fixed in order to get that done? Probably not.

Wiki markup, Confluence previous storage format, was proprietary, barely documented, inconsistent, ludicrously broken (c.f. escaping, nested elements, whitespace, macro syntax, block vs inline markup, the interaction of tables and lists etc. etc.), only parseable through a mess of ad hoc Java code and regular expressions held deep in the Confluence codebase, and completely un-amenable to deterministic round-tripping to other formats.

(I'm always amazed, considering the amount of developer time we had to pour into the The Universal Wiki Converter, how wiki markup somehow gets a free pass in the "impenetrable proprietary format" stakes.)

The new format, while not a good XML citizen by any stretch of the imagination, is still orders of magnitude easier for third parties to parse and transform than the previous one. Converting the storage format into a good XML citizen (or creating a public-facing API that is a good XML citizen) is a goal we've had since the beginning, but not one we were able to deliver on in the 1.0 release.

And this one is just for Bill Arconati, who said above:

To understand the content model for a specific element, you'll want to insert it via the WYSIWYG editor and then view the storage format.

Bill, your suggested process will most definitely, absolutely, not yield a "content model". It will instead yield only one possible instance of a content model, from which it will be impossible to extrapolate the actual content model. Do I really need to explain the difference here?

And this one is for Charles Miller, who said above:

Software development is a trade-off between competing priorities. The priority for Confluence 4.0 was concentrating on the editor itself and treating the storage format as an implementation detail. Would it have been nice to clean up the flagrant abuse of style attributes before release? Absolutely. Would we have liked to have a published schema on day zero? Absolutely. Would it have been worth the dozen or so major editor bugs that wouldn't have been fixed in order to get that done? Probably not.

You still have myriad major editor bugs. Until you fix all major (and minor) bugs–an impossible task, btw–your users need a way to work around the bugs. Releasing the new RTE without a way to fix the editor's bugs in the underlying storage format was a huge tactical mistake, and you're still paying for it even now, nearly 6 months after your first release of the new RTE. Why do you suppose that nearly every comment on these various 4.0 Editor feedback pages is negative and scathing? Where are the ardent admirers who really appreciate your new editor and come here to defend the harsh criticisms and complaints that have been posted here for the past 6 months? The best your CEOs could come up with were a handful of cherrypicked tweets and an assertion that they'd received some nice emails from happy customers. As I said then: if this is true then like any product with a user community, there would be equal representation in your discussion forums by both the fanboys and the haters. Which is not the case.

I understand competing priorities for software development; I've been in the field for 25 years. You picked the wrong priorities at first, and now you're doing damage control and triage. That's fine; we all make mistakes. Nobody expects Atlassian to be perfect. But we do expect you to be responsive when it turns out that your strategic and tactical decisions were obvious mistakes and your user community is telling you what they need to recover from those mistakes.

What I'm concerned about here and now in this thread is the apparent risk that Atlassian is going to prolong a fundamental mistake here: which is not giving us the tools we need to work around the bugs with your new editor. It does not matter if your current XML implementation is ugly and malformed and full of hacks. What matters is telling us what your storage logic and rendering engine will accept as valid. If you don't tell us what the exact content models are, we cannot debug the storage format. It's really that simple. It does not matter whether it's ugly; just tell us what it is right now so we can recover the lost ability we've always had in 3.5.x and prior: the ability to work around bugs in your editors by just fixing the underlying storage format to do what we need it to.

Wiki markup, Confluence previous storage format, was proprietary, barely documented, inconsistent, ludicrously broken (c.f. escaping, nested elements, whitespace, macro syntax, block vs inline markup, the interaction of tables and lists etc. etc.), only parseable through a mess of ad hoc Java code and regular expressions held deep in the Confluence codebase, and completely un-amenable to deterministic round-tripping to other formats.

As Jens Melgaard points out very eloquently above, you're making a fundamental error of logic here. Wikis are primarily used by people, not by code. What's difficult for code can often be trivial for people. The main selling point of wiki syntax is in its simplicity and human-friendly nature. It is FAST to work with.  XML and all semantic markup is designed to be good for code at the expense of being good for people. Now, you've made your strategic decisions in this regard and I won't rehash the arguable wisdom of that strategic move. But right here, right now, you have to empower the people to work around the serious and numerous bugs and flaws with your code, which in this specific case means your current very unpolished RTE. And that empowerment takes the form of an advanced editor to directly manipulate the storage format as well as the full documentation for the storage format. Not a handful of examples and a misguided suggestion to just look at a specific instance of a macro if you want to learn its underlying content model (which is an impossible task in many/most cases).

So you've "finalized" this storage format "documentation" as of today. I have many comments:

• You guys really aren't done. <beatdeadhorse>This is unacceptable documentation for an XML language reference. It's a crime that you don't have an XSD or DTD, that you won't publish that XSD or DTD, and that your internal validation code is not validating against an XSD or DTD. You fail at XML 101. I'm frankly shocked that despite several people attempting to educate you guys over the past three months, you still don't "get it".</beatdeadhorse>
• Regarding link bodies, why on earth do you bother with having both a PCDATA link body element and a CDATA link body element? That's just confusing. Why not stick with a single PCDATA body (<ac:link-body>)? Why does your RTE code create so many ugly, difficult to type and difficult to visually parse CDATA link bodies instead of just using <ac:link-body> with no markup inside the body? Seriously, what does the CDATA format for a link body buy you in terms of performance? How often do you ever see raw CDATA code like that in doc-oriented XML markup? I mean, really: you're forcing two different user patterns on your "advanced users" when clearly, one alone would suffice in all use cases (the PCDATA version). 
  • To make the problem crystal clear: where this double pattern presents the biggest problem is in searching a space for potentially "broken" link bodies (aka anchor text aka link text) after renaming a page. You are forcing us to do 2x the amount of searches. One search for the CDATA format of the link body. A second search for the PCDATA format of the link body. Because we won't necessarily know which format was used under the covers, do we?. And before you say "why not just search for the link body string in the rendered RTE text of the page?", please stop and think: do you really want your users to be forced to wade through potentially hundreds of search hits (search string in any possible context) or to wade through only dozens of search hits (search string only in context of a link body)?
  • The only place you should force a CDATA syntax on your users is inside of {code} and {noformat} macros.
• When is the "global search field supports searching for storage format syntax" coming? Your product is still wholly unusable until this feature is back in place.
• Lists: do you support the disc bullet item type? If so, please add that to the syntax reference above.
• Do you support <b> and <i> as alternatives to <strong> and <em>? This is a prime example of what I mean when I have complained about your facile statement that "our storage format is just XHMTL with some stuff tacked on to handle our macros. (If you would just publish your XSD or DTD, we wouldn't have to ask questions like this. If you would just publish a proper language reference, we wouldn't have to ask questions like this.) If you support <b> and <i>, please add that to the syntax reference above.
• NOT NEARLY ENOUGH DETAIL for the {toc} macro. omgwtfbbq? Where are the syntax details for include/exclude or all the other myriad parameters for that macro? Do you realize how much at least some of your customers depend on really specific include/exclude tricks to get {toc} to do all manner of very useful stuff? This is another example of what I mean when I complain LOUDLY that this "just a few examples" approach to your syntax language reference is simply not good enough. Why the frak should we need to ask questions like this?
• Where are the {children} macro details? Like {toc}, {children} is a critical, often-used macro. Of all your macros, I use {toc} and {children} the most.  Again, why the frak should I even have to spend time asking questions like these? 
• In general, your macro documentation is woefully lacking. I have no idea what my options are when it comes to nearly all of your macros. A handful of  examples simply does not suffice. You have no idea how angry I am right now because of this. You are effectively forcing me to deal with a data migration to some other wiki engine because you guys are too freaking lazy to properly document your own syntax that you shoved down our throats. I've wanted to give the Confluence team the benefit of the doubt because I know firsthand what a tough job it is to execute a major architectural change. But honestly, this is crazy bad architectural design and crazy bad documentation support. You didn't start with a rigorous XSD/DTD-based approach to your design and implementation and validation/parsing, and because of that, you're generating tons of new technical debt for yourselves AND you cannot properly support your user base with even the most basic information they need. FAIL.

Still waiting for some statement from Atlassian regarding when and if you plan to actually deliver meaningful, complete documentation on all your macro elements.

You have a crappy bug in your source editor. Why do hyperlinks with underscore characters not show up in source editor?

The hyperlink looks like this: <a href="http://download.java.net/media/jai/builds/release/1 1 3/">

But the hyperlink is this: <a href="http://download.java.net/media/jai/builds/release/1_1_3/">

Why is this?

Rudimentary validation of Confluence XML using a customized XHTML 1.0 Strict DTD

I took a few hours today to:

1. Create a rudimentary DTD for the Confluence storage format, based on the XHTML 1.0 Strict DTD. (Why a DTD and not a schema? To save time. Before today, I had not done any real editing of DTDs for years; it's all been schemas. Even so, doing the same thing with a schema would have taken me longer.)
2. Validate the XML source of a small set of pages (a few hundred) in a test "sandbox" installation of Confluence 4.

I have documented what I have done as a step-by-step procedure (using wording such as "Copy...", rather than "I copied..."). Sorry, these steps are not formatted as an ordered list (I do not yet feel adept enough with the new editor to attempt this).

Create a rudimentary Confluence DTD

Download xhtml1-strict.dtd and its three xhtml1-*.ent files (lat1, special, and symbol) from the W3C website.

Create a copy of xhtml1-strict.dtd and name it confluence.dtd.

Replace the references to the three xhtml1-*.ent files with the actual file contents, so that confluence.dtd is standalone. (This step is not strictly necessary, or even desirable; I did this for expediency.)

Create an entity that lists all top-level proprietary Confluence elements ("top-level", as in, proprietary Confluence elements that are not necessarily always nested inside another proprietary Confluence element). Add the following line near the top of confluence.dtd (just below the comment header):

<!ENTITY % ac "ac:macro | ac:link | ac:emoticon | ac:image | ri:page | ri:blog-post | ri:attachment | ri:url | ri:shortcut | ri:user | ri:space | ri:content-entity">

Allow all top-level proprietary Confluence elements as inline, block, and flow elements. (I did say this was a rudimentary DTD; I don't want to spend too much time guessing undefined rules!) Add a reference to the ac entity to the following existing lines:

<!ENTITY % Inline "(#PCDATA | %inline; | %misc.inline; | %ac;)*">

<!ENTITY % Block "(%block; | form | %misc; | %ac;)*">

<!ENTITY % Flow "(#PCDATA | %block; | form | %inline; | %misc; | %ac;)*">

Add xmlns:ac and xmlns:ri attributes to the attribute list of the html element:

<!ATTLIST html
  %i18n;
  id          ID             #IMPLIED
  xmlns       %URI;          #FIXED 'http://www.w3.org/1999/xhtml'
  xmlns:ac    %URI;          #IMPLIED
  xmlns:ri    %URI;          #IMPLIED
  >

Declare the proprietary Confluence elements and their attributes. Add the following lines to the bottom of confluence.dtd:

<!-- Atlassian Confluence -->
<!ELEMENT ac:macro ANY>
<!ATTLIST ac:macro
ac:name     CDATA          #IMPLIED
  >
<!ELEMENT ac:rich-text-body ANY>
<!ATTLIST ac:rich-text-body
ac:name     CDATA          #IMPLIED
  >
<!ELEMENT ac:plain-text-body ANY>
<!ATTLIST ac:plain-text-body
ac:name     CDATA          #IMPLIED
  >
<!ELEMENT ac:default-parameter ANY>
<!ELEMENT ac:parameter ANY>
<!ATTLIST ac:parameter
ac:name     CDATA          #IMPLIED
  >
<!ELEMENT ac:image ANY>
<!ATTLIST ac:image
ac:border   CDATA          #IMPLIED
ac:height   CDATA          #IMPLIED
ac:width    CDATA          #IMPLIED
ac:align    CDATA          #IMPLIED
  >
<!ELEMENT ac:link ANY>
<!ELEMENT ac:link-body ANY>
<!ELEMENT ac:emoticon ANY>
<!ATTLIST ac:emoticon
ac:name     CDATA          #IMPLIED
  >
<!ELEMENT ri:page ANY>
<!ATTLIST ri:page
ri:content-title CDATA     #IMPLIED
ri:space-key     CDATA     #IMPLIED
  >
<!ELEMENT ri:blog-post ANY>
<!ATTLIST ri:blog-post
ri:content-title CDATA     #IMPLIED
ri:posting-day   CDATA     #IMPLIED
  >
<!ELEMENT ri:attachment ANY>
<!ATTLIST ri:attachment
ri:filename CDATA          #IMPLIED
  >
<!ELEMENT ri:url ANY>
<!ELEMENT ri:shortcut ANY>
<!ELEMENT ri:user ANY>
<!ATTLIST ri:user
ri:username CDATA          #IMPLIED
  >
<!ELEMENT ri:space ANY>
<!ELEMENT ri:content-entity ANY>

Note the use of ANY in these element declarations. I have, so far, deliberately made no attempt whatsoever to define which elements are allowed inside each proprietary Confluence element. This is a major shortcoming. For example, this DTD will currently allow anything - potentially monstrous markup - inside ac:rich-text-body. I might look into fixing that later.

I did not add these declarations all at once. It was an iterative process. Each time the validation process (described below) reported an error for an undefined element or attribute, I went back and added the missing declarations to the DTD.

I do not claim that this is a comprehensive declaration of proprietary Confluence elements. In fact, I doubt that it is complete. I do not even claim that it is complete with regard to the source format documentation provided by Atlassian. All I have done is to add to my copy of the DTD the proprietary Confluence elements and attributes that occur in a small set of pages (a few hundred) on a test "sandbox" installation of Confluence 4.

If you follow this procedure using your own files, I expect that you will meet proprietary Confluence elements and attributes not covered in the declarations listed here. You will need to do as I did, and iteratively add to these declarations, whittling down the validation errors.

Get your Confluence XML source files

Use the free wget tool to get all .txt files from Confluence 4:

"C:\Program Files (x86)\GnuWin32\bin\wget.exe" --user=usr01 --password=pa55w0rd -r --no-directories -A.txt http://server:8090/plugins/servlet/confluence/default/

(Does this method of access require the Confluence WebDAV plugin? I do not know. This particular Confluence server does have the WebDAV plugin installed.)

Curiously, many of the .txt files had 0 length. I'm going to leave that mystery for another day.

Note that the --no-directories option causes wget to save all files to the current directory, without creating subdirectories.

Prepare the source for validation

Write a script (I used VBScript) to:

1. Delete .txt files that are actually plain text files (for example, plain-text attachments to Confluence pages), rather than .txt files that contain the XML source for Confluence pages. (My script sniffs the first character in the file: if it's a <, I keep it. Yes, risibly fallible and simplistic.)
2. Wrap the contents of the other (XML source) .txt files in the following markup, and then save the wrappered content in .xml files (and delete the original .txt files):

<!DOCTYPE html SYSTEM "confluence.dtd">
<html xmlns:ac="atlassian.confluence" xmlns:ri="resource.identifier">
<head>
<title>Title</title>
</head>
<body>

insert .txt file contents here

</body>
</html>

Confluence serves the source of a page as a .txt file containing an XML snippet - the contents of a <body> element, as displayed in the Source Editor plugin - rather than a complete XML document, hence the need to add this wrapper.

Validate the source

Copy confluence.dtd to the directory containing the .xml files.

Write a Windows batch file that uses the free xmllint tool to validate the .xml files.

Here's the heart of the batch file:

for %%f in ("*.xml") do "%_xmllint%" --noout --valid --loaddtd "%%f"

where the temporary environment variable _xmllint contains the path of xmllint (for example, set _xmllint=%cd%\libxml2\bin\xmllint)

Results

I did this largely because I was curious to see, after clearing away the issue of validating the proprietary Confluence markup (I will freely admit, in a risibly rudimentary fashion!), whether the non-proprietary markup (inherited from the original XHTML DTD) in my sample Confluence 4 pages was valid. As I mentioned in a previous comment on this page, I feared that, because Atlassian had not been using a schema to validate the source, Confluence 4 might be creating/allowing invalid XHTML structures (independent of any issues to do with the proprietary Confluence markup).

Of the few hundred pages I tested, I found only one such issue. One of the files contained the following invalid markup:

<ul> </ul>

That is, a <ul> without any <li> child elements.

So, as I suspected, Confluence 4 does indeed allow/create invalid XHTML, but I'm pleasantly surprised and relieved that I did not find more issues. I'm not completely relaxed about this subject, though, because I think that the majority of pages that I have tested have been migrated from Confluence 3.x without much subsequent editing in Confluence 4. At least now, though, I have a relatively automated method for detecting these issues. I can begin to validate Confluence 4 XML source.

I also forgot to mention: you can edit the resulting XML files (described in my earlier long comment) in a validating XML editor (for example, jEdit with XML plugin).

So, if you're using the Confluence Source Editor plugin, but you'd rather be using a validating XML editor, you could copy'n'paste the XML source displayed by the plugin into the wrapper I described earlier (here it is again):

<!DOCTYPE html SYSTEM "confluence.dtd">
<html xmlns:ac="atlassian.confluence" xmlns:ri="resource.identifier">
<head>
<title>Title</title>
</head>
<body>

insert XML from Source Editor plugin here

</body>
</html>

and then (with confluence.dtd in the same directory) edit the source in your favorite validating XML editor, before copying'n'pasting the contents of the <body> element back into the Confluence Source Editor. Ideal? Far from it. But at least you'll enjoy the benefits of a proper XML-aware editor (not just validation, but type ahead/autocomplete/selection lists, structure browsers etc.).

It wouldn't take too much to write some (perhaps, editor-specific) "glue" script/code to automate the getting/wrapping/unwrapping of the source, to make the process a little less cumbersome.

++ to what Shannon Greywalker, Graham Hannington, and others like Todd Day, Greg Whitfield,  Kathleen James, et al. have been saying.

I admire the civility and constructiveness of your criticism, as well as your willingness to share your technical knowledge.

I feel like you represent my position well, so have not felt the need to weigh in up to this point.

Thank you for your persistence and the amount of detail you guys have gone into.

Some summary points from discussion on these pages:

• Editing content is more common than creating new content.
• Without wiki markup, there is no EASY way to work around bugs in the RTE, which are somewhat of an inevitability for all RTEs.
• The Source Editor does not appear to be attempting to replicate the ease of use available through wiki markup as source.

Hallo everyone

Our sincere apologies for having to turn on the CAPTCHA validation on comments by logged-in users. I know it's a total pain. We've had to do it because this site has been receiving a huge number of spam comments recently, making some pages unreadable. The spam comments are obviously auto-generated, and contain long strings of text, links to various vendors sites, pictures and what have you. We investigated a number of options, including disabling the remote API in case the comments were coming in on that route. But alas, it seems that someone has written a routine that hits the UI and bombards it with comments. They use the "Sign Up" link to create a username first, and Confluence therefore sees them as legitimate logged-in users.

Our only resort is to enable CAPTCHA for logged-in users, across the entire site. We are able to identify certain groups, including the licensed community authors and Atlassian staff, who therefore do not see the CAPTCHA request. But unfortunately everyone else is affected, until we can think of another solution.

Cheers, Sarah

Dear Atlassian,

Regarding my earlier comment:

I'd be curious to use this validation technique - or see the results - on a larger set of source files that have been created, or at least updated, in Confluence 4.

Could you please email me, or make available via, say, FTP or HTTP download, a .zip of the XML storage format of as many of your public Confluence 4-sourced pages as you like? (If the .zip is bigger than, say, 10MB, then FTP or HTTP download would suit me better than an email attachment.)

If there's some automated method that I can use to scrape the XML storage format of your public pages myself, please let me know, and I'll do that instead. (I've tried pointing wget - as per the method I used with my local sandbox installation, described in that earlier comment - at your public sites, but either I'm using the wrong path, or those sites deliberately do not offer this method of access.)

I know that the simplistic DTD I've created does not fully cover the Confluence markup that you've documented (ac:image attributes spring to mind), only the markup that exists in my small set of "sandbox" files. A set of Confluence XML source files that uses a wider range of the possible markup would help me to more quickly refine the DTD. I think the resulting improved DTD might be of benefit to some users. I'm happy to send you the subsequent refined DTD (or even the current rudimentary version) if you think it might be useful as, perhaps, an attachment to CONF-24884 - Getting issue details... STATUS , as a temporary "placeholder" until you develop a (better, more accurate; definitive) DTD/schema. Let me know.

I'm working on an XSD version, too (which will probably refer to - import - the original XHTML1 Strict XSD untouched, redefining some of its groups to included proprietary Confluence markup, rather than being an edited version of that original XSD... at least, that's my current plan). Can't promise when, though.

I don't want anyone to think that I consider the html/body wrapper (presented in earlier comments) to be optimal. It's just an expedient way of getting Confluence page source to validate using a minimally customized XHTML DTD.

Probably a more appropriate wrapper - the source needs some kind of a wrapper; otherwise, it has no single root element,
making it at best a snippet, rather than an XML document - would be something like:

<!DOCTYPE ac:confluence SYSTEM "confluence.dtd">
<ac:confluence xmlns:ac="atlassian.confluence" xmlns:ri="resource.identifier" xmlns="http://www.w3.org/1999/xhtml">
insert XML from Source Editor plugin here
</ac:confluence>

which would require further customization of the DTD, to declare the newly coined ac:confluence root element and allow it to have the same dependent elements as an XHTML body element. I haven't done this yet.

Perhaps the root element name should be something like ac:page instead of ac:confluence... I tend to think that the namespace should provide the "Confluence" context, leaving the root element name to describe its contents within that context, such as a "page", but I feel that I'm just working on a stop-gap temporary solution (in the space that I hope Atlassian will soon fill), so I'm not going to spend too much time right now thinking about such choices. Same goes for the "atlassian.confluence" and "resource.identifier" namespace URIs; these are intended as temporary values only. I welcome suggestions.

One advantage of the html/body wrapper is that, if you give the file an extension that your web browser recognizes as being HTML (or XHTML), such as .html or .htm, then you can do a rudimentary preview by opening the file directly in your web browser (in which case, you might want to add a reference to your Confluence CSS). But your browser won't do much with the proprietary Confluence elements.

Perhaps a smarter approach would be to use the ac:confluence or ac:page (or whatever) root element, save as .xml, and insert a reference to an XSLT stylesheet, like this:

<?xml-stylesheet type="text/xsl" href="confluence-to-xhtml.xsl"?>

When you open the .xml file in your browser, the XSLT would transform/expand the root element into html, head, body, etc., and also transform the various ac:* and ri:* elements so that they stand out clearly among the formatted XHTML. Or you could write XSLT for some of the ac:* and ri:* elements to emulate their behavior in Confluence (some would be easy, some would be harder, and some - because, for example, they rely on information from elsewhere, such as inside Confluence - are probably best left to Confluence itself... adding such "preview" smarts is not high on my list of priorities).

Atlassian, you've so far made no comment on various references I've made to the subject of which, if any, elements and attributes should be removed from the XHTML schema when creating a schema for the Confluence storage format.

Let's look at an example.

Edit a Confluence page, and open the source in the Source Editor plugin.

Insert an XHTML definition list. For example:

<dl>
<dt>Term 1</dt>
<dd>Definition 1</dd>
<dt>Term 2</dt>
<dd>Definition 2</dd>
</dl>

Click Apply to close the Source Editor plugin and return to the RTE.

The definition list appears, and you can edit its contents, but now save the page. The definition list disappears.

Edit the page again, and open the source in the Source Editor plugin. It shows an empty definition list element, <dl/>.

This is just one example. I imagine there are probably many other similar examples.

What subset of XHTML does Confluence support?

I imagine your answer might be something like "The subset of descendants of the XHTML body element that can be created using the RTE". In which case: could you please be specific? I'd prefer not to have to guess/work this out for myself.

Atlassian, in case it occurs to you... please don't answer that "What subset...?" question with something like "Confluence supports all XHTML (body element descendants) ... if you use the HTML macro". Right now, I'm interested in the XHTML markup that can be used in the Confluence storage format without resorting to the HTML macro.

Regarding this is in my earlier comment:

When you open the .xml file in your browser, the XSLT would transform/expand the root element into html, head, body, etc., and also transform the various ac:* and ri:* elements so that they stand out clearly among the formatted XHTML.

I should have been more explicit: by "stand out clearly", what I have in mind is that such an XSLT stylesheet would transform the ac:* and ri:* elements into the same (or very similar) markup as that displayed by the Confluence rich text editor (I've used Firebug: I've seen what that markup looks like). One could go further, and have the XSLT stylesheet transform some of those elements into what you see in the Confluence preview, but I think that emulating the rich text editor display is a more pragmatic goal (more easily attainable with a higher degree of fidelity).

I can see this "out-of-Confluence editing experience" - a combination of Confluence page source wrapped in some newly coined root element (such as ac:confluence), open simultaneously in your favorite desktop XML editor and a web browser, with the editor using a schema to provide a validating, XML-aware (type ahead/context-sensitive selection lists, etc.) editing experience, and the browser using an XSLT stylesheet to transform the source into an RTE-like view - quite clearly in my mind's eye, and I know how to do it (write the schema and XSLT stylesheet). However: (a) I have a day job, so I have limited time, and (b) it's frustrating - and time-consuming - having to guess, or reverse engineer, many of the details.

Off-topic (sorry), but in case this has happened to anyone else: I've noticed that some of my recent comments have appeared as "Anonymous". I suspect - I could well be wrong - that what's going on here is that I return to this page after a few hours or days away, see that I am, apparently, logged in, add a comment, and then realize that my login must have timed out, because my comment is attributed to "Anonymous". Or maybe I'm doing something else wrong. Either way, in future, I'll try to remember to refresh the page to ensure that I really am still logged in before I add a comment. (Did I hear someone mutter "Stupid newbie?" Yeah, it's a fair cop.)

Here's a rudimentary working example of the XSLT stylesheet I mentioned in earlier comments (for viewing Confluence page source in a web browser as you edit it in your favorite XML editor):

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:ac="atlassian.confluence" xmlns:ri="resource.identifier" xmlns="http://www.w3.org/1999/xhtml">

  <!-- Transform Confluence storage format to XHTML -->

  <!-- Identity transform: by default, simply copy all attributes and nodes to output -->
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

  <!-- Replace the Confluence page root element with XHTML wrapper -->
  <xsl:template match="/*">
    <html xmlns:ac="atlassian.confluence" xmlns:ri="resource.identifier">
      <head>
        <title>Confluence page</title>
        <style type="text/css">
        /* <![CDATA[ */
          body {
            font-family: sans-serif;
            background-color: white;
            color: black;
          }
          .markup {
            color: #A9A9A9;
          }
          .element-name {
            color: #800080;
            font-weight: bold;
          }
          .comment {
            color: green;
          }
          .cdata {
            color: #CC0066;
          }
          .text {
            font-family: sans-serif;
            color: black;
          }
          .attribute-name {
            color: #800080;
          }
          .attribute-value {
           color: black;
          }
          div.extension-element {
            margin-top: 0.5em;
            margin-bottom: 0.5em;
            border: 1px solid #DDDDDD;
            background-color: #F0F0F0;
          }
          p.extension-element-markup {
            margin-top: 0.2em;
            margin-bottom: 0.2em;
            padding-left: 0.5em;
            padding-right: 0.5em;
          }
          div.extension-element-contents {
            border-top: 1px solid #DDDDDD;
            padding: 0.5em;
            background-color: #FFFFFF;
          }
          /* ]]> */
          </style>
      </head>
      <body>
        <xsl:apply-templates/>
      </body>
    </html>
  </xsl:template>

  <!-- Represent extension elements as their XML source -->
  <xsl:template match="ac:* | ri:*">
    <div class="extension-element">
      <p class="extension-element-markup">
        <span class="element-name">
         <xsl:value-of select="name(.)"/>
        </span>
        <xsl:apply-templates select="@*"/>
      </p>
      <xsl:if test="node()">
        <div class="extension-element-contents">
          <xsl:choose>
            <xsl:when test="name(.) = 'ac:plain-text-body'">
              <pre><xsl:apply-templates select="node()"/></pre>
            </xsl:when>
            <xsl:otherwise>
              <xsl:apply-templates select="node()"/>
            </xsl:otherwise>
          </xsl:choose>
        </div>
      </xsl:if>
    </div>
  </xsl:template>

  <!-- Represent extension attributes as their XML source -->
  <xsl:template match="ac:*/@* | ri:*/@*">
    <xsl:text> </xsl:text>
    <span class="attribute-name">
      <xsl:value-of select="name(.)"/>
    </span>
    <span class="markup">
      <xsl:text>="</xsl:text>
    </span>
    <span class="attribute-value">
      <xsl:value-of select="."/>
    </span>
    <span class="markup">
      <xsl:text>"</xsl:text>
    </span>
  </xsl:template>

</xsl:stylesheet>

To use this stylesheet, you need a Confluence .xml source file like this:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="ConfluenceToXHTML.xsl"?>
<!DOCTYPE ac:confluence SYSTEM "confluence.dtd" [
  <!ENTITY ndash  "&#8211;">
  <!ENTITY nbsp   "&#160;">
]>
<ac:confluence xmlns:ac="atlassian.confluence" xmlns:ri="resource.identifier" xmlns="http://www.w3.org/1999/xhtml">
<!-- Copy source from the Confluence Source Editor plugin to here -->
</ac:confluence>

where I have further customized the confluence.dtd (which is based, as mentioned in an earlier comment, on the XHTML 1.0 Strict DTD) with the following new lines (to declare the newly coined ac:confluence root element):

<!ELEMENT ac:confluence %Block;>
<!ATTLIST ac:confluence
  %attrs;
  xmlns           %URI;      #FIXED 'http://www.w3.org/1999/xhtml'
  xmlns:ac        %URI;      #IMPLIED
  xmlns:ri        %URI;      #IMPLIED
  >

With this setup, I can edit the .xml file in a validating XML editor (I happen to be using jEdit with XML plugin), and browse the .xml file in Firefox. In Firefox, due to the XSLT stylesheet, the proprietary ac:* and ri:* elements appear in similar (though not identical) fashion to their appearance in the Confluence rich text editor.

Some issues:

• Of the three web browsers I've tried (Chrome, IE9, and Firefox), only Firefox works for me. The other browsers appear to restrict local client-side XSLT transformations. I used to use this feature frequently in previous versions of IE; the Googling that I have done seems to indicate that it should still work in IE9 (since the XSLT stylesheet and the .xml file are both local files), so I'm stumped. I'm less familiar with Chrome.
• This rudimentary XSLT stylesheet uses the browser's default CSS to render most of the XHTML. It would be better if the XSLT stylesheet output link elements that referred to your Confluence CSS files (source editor CSS or preview CSS; not quite the same thing). Maybe in a later version.
• Ugly: Firefox does not load external DTDs. For the client-side XSLT stylesheet processing to work, you must include in the DTD subset of the .xml file entity declarations for any entity references that you use, such as &nbsp; or &ndash; (hence those <!ENTITY ndash "&#8211;"> etc. declarations in the example listing). Yes, a pain.

At some point, if I get time (and especially, if anyone expresses interest), I might consolidate the contents of several of my recent comments in a page on some external website, with downloadable files (DTD, XSLT stylesheet, various related scripts). If anyone has followed what I've done so far, it should be fairly easy to reproduce, but I guess it would be easier if I just made the complete, latest files downloadable. But what I'm really hoping is that Atlassian will beat me to it, and render this work obsolete.

Use of this XSLT stylesheet is not restricted to an xml-stylesheet processing instruction in web browsers. There are many XML applications (XML editors with XSLT capability, as well as standalone XSLT processors) that you could use with this XSLT stylesheet to transform Confluence source into XHTML (essentially, to highlight proprietary Confluence markup, and leave the rest as-is) for viewing in a browser.

Getting back to the schema, I can fully understand why Atlassian was reluctant to introduce a source editor. It's not just a matter of what subset of XHTML elements (and attributes) Confluence supports, but what combinations of XHTML elements - combinations that are valid in XHTML - Confluence supports, and to what extent, and in what context.

For example, you can use the Source Editor plugin to enter the following source:

<p>Paragraph above bullet list.</p>
<ul>
  <li>Bullet 1 of 3</li>
  <li>Bullet 2 of 3
    <p>Paragraph 1 of 3 inside bullet 2</p>
    <p>Paragraph 2 of 3 inside bullet 2</p>
    <p>Paragraph 3 of 3 inside bullet 2</p></li>
  <li>Bullet 3 of 3</li>
</ul>
<p>Paragraph below bullet list.</p>

Back in the rich text editor, you can edit the text of the paragraphs that are nested inside the bullet list items, and those paragraphs will still be there when you save the page. But can you create new nested paragraphs? I can't see how. (I believe the current answer is to use line breaks inside list items.)

So, one cannot simply say, without qualification, that Confluence supports XHTML <p> and <li> elements (or use the corresponding XHTML schema definitions unchanged). The Confluence rich text editor does not (appear to; I would be happy to be shown otherwise) allow you to create these elements in some of the combinations that are valid according to XHTML.

Such different degrees of support complicate the task of developing a schema. Sorry for mentioning it again, but such issues would not exist (or at least, there would have been a much better chance of identifying and minimizing such issues) if a schema had been used as a primary reference during design, development, and testing (and was intrinsic to the editor).

This comment describes a request for a new feature (or, if you prefer, a request for a change to the behavior of an existing feature).

I have deliberately added this request to this page as a comment. I will leave it to you (Atlassian) to decide whether it is worthy of creating a JIRA issue for development.

Everyone/anyone: please feel free to comment on my suggestions for the names of various items in this request.

Feature request: Serve page source via WebDAV as an XML document, not an XML snippet (and .xml, not .txt file extension)

Currently, the Confluence WebDAV plugin serves the source of a page as an XML snippet - that is, a collection of XML elements, without a single root element - in a file with extension .txt.

Being a snippet prevents the file from being treated as an XML document that can be validated or edited as an XML document in a validating XML editor.

When serving the page source via WebDAV, could you please wrap the XML snippet in a root element, with a document type declaration, and also (optional, but preferable) an XML declaration, with the file extension .xml (for more convenient association with XML applications)?

For example, instead of serving the file mypage.txt, like this:

<p>Paragraph text.</p>
<ul>
<li>Bullet list item</li>
<li>Bullet list item</li>
</ul>

serve mypage.xml, like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ac:confluence PUBLIC "-//Atlassian//Confluence 4 Page//EN" "http://www.atlassian.com/schema/confluence/4/confluence.dtd">
<ac:confluence xmlns:ac="http://www.atlassian.com/schema/confluence/4/ac/" xmlns:ri="http://www.atlassian.com/schema/confluence/4/ri/" xmlns="http://www.atlassian.com/schema/confluence/4/">
<p>Paragraph text.</p>
<ul>
<li>Bullet list item</li>
<li>Bullet list item</li>
</ul>
</ac:confluence>

The example listing above contains my initial suggestions for various items, as described in the remainder of this comment.

XML declaration

Suggestion:

<?xml version="1.0" encoding="UTF-8"?>

Issues:

• From the W3C Recommendation for XML: "XML documents SHOULD begin with an XML declaration which specifies the version of XML being used."
• Are the source files served via WebDAV UTF-8-encoded? I think they are. If not, replace UTF-8 in the XML declaration with the appropriate encoding.

Document type declaration

Suggestion:

<!DOCTYPE ac:confluence PUBLIC "-//Atlassian//Confluence 4 Page//EN" "http://www.atlassian.com/schema/confluence/4/confluence.dtd">

Issues:

• The formal public identifier (FPI) is up to you. My suggestion is "-//Atlassian//Confluence 4 Page//EN", but you might want to think about using a more specific version number, or a version number for the schema that is independent of the Confluence product version. And you might not like "Page" (for example, if pages and blog posts will always have the same schema, you might prefer to use a different term that encompasses both; although, I think that "page" is generic enough for this purpose, and one could argue that a blog post is a "specialization" of a page).
• The system identifier - my suggestion is "http://www.atlassian.com/schema/confluence/4/confluence.dtd" - gives the address (URI) of a DTD for the document, but, in practice, the DTD does not necessarily have to exist at that address (applications might not always have access to network-based resources). In practice, one can use a catalog that maps the FPI to a local copy of the DTD (which is what I have been doing in testing).

Document root element name

Suggestion:

ac:confluence

Issues:

• If you plan to have multiple XML vocabularies for Confluence - perhaps blog posts might not have quite the same vocabulary? - then perhaps ac:page might be a better choice of name for this root element (more specific).
• I have used the namespace prefix ac: (your coinage: I assume, perhaps incorrectly, that this is an abbreviation of "Atlassian Confluence") for the root element. I did this to (a) avoid coining another namespace, and (b) avoid including the root element in the default namespace (that is, elements without a namespace prefix), because you have so far used that namespace for elements and attributes that use the same names as XHTML elements and attributes.

Namespace declarations

Suggestions:

xmlns:ac="http://www.atlassian.com/schema/confluence/4/ac/"
xmlns:ri="http://www.atlassian.com/schema/confluence/4/ri/"
xmlns="http://www.atlassian.com/schema/confluence/4/"

Issues:

• For the default namespace, I have quite deliberately not used the XHTML namespace name "http://www.w3.org/1999/xhtml". It is apparent that elements and attributes in the Confluence storage format that have the same names as elements and attributes in XHTML do not necessarily share the same characteristics (for example, the contexts in which these elements can appear). To use the XHTML namespace for these elements and attributes would be, at best, misleading.
• A namespace name must match the format of a URI, but there need not be an actual resource at that URI. I have coined URIs based on my observations of current common practice, but the final decision is, of course, yours. (Note that I have deliberately terminated the URIs with slashes.)
• I'm least confident about my suggestion for the URI of the default namespace; specifically, the way it ends in just ".../4/" (the Confluence version), compared to the other two namespace URIs, ".../4/ac/" and ".../4/ri/". I could not think of an appropriate choice of term to describe this default namespace. ".../4/xhtml/" felt wrong to me, for reasons already described.

I'm going to guess that the reason they can't/won't provide a DTD or XSD is closely tied to the reason they deliver WebDAV output as a .txt file and you are seeing no root element or DOCTYPE declaration. And my guess is that they're using XPATH to let their code get at everything they care about in their "implementation details". So essentially, they're approaching XML like non-doc-oriented coders do. Like programmers. They have not approached their XML implementation like doc people do. Just a guess. This is Shannon btw.

I took the liberty of throwing together a very simple plugin that lists all of the macros installed in a given Confluence instance, along with the supported parameters (and the data types of said parameters). The hope is that this will be useful to those who have been wondering about macro parameter documentation. Here's the JAR:

Arsenale Macro Lister JAR

To use, install the plugin via the UPM, make sure that you're logged in as an admin, and then browse to YOURCONFLUENCE.COM/admin/macrolister.action

Caveats:

• Tested with Confluence 4.2 and 4.1. I'm guessing that it will probably also work with 4.0, but YMMV.
• This plugin only displays macro parameters that are exposed to the Macro Browser. The output looks pretty comprehensive at first glance, but it's possible that some macros have hidden parameters that are not exposed.
• No support, no warranty. This was an hour out of my morning and there are no plans to develop it further.
• The source is available at Bitbucket. Feel free to fork it and do whatever you like.

If one in the community were so inclined, it probably wouldn't take much more effort to make the plugin output the data in a format that is more amenable to XML-processing applications.

For example, I can envision a plugin that would automatically output an entire DTD (or at least a DTD-like object, pending resolution of the "no root element" question) that corresponds to all of the entities that are available for a specific Confluence instance, including macros, and presumably integrating the groundwork that Graham has already laid.

However, I'm not that much of an XML-head, so the latter is a task for somebody else.

Some errata (omissions in the documentation in this page), based on my latest experimentation in the source editor and validation of the source in a Confluence 4 "sandbox":

• The ac:name attribute of the ac:emoticon element can also have various other values not listed in this page (such as question and minus, among others; this is obvious if you look at all of the available options in the rich text editor)
• The a element can have class and rel attributes. For example:
  
  <a class="external-link" href="http://localhost:9999/help/index.jsp" rel="nofollow">
  
  
• The ul element can have a class attribute. For example:
  
  <ul class="alternate">

I cannot reproduce the last two items (for the a and ul elements) using the Confluence 4 rich text editor, so perhaps these attributes are artefacts of the Confluence 3-to-4 conversion (the pages in which these attributes occur were created in Confluence 3.x). Either way, I'm going to allow these attributes on these elements in the DTD, so I might need to expand my definition of what I'm covering in the DTD to include not just what can be created in the Confluence 4 rich text editor, but also what can exist in source converted from Confluence 3. Atlassian, any comment on this? (When I open these pages in the Confluence 4 rich text editor, save the page, and then edit again and inspect the source, these attributes remain.)

A draft DTD for the Confluence storage format

I have made available for download a draft DTD for the Confluence storage format in a .zip file that also includes three .ent files referred to by the DTD (as used by the XHTML 1.0 Strict DTD).

This is an ad hoc, temporary location for these files; not intended to be a permanent home. I hope that Atlassian will soon render these files obsolete by providing a definitive schema (here, I'm using "schema" in the generic sense that encompasses both DTDs and W3C XML Schema - XSD - documents).

To use this DTD, extract the contents of the .zip file to a folder where you have a .xml file containing Confluence page source in the following "wrapper":

<!DOCTYPE ac:confluence SYSTEM "confluence.dtd">
<ac:confluence xmlns:ac="http://www.atlassian.com/schema/confluence/4/ac/" xmlns:ri="http://www.atlassian.com/schema/confluence/4/ri/" xmlns="http://www.atlassian.com/schema/confluence/4/">
<!-- Copy source (for example, from the Source Editor plugin) here -->
</ac:confluence>

then open the .xml file in a validating XML editor (for example, Altova XMLSpy).

In case it needs to be said: this DTD is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.

Regarding the .ent files in the .zip: I do not know whether the Confluence storage format supports all XHTML character entities (such as &nbsp; ). I know that the Confluence rich text editor "Insert Symbol" option does not display all of the symbols available in XHTML. For now, this DTD uses the XHTML .ent files as is.

I have had to make guesses about some markup structures. It would not surprise me to find out that some of those guesses were wrong.

This DTD allows/describes (to the best of my ability and understanding) markup consisting of the superset of the following intersecting sets:

• Markup that the Confluence 4 rich text editor can generate (more specifically, the markup that I have been able to make the rich text editor generate)
• Markup described in the documentation in this page
• Markup in the page source of my local "sandbox" installation of Confluence 4 (where most of that page source has been migrated from Confluence 3.x, and has not yet been edited in Confluence 4)

This DTD is a stepping-stone to an XSD, which can then be annotated and transformed into XHTML-source documentation for the Confluence storage format.

It's a start.

Feedback welcome.

Atlassian, so far you have not responded to my request (made in an earlier comment) for a .zip of a set of Confluence XML page source (from your public pages). That would have made the job of developing the DTD to this stage easier, and would have improved my confidence in its fidelity. (As I've mentioned, my "test bucket" of page source consists of a relatively small set migrated from Confluence 3.x, mostly not yet updated in Confluence 4, and brand new pages that I have created in Confluence 4 specifically to "exercise" the full set of markup that the Confluence 4 rich text editor can generate.)

I see no (public) progress on CONF-24884 - Getting issue details... STATUS .

Could you please either (preferably, both):

• Send me (or otherwise make available for download) a nice fat .zip of Confluence 4 page source that I can use to confirm and/or correct the guesses that I have so far had to make (regarding, for example, the child elements that each element can contain).
• Download the DTD that I have developed, use it yourself, and send me your feedback.

While developing the DTD, I noticed that, of the elements in the Confluence storage format that have the same name as XHTML elements, all except two exist in XHTML 1.0 Strict. The elements s (strikethrough) and u (underline) are "interlopers" from XHTML 1.0 Transitional. I make this comment to forestall anyone claiming that elements in the Confluence storage format that have the same name as XHTML elements are all from XHTML 1.0 Strict. (Who would do that? Er, perhaps the same people who said that the Confluence storage format is XHTML?  )

Some source I've just generated using the rich text editor:

<ol>
  <li style="list-style-type: none;">
    <ol>
      <li>Numbered list item (indented once)</li>
    </ol>
  </li>
</ol>
<h5 style="text-align: center;">Heading 5 (align center)</h5>
<h4 style="margin-left: 30.0px;">Heading 4 (indented)</h4>

So, li and heading elements can have style attributes, which is not shown in the documentation in this page. (In case someone reads this and goes "Duh! Of course they can!": I know this is possible in XHTML, but I'm talking about the Confluence storage format; and, in particular, what markup the Confluence rich text editor can generate.)

In case you're wondering, I created that li with style="list-style-type: none;" using the indent toolbar button.

Following on from my previous questions on this subject: I don't know if you (Atlassian) plan to add this level of detail (which attributes are allowed on each element) to the comprehensive (sic) documentation on this page, but I'll be adding this discovery to the DTD.

You haven't yet answered my specific questions regarding exactly what you mean by "comprehensive". I've already noted that it does not include complete information about which elements can be contained inside other elements, but I've so far assumed that it does include which attributes can be used on each element (except for inadvertent omissions, which you've added as I've pointed them out), so I've been pressing ahead with the DTD on that basis. However, it seems now that the documentation in this page includes only some examples of which elements can have particular attributes (such as style); this incompleteness might be inadvertent, I don't know.

Atlassian, please add the ac:queryparams attribute of the ac:image element to the - I'm not going to use the "c" word - documentation on this page.

For example:

<ac:image ac:queryparams="effects=drop-shadow">

I've updated the draft DTD .zip:

• Updated: DTD (refined after further experimentation to see what markup the rich text editor can create)
• New: example Confluence page source file (containing a variety of markup; open this in your favorite validating XML editor), with a variant that refers to an XSLT stylesheet (below)
• New: "Confluence to XHTML" XSLT stylesheet (confluence2xhtml.xsl)

Tip: Try opening confluence-page-example-with-xslt.xml in Firefox.

I'll add a readme file soon.

Atlassian,

Why do you prefix the attribute names in your ac:- and ri:-prefixed elements?

From the W3C Recommendation "Namespaces in XML 1.0":

Default namespace declarations do not apply directly to attribute names; the interpretation of unprefixed attributes is determined by the element on which they appear.

Progress over the weekend: I now have a set of W3C XML Schema 1.0 (.xsd) documents that are (intended to be; I'm still doing some testing) functionally equivalent to the DTD I recently made available. (Except, that is, for character entities - referenced by, for example, &nbsp; - which you can't define in an XSD.) I'm also playing with some XSD-to-XHTML transformation options (for producing schema documentation); I want to do this myself, and get some feedback (and then apply corrections), before perhaps bugging the owner of www.schemacentral.com to host the documentation (as mentioned in an earlier comment).

I've replaced confluence-dtd.zip with confluence-schema.zip.

Changes:

• Added a readme
• Refined DTD after testing against Atlassian's own Confluence Documentation source
• Added XSD (constraints match DTD, as far as possible)
• Added catalog and related sample files

For details, see the readme.html file in the .zip.

This is a work in progress. I am awaiting answers from Atlassian to several questions that might affect the DTD/XSDs.

Feedback welcome.

Developing those DTD/XSDs burned some midnight oil, so I hope that readers of this page will briefly indulge me here...

What's in a name? The term "Confluence storage format" works in the sense that "it is what it says on the tin" (if you allow that there's an implicit "page/blog post" qualifier in there somewhere), but, having used this term for a while now, I feel that it's clumsy and too long. (Compare "Confluence storage format" with, for example, "DocBook" and "DITA"). I can live with this existing term, but if I'm going to be spending more time with the language (it seems likely), I'd prefer a shorter, sweeter-sounding name. (I realize "sweeter" is a subjective judgement.)

The first name that occurred to me was "Confluent": that is, to appropriate the adjective "confluent" as a proper noun. Then I did a trademark search: I found no "live" US trademarks for just the one-word "Confluent", and certainly not in this market space, but there are several (mostly medical) companies that use the word "Confluent" in their names. So, maybe it's a contender. One point in its favor is that you could just refer to "Confluent", or, if the need arose, firmly place it in context by appending the word "XML" - so, "Confluent XML" (getting longer, though) - returning "confluent" to its roots, albeit with a capital C, as an adjective.

Then I saw that "conflux" was a synonym for "confluence", and it occurred to me that its trailing letter x had potential. "Confluxml" (lowercase "ml")? "ConfluxML" (uppercase "ML")? Currently, I think that these are my favorites; I can't quite decide between the two, although I somehow find it appealing that "ConfluXML" begins with the same letters as "Confluence", and the uppercase "XML" provides a clear break. (Pronounced - regardless of whether it's spelled "...ml" or "..ML" - "conflux em el".)

Google "conflux":

Your search - "confluxml" - did not match any documents

Looks like a winner to me!

Then again... maybe something more prosaic, such as "Confluence page XML" (where "page" is used here in a generic sense intended to encompass "blog post").

Some candidate terms (including the "incumbent"):

• Confluent
• Confluxml
• ConfluXML
• Confluence XML
• Confluence page XML
• Confluence source (another sort-of incumbent, but I would argue that this term is too ambiguous, unless it's used strictly in context: I think of "Confluence" as software, and "Confluence source" as the source code for that software)
• Confluence storage format

Opinions? ("Why bother?") Other suggestions? (I'm half-expecting a collective yawn  ) I certainly don't want to spend too much time on this.

Finally, I'm not looking for a term that I would expect general Confluence users to see, or even be aware of. For example, I'm not about to propose that Atlassian renames the "Confluence Source Editor plugin" to, say, "ConfluXML Editor plugin". I'm really just after something pithy that I can use in comments on pages such as this, and in the DTD/XSD documentation (the readme.html), instead of the rather unwieldy "Confluence storage format".

A nit in the readme.html of that XSD/DTD .zip: I realize I haven't quite got the XSD aspects of the catalog-related tips quite right. Specifically, I can't get any XML editor to use the XSD for validation unless the document contains a schemaLocation attribute. I'm clearly missing something. The XML editors (specifically, jEdit and Altova XMLSpy 2012) use the same catalog.xml to map a DOCTYPE FPI to the DTD file just fine. And, as described, I can specify the xmllint --schema command-line option to use the XSD for validation, whether or not a document contains a schemaLocation attribute.  Any ideas?

I have just sent the following email to the XML-DEV mailing list:

To:      xml-dev@lists.xml.org,
Date:    20/04/2012 04:30 PM
Subject: Request for review: DTD/XSDs for the Confluence Atlassian 4 storage format

From my earlier email:

> [I am] hoping to learn more. And I am prepared to publicly expose my ignorance to do so

On that note...

I am a user of Atlassian Confluence, a commercial wiki product. I have no other affiliation with Atlassian, (Okay, I'm Australian, too.)

Previous (3.x) versions of Confluence stored content in a wiki markup format. The current version of Confluence (4) stores content in a proprietary XML vocabulary that consists of:

- Elements and attributes that have the same names as a subset of elements and attributes in XHTML 1.0 (mostly from XHTML 1.0 Strict, with a few additional elements from XHTML 1.0 Transitional). Elements in the Confluence XML vocabulary that have the same name as elements in XHTML 1.0 have a much more limited set of attributes than the XHTML 1.0 elements of the same name.

- Proprietary elements and attributes

Some Confluence users have asked Atlassian to provide a schema (DTD, XSD, or both) for this XML vocabulary. However, Atlassian has indicated that providing a DTD/XSD is not high on its list of current priorities.

So, I have attempted to create the DTD/XSDs myself, based on a combination of:

- The XHTML 1.0 Strict DTD (and a few snippets of Transitional)
- The documentation that Atlassian has provided (which falls far short of the detail required to develop a DTD/XSD)

- Experimentation with the Confluence rich text editor (to exercise the full range of markup that the editor can generate)

- "Forensic analysis" of the XML source of Atlassian's own Confluence documentation (Atlassian publishes "exports" of their public documentation, containing the XML source of over 4000 pages; so, a reasonable-size test bucket) - xmllint was very useful here for automating validation, and iteratively refining the constraints of the XSD/DTDs

I hope I'm not breaking any mailing list guidelines here (I don't think so; I've just re-read them)...

I welcome feedback (constructive criticism) of these DTD/XSDs.

I'm not offering any money, so feel free to have fun at my expense by flaming these files, and my questionable XML skills, to a cinder. Perhaps a phoenix will rise from the ashes ;-).

You can download the package (a 30 KB .zip file) from:

http://www.amnet.net.au/~ghannington/confluence/confluence-schema.zip

For details, see the readme.html in the .zip.

I am aware that the tips in the readme on "Using a catalog, rather than explicitly referring to the DTD/XSD in each document instance" are, at best, incomplete... I'll fix this in the next couple of days, with thanks to Liam for his recent advice (on this mailing list).

I hope that the above address will be a temporary home for this package; I hope that Atlassian will soon either render the package obsolete by providing their own - although I think that's unlikely, given its stated priorities - or take ownership of this package.

Graham Hannington
Perth, Western Australia

Hallo all

We're planning to document the markup for all macros shipped with Confluence. For each macro, the documentation will define the macro name, parameter names, accepted parameter values, default value, and whether the parameter is required or not. We'll document both the wiki markup and the xHTML-based storage format for each macro.

Here's the tracking issue: CONF-24972 - Getting issue details... STATUS

I've started the document design and prototyping. We're giving this task high priority, although there are other high priority tasks too.

Cheers, Sarah

New: Confluence XML schema documentation (generated from the XSD).

Atlassian,

Now that you can view the schema in a human-readable format (see the link in my previous comment), I would really appreciate it if you could answer my remaining questions (in my earlier comment, based on analysis of the 4000+ pages in your Confluence Documentation export), regarding, for example, style and class attributes on various elements, and the presence of div elements. I'd like to get these issues cleared up before I ask the owner of Schema Central to host the documentation. These issues with the formal definition of the storage format (in this case, the XSD) are easier to see when you can view the formal definition in a human-readable format.

For example, despite the presence of the class attribute on various elements in your source (see my earlier comment), I think that the only way to get the rich text editor to create a class attribute (please correct me if I'm wrong: all I have to go on is my own observations) is to highlight table contents (class="highlight" on td and th; I've not seen it on a tr). Currently, as you can see in the schema documentation, the XSD allows the class attribute on the following elements (based on earlier analysis of my local "sandbox" source): a, br, td, th, ul. I'm now wondering whether, despite the presence of the class attribute on various elements in actual source, I should restrict the DTD/XSDs to allowing the class attribute on only the td and th elements (that can have class="highlight" applied by the rich text editor).

(Why I'm bothering with this level of detail - and repeatedly asking for clarification from Atlassian, when it has clearly stated that its current priorities lie elsewhere - is another matter entirely. Something like, I guess: since I've started, I might as well do it right, and I figure it doesn't cost Atlassian much time to answer these questions.)

I have developed a rudimentary working XSLT stylesheet that transforms Confluence XML page source (wrapped in a document root element) into Confluence 3.x wiki markup.

The stylesheet handles headings, paragraphs, lists (including nested lists), text effects (for example, it converts decimal RGB color values into a hex string for the color macro), text breaks, tables, macros (still finessing/testing this, but it works for the several types of macro I've tested, including, for example, code and section/columns), images, emoticons, and some types of links (still working on this).

Looks promising so far (I only started developing it this afternoon), at least for my use case, which is to copy relatively simple content from Confluence into JIRA. (Although I'm also testing it by pasting into the Confluence 3.x wiki markup editor.)

More news soon.

Hallo to all the contributors to this page

I'm planning to "move" this page into the main body of Confluence documentation. It won't go far – it will still be in this space on this wiki. At the moment, it's buried under Confluence 4.0 Editor - Customer Feedback, which is in turn a child of Planning for Confluence 4.0. In the next few days, I'll move it into the Confluence User's Guide, under a new section called "Advanced Uses of Confluence".

So, why did I put quotes around the word "move" in the previous paragraph? Well, the aim is to move the information about the storage format into the user's guide. But we want to retain all the comments on this page, and at the same time make sure that people reading the core information about the storage format can load the page without the long tail of comments.

To make that happen, I'll rename this page to "Feedback on Storage Format". The new page in the user's guide will be named "Confluence Storage Format". It will have the same content as this page (by dint of content reuse) but will start out clean of comments. The comments are great, but we'd like the page to focus documenting the current solution. I'll add a note on the new page, directing people to this page for discussion and new ideas.

Could you please update this page with the new-for-4.2 use of <div> elements for visual page layouts? (And any other new markup not yet documented on this page?)

The new, proprietary data-atlassian-layout attribute on div elements

Based on this request on the new Confluence Storage Format page:

If you have feedback on the Confluence markup ... please add your suggestion to the page titled Feedback on Confluence Storage Format.

I think/hope the following comment belongs here...

I've just been playing with the new-for-4.2 "Page Layout" feature, and looking at the corresponding XML source generated by the rich text editor.

For example:

<div class="contentLayout" data-atlassian-layout="{&quot;name&quot;:&quot;pagelayout-three&quot;,&quot;columns&quot;:[&quot;&quot;,&quot;&quot;,&quot;&quot;],&quot;header&quot;:true,&quot;footer&quot;:true}">
  <div class="header">
    <div class="innerCell">
      <p>Header</p>
    </div>
  </div>
  <div class="columnLayout threeColumns">
    <div class="cell ">
      <div class="innerCell">
        <p>Column 1 of 3</p>
      </div>
    </div>
    <div class="cell ">
      <div class="innerCell">
        <p>Column 2 of 3</p>
      </div>
    </div>
    <div class="cell ">
      <div class="innerCell">
        <p>Column 3 of 3</p>
      </div>
    </div>
  </div>
  <div class="footer">
    <div class="innerCell">
      <p>Footer</p>
    </div>
  </div>
</div>

I'm curious about the new, proprietary data-atlassian-layout attribute (on the highest-level div element in the example above).

I think that this is the first time that I have seen, in the Confluence storage format, a proprietary Confluence (or, if you prefer, Atlassian) attribute on an element that has the same name as an XHTML element.

Paul Curren (Atlassian) has previously made the following statement:

we are developing to the full XHTML spec

I realize that the letter of Paul's statement does not preclude adding proprietary attributes to elements that are from the XHTML spec. However, I felt it was worth at least mentioning this attribute, based on a combination of the following two factors:

• The "novelty" of this data-atlassian-layout attribute (that is, it's the first time I've seen in the Confluence storage format a proprietary attribute on an element with the same name as an XHTML element)
• My - I now realize, probably incorrect - interpretation of the "vibe" of Paul's statement (here, I'm using the word "vibe" in the same sense in which it is used in the Australian movie The Castle)
  

I thought, in a hazy, vibe-y kind of way, that, as per all previous cases (of which I am aware), any new uses in the Confluence storage format of elements that have the same name as XHTML elements would follow the rules of the XHTML spec (or at least, in the case of attributes, a "subset" of the full XHTML spec). Specifically, in this case, elements with the same name as XHTML elements would not contain proprietary attributes. Clearly, I was wrong. But I do wonder, since, as I've mentioned, it's the first time I've seen this: are you sure you want to do this? "Yes" is a perfectly acceptable answer. I don't expect to be privy to your design discussions.

I'm about to update the DTD/XSDs to allow the div element, and to allow this new, proprietary data-atlassian-layout attribute on div elements. (I note that, since you have not namespace-prefixed the data-atlassian-layout attribute, it belongs to the same namespace as the div element. Which is fine, not a problem at all, as far as the XSDs I've developed are concerned, because they already coin a new namespace for elements with the same name as XHTML elements. However, in case it ever occurs to you, this new attribute is one more reason - there are already many others - not to claim that such elements and attributes belong to the XHTML namespace "http://www.w3.org/1999/xhtml".)

5 stars for the burgeoning Confluence Storage Format for Macros page. This is exactly the level of detail we need. Thanks, Sarah!

My apologies if this has already been covered somewhere in the 40,000+ words above.

I have thousands of pages of documentation in FrameMaker format that we want to migrate to either Confluence 4.2 or MindTouch Core. (The WebWorks tool is OK for publishing static pages but they're not editable.)

I've set up a conversion process with MIF2Go that generates XHTML I can paste into Confluence's source editor on a page by page basis. This is better than nothing, but it's a lot of work to re-create the internal cross-reference links.

I'd like to expand that into something that could convert a whole FrameMaker project into something I can import into Confluence, but I can't find documentation of such a format. What Confluence's XML Export generates is not self-explanatory.

Is there a documented or human-readable format I can use for this?

I'm still working on converting from FrameMaker to Confluence 4.2 with the source showing up as proper storage format XHTML rather than some mashup including wiki markup and macros.

I've got conversion working using MIF2Go to the point that each FrameMaker topic (topic breaks before H1, H2, and H3) is exported to an XHTML file that conforms to the Confluence 4.x storage format.

I can paste that page by page into the Confluence 4.2 source editor, but with thousands of pages of documentation, that's not a solution, plus all the internal links need to be recreated.

I can't figure out how to do a bulk import. Everything I've found that does that produces wiki markup. Is the Universal Wiki Converter being updated to produce 4.x storage format?

This sounds intriguing:

Wednesday 4:45 pm - 5:30 pm

Extending XML with SHORTREFs specified in RELAX NG

Mario Blažević, Stilo International

When SGML was replaced by its simplified successor XML, nobody regretted the omission of SHORTREFs. Or did they? Non-XML syntaxes stubbornly persist in programming languages, schema languages, and most visibly of all in wikis. We present a novel method for specifying a concrete syntax that combines the notational convenience of non-XML markup with the structure, searchability, and tool-chain support of XML, allowing authors to create valid XML without entering any XML tags. Using an extension of Relax NG to specify a concrete syntax, a parser can read a well-formed XML document conforming to the given concrete syntax specification. The output of the parser is another XML document conforming to the abstract syntax described by the base Relax NG schema.

(Balisage 2012 Conference)

A syntax with the convenience of wiki markup, but without XML round-tripping problems, because you're not really round-tripping? Sounds like a refreshingly technically rigorous new take on an old chestnut.

I'd like to understand the constraints of such a "concrete syntax". If Confluence 3 wiki markup does not pass muster as is, perhaps a recognizable variant might? Would that change your mind about a wiki markup editor in Confluence?

Simple dumb question: how do I get the text in the storage format pop-up window to wrap? TIA.

Have a question: How can I get rid of anchor underlines in the macro for linking to other confluence pages. 

Please Can you make a DTD or an XSD schema for validating storage format available?

user-b12f0 Check this user created "XML schema for the Confluence storage format".