Specification - Confluence Advanced Editor

Bravissimo. What's the timeline?

Kudos on realizing this... Finally!..

Just wanted to add a few comments because I think you are seeing certain things a bit wrong/from the negative point of view.

 

Editor Niggles

User can access source view from inside the Confluence editor.

1: Must have

This is a short-term solution to working around around minor bugs. Long term we should be solving these problems. Need to determine whether we use editor HTML source or the storage format source.

 

You write this is short term, but I would say on the contrary, this is very long term in my opinion (which is a good thing). What you are focusing at here is "Bug A", and for that it is true that the a "Advanced Editor" is a short term workaround to solve "Bug A". BUT it is a long term solution to solve "Bugs" (plural). Because face it, your RTE will never be perfect, there will always be small "Niggles" as you name it. This editor is a long term solution to work around any future found "Editor Niggles".

To the question of weather to use "HTML" or "Confluence Markup". I would say use the raw storage format, I think from the status macro here that is easier to deal with. But whatever you do don't add a third conversion!.
 

 

Copy paste source

User can copy / paste source in full fidelity. Users should be able to: Copy the entire content of a page Paste into a separate editor Manipulate it and Paste it back into the source view of the editor Save the page

1: Must have

 

Will a XSD be available here? Personally I don't use this to much, and not at all now, what I miss is the auto generate capabilities but never-mind that because that will be solved as well, however an XSD could make on the fly validation for persons having an editor supporting it (E.g. Visual Studio), and since an XSD would be a way to validate your input otherwise (you should already have one!!!!!!) then I can't see a public released XSD should be hard to come by?.

 

 

Validation

The user should not be able to save any content that will break the page. The editor should do validation to make sure only supported markup is being sumbitted.

1: Must have

Might not be able to guarantee this. Macro markup is quite specific, and I'd image this could be broken fairly easily. Trying to stop this would make this a much bigger solution. I think that reverting an edit via the history is an acceptable trade-off. I'm not expecting any kind of server round-trip for validation until saving in the main editor.

 

I am not sure how the invocation of macros are, but would it be possibly to just catch any errors from them (on render time) and if an error happens display it as a "red cross" or maybe a "red error box" where some error text could be displayed from the error thrown?... That would be more than enough. (Much like the old confluence would actually display errors for macros in the page).

After all, having to accommodate for "macros" here would make you unable to use XSD's to do the validation as that would be a different XSD pr. installation depending on what plugins it had installed.

 

Syntax highlighting

The markup should be formatted and indented so that the source is as easy as possible to visually parse.

2: Should have

As discussed if this is a quick win from reusing some code formatting from speakeasy, we can do this. Not sure if it works with HTML, or only well-formed XML.

 

If you do this, make sure it is 110% rock solid, I know this is far more simple that a RTE, but in worst case ending up with another bug prone editor won't help us a bit. We can always copy things out of confluence to begin with if we wan't this type of editing experience, focus on the important pieces first. (The indentation should not be an issue to provide, that should be possible though all standard XML libraries I hope, at least it is for the big ones).

 

Group Permissions

Admin can specify which users / groups can have access to edit source

3: Nice to have

This is necessary so that only people who REALLY need access can get access.

 

This is not needed at all I would say, if people are scared of an XML format, they will stay away. And if the don't reverting pages is luckily really easy!. I would rather have the editor "snapshot" the page on switching to source so you could revert within edit mode (Higher level undo). But that is also nice to have, but I would say more usefull.

I mean even with your "Validation" as a Must have, I would say that restricting this feature is pointless, people can't really break pages by accident, they can break macros, but then I would much rather that you put effort into telling people just how easy it is to revert when you made such a mistake. And since it is confined to macros these would also break in the RTE view?... making them maybe rather easy to fix up before submitting the final version of a page.

My strong preferences regarding specific user stories listed above:

Editor Niggles

You must enable us to edit the storage format. Editing the HTML used to render the RTE in a browser is essentially useless. How can you even consider the HTML? What's going to be round-tripped to other environments by people who need/choose to do so is the storage format. What you're documenting over in the other new page is the storage format. The people who need access to "fix bugs in the RTE" need to understand and apply and work directly in the storage format. There simply is no choice here.

Also I agree whole-heartedly with Jens' comment above about your framing/understanding of "Editor niggles". Your RTE will never be perfect. Never. Not once in the history of computing so far has any attempt at an RTE ever been flawless, and as I've mentioned before in the feedback thread, any product that lasted for any decent length of time in the marketplace has always needed to provide a way to get at and directly edit the underlying markup.

Edit macro parameters

Same comment as for editor niggles. If I'm tweaking a macro, I need to do it in the raw storage format with the real underlying XML content model and attributes. Editor HTML is not a viable option at all. Don't even think for a moment that trying to fix things through the Editor HTML is an option.

Validation

So what if it's harder for you guys to build a validating parser on check-in with error messages that spell out exactly what's invalid? If you're making us work in XML-land, you need to provide us with at least that minimal aspect of a decent XML editor. The folks over at Arsenale seemed to quickly implement a rudimentary validating parser with syntax formatting/highlighting and validation error messages with line numbers, so why can't you? If you need a shortcut, go buy their IP and integrate it with your solution.

To be clear, simply not saving the page and throwing up a generic error message that there was some invalid content is NOT sufficient. You must tell us what element on what line is invalid, and why. How else will we be able to zero in on what needs fixed? This is SMGL/XML 101 here, folks. You forced this on us, so do the right thing.

To be crystal clear, the general workflow should be:

1. Open page in "storage format editing" mode.
2. Edit the storage format (XML) directly.
3. Click Save or Preview (ideally you'll provide a preview option even for the storage format editor).
4. Before the editor actually checks in the changes, it runs validation. If errors occur, the state of the storage format "draft" is saved and a set of validation errors are displayed.
5. If the user resumes editing, no changes in the draft were lost and they can try to fix the errors reported by the validation report.
6. Iterate steps 3-5 until the user cancels out or the draft is successfully validated and stored in the system as a new version of the page.

Syntax highlighting

Again with the "HTML" option. No, this is not an option. We do not want to edit the HTML used to render the RTE editor. We want to edit the storage format directly. Really, please stop barking up the wrong tree. Source source source source source.

Group permissions

If you will not provide true validation on save (with useful parser error messages and specific line numbers), then IMO this is a 2. should have at the very least. However, if you do implement a validating parser, then I agree with Jens that this is not necessary at all.

As for this comment below the table of user stories:

Not Doing

• Not doing find / replace in source - users can copy/paste into a client-side editor to do this.

It's not clear what you're trying to say here. But I have some grave concerns depending on exactly what you mean by this:

• If your source editor will allow a browser's built-in search functionality to work unhindered (for example, Chrome's built in Ctrl-F and Ctrl-G functionality), then there's no problem at all and I agree that you do not need to build a special find/replace in the source editor. For example, your 3.5.x wiki syntax editor worked perfectly fine with browser's built-in Ctrl-F and Ctrl-G.
• However, if your source editor requires a bunch of custom javascript similar to Arsenale's Invisible Ink plug in and therefore the browser's inherent Ctrl-F/Ctrl-G functionality is overridden will not actually find things inside the editing field, then Houston, we have a HUGE problem!

Bottom line is that you must enable users of the source editor to at least search for specific strings directly within your editing field. It will be far too cumbersome to be required to Ctrl-A > paste into external editor > search/scan/replace in external editor > Ctrl-A > paste back into your source editor. Why? Because much of the time real-world workflow requires us to simply scan the source for errors that probably don't exist but we're checking to make sure. The most obvious use case here is checking for spacekey strings when we've moved a set of pages from one space to another space, which creates all manner of broken links in the moved pages (based on spacekeys being auto-written by Confluence during the moves). I might scan 10 such pages for the original spacekey and find instances of such in only 2 of the set that was moved. Please don't make me copy/paste 10 separate pages into external editors to check for this sort of thing. I just want to open your source editor, do Ctrl-F and let Chrome's native Ctrl-F functionality tell me the info I need to know.

Syntax highlighting

Again with the "HTML" option. No, this is not an option. We do not want to edit the HTML used to render the RTE editor. We want to edit the storage format directly. Really, please stop barking up the wrong tree. Source source source source source.

What does syntax highlighting have to do with HTML? I think (Hope) they are referring to XML syntax highlighting as well as such:

<?xml version="1.0"?>
<myAnimals count="2">
  <cat name="Fister" age="10" />
  <dog name="Caton" age="12" />
</myAnimals> 

I mean that is what it in essence is to me, the Indentation and colors.

Hurrah! Very glad to see this.

The "Find/replace in source" is important in the case where procedures are being updated and references need bulk changes.  We have to do these kind of changes in SQL today.  Virtually any text editor allows find/replace.  This should have that functionality as well.

There's one user story that is missing entirely. This story should be a release requirement for the first push of the new Advanced Editor feature set. Your "strategic fit" for 3. Offline content editing is glad-handing the user requirements for "bulk edit link properties" and "bulk edit macro properties" by limiting the scope of these requirements to single-page editing scenarios. In fact, the scope for these two requirements is much larger than that. Hence, the following story you should add to the set above:

Story title: Search storage format in global page search and {pagetreesearch} macros

Story description: AS AN advanced content author I WANT to be able to use the global search control and {pagetreesearch} macros to search for specific storage syntax across all multiple pages in one operation SO THAT I can find pages that might need to be fixed in a bulk cleanup operation when I am unsure how many pages, or exactly which pages, might contain some "broken" storage format syntax.

Priority order: 1. Must have

Implementation notes: A simple one-off page editing solution does not solve typical use cases where reference links might require a bulk change operation, or where a bulk page move operation from one space to another results in incorrect spacekey links, etc. Users must be able to find all potentially broken pages by inputting a string from the XML storage format syntax that they know is problematic. The simplest way to achieve this goal (from a UI point of view), is to use a search prefix similar to the current all: search prefix that can be used in the search control of a space that has been restricted to search only within that space (via, for example, the limit search results to this space checkbox in the Documenation Theme). For example, a search prefix of xml: or storage: could be used to denote that the following string should be searched for within the storage format of all pages instead of in the rendered rich text of all pages. Such a search prefix should be able to be combined with other search prefixes. For example, if a space uses the Documentation theme and is constraining searches to that space alone, you should be able to use all: xml: searchstring to expand to the scope of the search to the entire Confluence instance.

I'll put it this way to emphasis the importance of this story: Even if every other story listed above is in place, and even if the full storage format language reference is finished and is a masterpiece of technical writing, my company would still not be able to migrate from 3.5.x until this particular story is delivered to the field. This is critical functionality. It's one of the legs you cut out from under us with the new 4.0 editor. In 3.5.x and earlier, these use cases were supported because you could search the underlying wiki markup across all pages in a spacetree/space/instance by using quoted strings in the search controls.

excellent - this is why I trust Atlassian." be the change you seek" indeed

Which existing validating XML editors are you looking at as examples?

I have played with many, but I mostly use the following three:

• Arbortext Editor (costs money)
• Altova XMLSpy (costs money)
• jEdit with XML plugin (free)

For "developer"-type XML editing purposes (such as creating XML schemas, and creating small-ish XML document instances for, say, web service testing), I prefer jEdit, because it retains everything that I like about text editors, such as regular expression-based search and replace (across files) that you can use to edit the markup just as easily as the content, with dedicated XML features such as schema-based auto-completion. It just feels a little less restrictive to me.

For writing documentation (specifically, in DITA), I typically use Arbortext. The pseudo-wysiwyg display - still showing markup, to a greater or lesser degree, but where the markup is protected from being edited like text content - makes for quicker authoring. But I often still revert to jEdit when I want to make bulk source changes.

I know, these are all desktop applications, not browser-based.

I think the following browser-based editors are at least worth looking at (I've only tried their demos):

• WYMeditor (open source, and integrated in MediaWiki, according to the WYMeditor website... I suspect someone at Atlassian will be rolling their eyes at this "tip", because it seems like an obvious starting point)
• Xopus

I have not yet played with DITA Storm.

A suggestion of a 'nice-to-have' would be something equivalent to the 'Insert Wiki' feature in the standard RTE for 'Insert Storage Format'. Would probably follow the same rules as the main Source Edit view with regards to permissions. This would make it simple to pass around snippets of how to use our more complicated macros without sending more novice users to the full Source view. It would validate the content and insert it directly.

If I use the advanced (source) editor to insert a <p> element into an <li> element, then I hope that the rich text editor will:

• Correctly render the nested paragraph
• Allow similar markup structures to be created
• Not replace the nested paragraph with two line breaks!

This is just an example. The same applies to any other markup created using the advanced editor. The rich text editor should handle any valid markup (that is, valid according to the - published! - schema), and not change the markup. Even - at the risk of sounding like I'm contradicting myself - if the markup inserted in the advanced editor is non-semantic. That is, I would not want the rich text editor to change "<br/><br/>..." into "<p>...</p>".

Bravo! Thank you for re-establishing text-based edits!

Now that I've had a chance to read up on this markup issue, and to spend some time looking at the storage format, I have to say that this proposal is a hilariously terrible solution to the many problems caused by your removal of the wiki markup option — not least of which is forcing alien workflow styles on people and wasting their time to retrain on something that had once been straightforward.  The entire point of a wiki is to not have to edit *ML languages.

I've been the point person on Confluence at my place of work, and I cannot tell you what sort of grief I've been getting since this change.  Making people edit custom XML is not going to make them any happier.  I've already been snorted at when discussing this proposed solution with a co-worker.

If you are really committed to this appalling solution, allow me to at least suggest for the "Validation" story that it should refuse bad mark-up, not strip things out.  We already have more than enough mystifying behaviors from the RTE.  Also, if we're to be forced to look at XML, syntax highlighting is a must-have, not a should-have.

I like to add another user story  / strategic fit which I'm missing from the specification definition:

While the above description about copy/paste talks about pasting into 3rd party editors for further processing, I frequently copy/paste from an existing Confluence page into a new (or existing) one. You may call it "code reuse", "program by sniffing code" or what-ever, anyway it's an effective way to create pages by looking at examples from other pages (maybe even on different confluence databases) created by Confluence users which are smarter than myself. "Reusing" great examples is easier than to study documentation

So in 3.x, I used to view the Wiki markup, copy and paste into my page using Wiki markup editor.

I tried to do the same in 4.0 (haven't tried it in 4.1 yet).  The problems I was facing:

1) View Source does not reveil which macro was used (ok, some do meanwhile) neither which parameters were provided

2) When I pasted into the 4.0 editor, various macros were not interpreted correctly (see related reports on the 4.0 editor feedback page). You may classify this as bug-fix requirement to the 4.0 editor, but in the meantime being able to edit the source is the only way to fix such errors.

Being able to view the macro parameters beyond what "View Source" provides helps to add them to my pages (learning about them, and how to use them).

The above does not add anything to the requirements, but may add to the test scenarios and to the reasons for the development of the advanced editor.

and here is one more user story (from 3.5)

I had added a plugin from different source (I think it was table-plus from Bob Swift, but not sure). I faced the issue that the 3.5 RTE did not know about the added macro (maybe this was an error in the plugin registration?), so its parameter could not be edited in the RTE. If I recall correctly, the RTE did even escape the \\{ of the macro because it didn't understand that it was such.

Using the Wiki Markup Editor was the only way to add this macro to my page.

I'm not sure whether the same can happen in 4.0 ? And whether this would could be handled then through the Advanced Editor?

So when are we getting this?

I'm concerned about your use of the terms "valid", "invalid", "validation", and - others have also mentioned this last one - "XHTML".

XHTML?

In the body of this page, you write:

With the switch from wiki markup to xhtml

as if your source format (which includes proprietary markup) is XHTML.

Validation?

Also from the body of this page:

Validation ... There should be at least some basic validation (e.g. that the xml is well formed). This validation is performed when applying the changes in the source editor and returning to the Rich Text Editor.

...

Other markup, that is considered invalid,

...

Validation ... editor currently strips out bad markup.

...

visually parse and spot invalid markup

I am concerned that you are mixing colloquial and technical definitions of the terms "valid", "invalid", and "validation". And that you are blurring the distinction between well-formedness and validity by coining the term "basic validation".

By "technical definitions", I am referring specifically to the definitions in the XML W3C Recommendation:

Definition: A data object is an XML document if it is well-formed, as defined in this specification. In addition, the XML document is valid if it meets certain further constraints.

Definition: An XML document is valid if it has an associated document type declaration and if the document complies with the constraints expressed in it.

Not XHTML, no validation

I am concerned that the information that you (Atlassian) have provided in the body of this page might lead readers to think that your source format is XHTML, and that the advanced editor will (in v1, "1: Must have") validate that XHTML (according to the technical definition of this term).

My understanding is that neither is true. I would be happy to be wrong in both cases.

Questions

I am hoping for single-word, "yes" or "no" answers from you. (I think the answer to both questions is "no".)

1. Is the Confluence storage format XHTML?
2. Will the Confluence advanced editor validate the XML source (according to the XML definition of the term "validate")?

(If your answer to question 1 is "yes", please cite the specific type: for example, XHTML 1.0 Strict?)

I miss the dynamic Reveal Codes/WYSIWYG "split window" view of WordPerfect, where you can edit content in a Reveal Codes view - effectively, a markup view - occupying the bottom half of the application window, and immediately see the effect in the WYSIWYG view in the top half of the application window, without having to hide (switch between) one view to see the other. A similar "XML source"/"preview" arrangement (with both views visible at the same time) would be nice to have in the new Confluence editor.

Actually, "preview" is probably the wrong term, because, ideally, I would like that "preview" to be editable. So, ideally, I guess I really mean (to match WordPerfect) a split "XML source"/"rich text" editor arrangement. But a split "XML source"/"non-editable preview" arrangement (where the preview dynamically updates as you edit the XML source) would certainly be nice to have.

Guys,

whatever you do, do not do anything that will require anything more complicated than a plain old HTML <textarea>.

It will be a huge maintenance burden and a pain to use if it’s not 100% perfect, and we all know it never will.

If you want to give users syntax highlighting for XML/HTML/XHTML source, do it in a way that does not require implementing a full-blown text editor in Javascript and will benefit users of other web-based applications as well — 1) have HTML5 extended with a syntax highlighting specification for textarea and pre/code, 2) contribute an XML/HTML/XHTML syntax highlighter to Firefox and Chrome, 3) use that in Confluence/JIRA/whatever.

Or, if you are not prepared to do that, make it just plain old textarea without syntax highlighting. Those who need highlighting can copy-paste the markup into an external editor, do the necessary changes, and copy-paste new text back. Advanced users will just use a browser extension, such as It’s All Text, that adds an external editor to all textareas.

Will this Plugin go into the OnDemand system as a matter of course or will someone have to request it do you know?

Bill, John: When will the next version be ready? Not trying to be harsh, but this "beta" version is not ready for even testing, let alone production use. I've already tried to help you guys understand why stripping bad markup instead of simply refusing bad markup and displaying validation messages to help find and correct the bad markup is a very bad idea. In response: crickets.

And regardless of whether you think that's an arguable point or not, without search and replace within the editor, again, this is simply not even worth testing let alone trying to use in a production context.

I'd really appreciate a rough estimate on when you think you'll have the remaining features ready.

This seems to be missing my user story.

I would like to be able to extract the contents of the page as wiki-markup so that my script which updates information on the page can modify the page, allowing me to paste the updated page back into Confluence.

Please install the beta plugin in the Confluence 4.x sandbox.

When is the ability to search for underlying storage format syntax across all pages in a space (or across the entire confluence instance) in the global search control AND in any {pagetreesearch} search control coming?

using a code macro appears to crash the plugin (beta 1).  when inserted into the editor, it will hang after clicking the apply button.  when editing a page where the macro already exists, the <> button does not appear at all.  not sure if its the code macro itself, or the text I'm using inside it, this is what shows up when I retrieve the source format through the tools button:

<ac:macro ac:name="code">
  <ac:parameter ac:name="language">
    none
  </ac:parameter>
  <ac:plain-text-body>
    <![CDATA[dxdiag /t %USERPROFILE%\desktop\dxdiag.%USERNAME%.txt]]>
  </ac:plain-text-body>
</ac:macro>

"With the switch from wiki markup to the new XHTML-based storage format..."

Where is the discussion/documentation on the role and future of wiki markup? Are you saying here that wiki markup has essentially been replaced with XHTML? If so, where is this switch discussed?

Atlassian,

Regarding validation of XML in the advanced editor (Source Editor plugin): you might be interested in looking at the hurriedly cobbled together JavaScript in Wikifier (see my recent announcement on the parent "Confluence 4.0 Editor - Customer feedback" page), which uses the browser's (IE9, or current production version of Chrome, Firefox, or Safari) built-in schema-based validation to provide more than just the completeness checking that is currently provided by the advanced editor.

If I were going to further develop Wikifier (probably not), I would write more code to further "parse the (XML) parsing errors", and so offer users better feedback about XML errors, without requiring any server-side processing. In case it's an issue: I would personally have no qualms specifying different (newer/later) minimum browser levels for the advanced editor than the rich text editor (but I'd want to check the browser level in code, and either gracefully "degrade" the experience - for example, no schema-based validation - or require users to upgrade).

I realize that this suggestion contradicts another suggestion I've just made elsewhere :

Rather than investing more development effort right now in the Source Editor plugin, spend some time making the data more accessible to external XML tools and methods.

How can I copy/past from one wiki page into another one using only the RTE in Confluence On Demand? When I copy/paste content, all the formatting in the source is lost. My intent was to retain the formatting. If this is not possible, are there plans to implement this? I looked at the confluence Jira tickets on copy/paste and could not find this specific issue. If there is a ticket already, can you point me to the issue? Thanks.

Sorry for bringing this up if this has already been covered.

As a developer i cannot appreciate that wiki markup is not accessible anymore. If I want to document technical information the editor is not at all user friendly.

i.e.:

If I write a code block and after that a new h1 header there is no way to add simple content after the code block anymore unless I set my cursor before the header, type enter and then alter the empty header line to paragraph font. This is exactly one of the reasons wiki-markup should be available for exclusive issues.

{code}
  some code
{code}

??? content here afterwards ???

h1. Some header

I solve this issue by editing my wiki-markup in a text-editor .... and when updated I delete all content on the page, press ctrl + shift + D and paste my new markup in there. This gives me the slightest feeling of going back to the stone age.

I really do understand why making it more available to the non-it personnel, but blocking this functionality just isn't the solution. Wiki-markup goes very fast for people who know how to use it. Right now you chose to be just another rich text editor. And that is not a hole in the market. It saddens me that you chose this easy path instead of making a very nice hybrid for a greater public.