Documentation for Confluence 5.4.
Documentation for Confluence OnDemand and earlier versions of Confluence is available too.

Skip to end of metadata
Go to start of metadata
Icon

1.0 for this plugin is now available (3/Apr) on the plugin exchange under the name Confluence Source Editor.

Fixed:

  • Support for using the source editor in comments.

Beta 2 included enhancements include support for:

  • Find and replace within the page
  • Restricting access to the Source Editor to particular user groups.

We welcome your feedback - please file issues in the JIRA project at https://jira.atlassian.com/browse/SOURCE

Icon

Background and strategic fit

Up until now, we haven't allowed users to edit the underlying editor storage format (though this was easy for anyone to hack, using Firebug). As customers upgrade to Confluence 4.0, we're learning there are three main reasons people want source access: 

1. Customers Feel "Locked-in"

With the switch from wiki markup to the new XHTML-based storage format, and because people were no longer able to edit a text-based version of their page directly, they felt they were locked into Confluence. While you could always still get your content out via the API or through WebDAV but people wanted to be able to see it directly.

2. Fixing Formatting Niggles

When a user hits an occasional bug, he or she wants a quick workaround. Over the years we trained everyone on this - what do you do when you hit a bug in the RTE? Switch to wiki markup. This wasn't really acceptable then and definitely isn't now, but some customers are asking for it... "give me access to the source so I can quickly fix something that's gone wrong."

3. Offline content editing

Some users want to copy the source of their page out of Confluence to work with it in other ways, whether it be updating content offline, sharing with others via email, or using a preferred text editor for things like bulk changing of link properties. Use cases include:

  • Bulk edit link properties
  • Bulk edit macro properties
  • Confluence offline editing
  • Editing via preferred editor (emacs, textmate, etc)

Requirements

User Story Title

User Story Description

Priority Order

Implementation Notes

Status

Editor Niggles

User can access source view from inside the Confluence editor.

1: Must have

  • This is a short-term solution to working around around minor bugs. Long term we should be solving these problems.
  • This will edit the storage format source.
BETA 1
Copy paste source

User can copy / paste source in full fidelity. Users should be able to:

  1. Copy the entire content of a page
  2. Paste into a separate editor
  3. Manipulate it and
  4. Paste it back into the source view of the editor
  5. Save the page
1: Must have BETA 1
Edit link sourceUser should be able to change link source from inside the editor1: Must haveThis would help with bulk change operations like changing URLs, or parts of URLs in links.BETA 1
Edit macro parametersUsers should be able to change macro parameters from inside the editor1: Must have

Example macro storage format:

BETA 1
Validation
  • The user should not be able to save any content that will break the page.
  • The editor should do validation to make sure only supported markup is being sumbitted.
1: Must have

There should be at least some basic validation (e.g. that the xml is well formed). This validation is performed when applying the changes in the source editor and returning to the Rich Text Editor.

Other markup, that is considered invalid, is either tolerated or stripped when saving the file (the content itself is not lost, just the markup tags). It would be desirable to validate this before save, but this may not be available in the first release.

If a save does result in a bad rendering of a page, then reverting an edit via the history is an acceptable trade-off.

BETA 1
Syntax highlightingThe markup should be formatted and indented so that the source is as easy as possible to visually parse and spot invalid markup.2: Should have BETA 1
Search and replaceIt should be possible to search and replace text in the source markup.2: Should have

There should be some basic search and replace functionality. More sophisticated searching may be possible (e.g. the reg. ex) depending on the final editing framework used. See CONF-24200 - Ability to Find/Replace on Formatting and Macro parameters within a page Open .

BETA 2

Group Permissions

Admin can specify which users / groups can have access to edit source.

3: Nice to have

This is necessary so that only people who REALLY need access can get access.

BETA 2
ValidationUser can see which markup he posts is invalid - editor should refuse bad mark-up2: Should have

(tick) The source editor does validation for malformed XML - so if you forget to open or close a tag, it will not let you post it and will move your cursor to where the error exists if that location is knowable.

(tick) If you define a bad parameter in a macro, Confluence will ignore this parameter:

(error) If you erroneously put a macro or link parameter in a rich text field the tags get stripped

BETA 1

 

UX

  • Expecting an icon to the left of help  in the toolbar that will pop up a dialog when enabled for a user.

Supportability

  • This will be implemented as a separate plugin that can be installed by a Confluence site administrator.
  • The plugin will be maintained by Atlassian.

Notes

We refer to the Confluence storage format as 'XHTML-based'. To be correct, we should call it XML, because the Confluence storage format does not comply with the XHTML definition. In particular, Confluence includes custom elements for macros and more. We're using the term 'XHTML-based' to indicate that there is a large proportion of HTML in the storage format.

  • No labels

84 Comments

  1. Bravissimo. What's the timeline?

    1. Yes, please state at least a ballpark date/release number where we can expect to see this. Will it happen in Q1 of this year? Q2?

      The answer is important for those of us who are on the line to advise our own enterprises of whether we can afford to stay in "wait and see" mode for a bit longer or need to actively throw resources at planning a migration to another wiki solution.

      1. We're aiming to have it this quarter.

        1. and while you are at it.

          have confluence convert the default layout stuff from xhtml to wiki markup. and leave the other new things in xhtml.

          Place a button next to it with xhtml only or hybrid wikimarkup and voila we have a wikimarkup editor again that is more powerfull then before and now with xhtml!

           

          1. I'm not sure that making Confluence a hybrid of wiki and XHTML markup will really do anyone any favours in the long run. As a developer, I don't really want to have to support both options long term.

            1. why not? you could easily parse the old style formatting and output it to wiki markup. and the new style advanced stuff then simply output it to xhtml, then both worlds are happy.. do you think the old style formatting wiki markup macro will change vs the formatting? if so then simply mark these things tagged with a new or old version. then people can really still use wiki markup with all new style things mixed. 

               

              /offtopic: why was not chosen to create a new advanced wiki markup vs xhtml? or even better an advanced version of wikimarkup could easily replace most xhtml tag open/close editing. (if you make mistakes etc in this) confluence should be a wiki not only an online wysiwyg editor that porely competes with googledoc

  2. Kudos on realizing this... Finally!..


    Just wanted to add a few comments because I think you are seeing certain things a bit wrong/from the negative point of view.

     

    Editor Niggles

    User can access source view from inside the Confluence editor.

    1: Must have

    • This is a short-term solution to working around around minor bugs. Long term we should be solving these problems.
    • Need to determine whether we use editor HTML source or the storage format source.

     

    You write this is short term, but I would say on the contrary, this is very long term in my opinion (which is a good thing). What you are focusing at here is "Bug A", and for that it is true that the a "Advanced Editor" is a short term workaround to solve "Bug A". BUT it is a long term solution to solve "Bugs" (plural). Because face it, your RTE will never be perfect, there will always be small "Niggles" as you name it. This editor is a long term solution to work around any future found "Editor Niggles".

    To the question of weather to use "HTML" or "Confluence Markup". I would say use the raw storage format, I think from the status macro here that is easier to deal with. But whatever you do don't add a third conversion!.
     

     

    Copy paste source

    User can copy / paste source in full fidelity. Users should be able to:

    1. Copy the entire content of a page
    2. Paste into a separate editor
    3. Manipulate it and
    4. Paste it back into the source view of the editor
    5. Save the page
    1: Must have 

     

    Will a XSD be available here? Personally I don't use this to much, and not at all now, what I miss is the auto generate capabilities but never-mind that because that will be solved as well, however an XSD could make on the fly validation for persons having an editor supporting it (E.g. Visual Studio), and since an XSD would be a way to validate your input otherwise (you should already have one!!!!!!) then I can't see a public released XSD should be hard to come by?.

     

     

    Validation
    • The user should not be able to save any content that will break the page.
    • The editor should do validation to make sure only supported markup is being sumbitted.
    1: Must haveMight not be able to guarantee this. Macro markup is quite specific, and I'd image this could be broken fairly easily. Trying to stop this would make this a much bigger solution. I think that reverting an edit via the history is an acceptable trade-off. I'm not expecting any kind of server round-trip for validation until saving in the main editor.

     

    I am not sure how the invocation of macros are, but would it be possibly to just catch any errors from them (on render time) and if an error happens display it as a "red cross" or maybe a "red error box" where some error text could be displayed from the error thrown?... That would be more than enough. (Much like the old confluence would actually display errors for macros in the page).

    After all, having to accommodate for "macros" here would make you unable to use XSD's to do the validation as that would be a different XSD pr. installation depending on what plugins it had installed.

     

    Syntax highlightingThe markup should be formatted and indented so that the source is as easy as possible to visually parse.2: Should haveAs discussed if this is a quick win from reusing some code formatting from speakeasy, we can do this. Not sure if it works with HTML, or only well-formed XML.

     

    If you do this, make sure it is 110% rock solid, I know this is far more simple that a RTE, but in worst case ending up with another bug prone editor won't help us a bit. We can always copy things out of confluence to begin with if we wan't this type of editing experience, focus on the important pieces first. (The indentation should not be an issue to provide, that should be possible though all standard XML libraries I hope, at least it is for the big ones).

     

    Group Permissions

    Admin can specify which users / groups can have access to edit source

    3: Nice to have

    This is necessary so that only people who REALLY need access can get access.

     

    This is not needed at all I would say, if people are scared of an XML format, they will stay away. And if the don't reverting pages is luckily really easy!. I would rather have the editor "snapshot" the page on switching to source so you could revert within edit mode (Higher level undo). But that is also nice to have, but I would say more usefull.

    I mean even with your "Validation" as a Must have, I would say that restricting this feature is pointless, people can't really break pages by accident, they can break macros, but then I would much rather that you put effort into telling people just how easy it is to revert when you made such a mistake. And since it is confined to macros these would also break in the RTE view?... making them maybe rather easy to fix up before submitting the final version of a page.

    1. Jens (and Atlassian), remember that there are some strong criticisms of the W3C XML Schema specification (XSD documents):

      • They are much harder to mentally parse and follow (by a human) than an XML DTD.
      • XSD provides no way to state that the value of one attribute is dependent on the values or presence of other attributes (aka co-concurrence constraints). Because Confluence is macro-heavy, understanding how to tweak the storage format for a macro will require us being able to clearly understand the co-concurrence constraints of the macro attributes. Only DTD-style element declarations will help us in this regard.
      • XSD doesn't clearly specify the rules for unordered content, and often can't specify unordered content rules in some cases.
      • There are a lot more DTD-validating parser libraries out there than there are XSD-validating parser libraries. If we want Confluence's built-in source editor feature to provide validation with useful error messages, we should be steering them to develop a DTD for their proprietary XML schema, and use a DTD-validating parser in their new source editor. (That said, the last time I looked hard at this was several years ago, so maybe by now there are in fact more and better XSD-parser libraries.)
      • Plus, DTD-style element declarations are much easier to read/parse by humans than XSD-style declarations.
      1. If a DTD can cover it then the choice comes down to being widely supported by existing editors or not to me. XSD's are more powerful in many ways, I can't say if that is needed however. It's properly not when i think about it because this is largely structure with "general" content (Which CDATA and PCDATA should more than cover) so I guess a DTD is fine. I Am not getting into a religious war here, XSD and DTD are different things that overlaps in some capabilities.

        As for the points you list, they are hardly relevant to me.

        • I Won't need to read and understand the DTD nor the XSD, I just need it to veryfy my content.
        • Macroes should merely be validated against "outer" structure, which will be static so no need to validate internal content based on "macro name", that would mean you need to dynamically generate new DTD's or XSD's each time a new plugin is deployed. I would advice against that.
        • I have never had issues with specifying un-ordered content in XSD's
        • Last point is the same as the first i believe

         

    2. Anonymous

      I do understand the need for validation. Atlassian needs to make sure that source editing doesn't mess up a page up to a point that even the javascript features don't work anymore. Imagine the user introduces (saves) an error which results in a broken "Tools" menu.

      Then "how easy it is to revert " is no longer available.

  3. My strong preferences regarding specific user stories listed above:

    Editor Niggles

    You must enable us to edit the storage format. Editing the HTML used to render the RTE in a browser is essentially useless. How can you even consider the HTML? What's going to be round-tripped to other environments by people who need/choose to do so is the storage format. What you're documenting over in the other new page is the storage format. The people who need access to "fix bugs in the RTE" need to understand and apply and work directly in the storage format. There simply is no choice here.

    Also I agree whole-heartedly with Jens' comment above about your framing/understanding of "Editor niggles". Your RTE will never be perfect. Never. Not once in the history of computing so far has any attempt at an RTE ever been flawless, and as I've mentioned before in the feedback thread, any product that lasted for any decent length of time in the marketplace has always needed to provide a way to get at and directly edit the underlying markup.

    Edit macro parameters

    Same comment as for editor niggles. If I'm tweaking a macro, I need to do it in the raw storage format with the real underlying XML content model and attributes. Editor HTML is not a viable option at all. Don't even think for a moment that trying to fix things through the Editor HTML is an option.

    Validation

    So what if it's harder for you guys to build a validating parser on check-in with error messages that spell out exactly what's invalid? If you're making us work in XML-land, you need to provide us with at least that minimal aspect of a decent XML editor. The folks over at Arsenale seemed to quickly implement a rudimentary validating parser with syntax formatting/highlighting and validation error messages with line numbers, so why can't you? If you need a shortcut, go buy their IP and integrate it with your solution.

    To be clear, simply not saving the page and throwing up a generic error message that there was some invalid content is NOT sufficient. You must tell us what element on what line is invalid, and why. How else will we be able to zero in on what needs fixed? This is SMGL/XML 101 here, folks. You forced this on us, so do the right thing.

    To be crystal clear, the general workflow should be:

    1. Open page in "storage format editing" mode.
    2. Edit the storage format (XML) directly.
    3. Click Save or Preview (ideally you'll provide a preview option even for the storage format editor).
    4. Before the editor actually checks in the changes, it runs validation. If errors occur, the state of the storage format "draft" is saved and a set of validation errors are displayed.
    5. If the user resumes editing, no changes in the draft were lost and they can try to fix the errors reported by the validation report.
    6. Iterate steps 3-5 until the user cancels out or the draft is successfully validated and stored in the system as a new version of the page.
    Syntax highlighting

    Again with the "HTML" option. No, this is not an option. We do not want to edit the HTML used to render the RTE editor. We want to edit the storage format directly. Really, please stop barking up the wrong tree. Source source source source source.

    Group permissions

    If you will not provide true validation on save (with useful parser error messages and specific line numbers), then IMO this is a 2. should have at the very least. However, if you do implement a validating parser, then I agree with Jens that this is not necessary at all.

  4. As for this comment below the table of user stories:

    Not Doing

    • Not doing find / replace in source - users can copy/paste into a client-side editor to do this.

    It's not clear what you're trying to say here. But I have some grave concerns depending on exactly what you mean by this:

    • If your source editor will allow a browser's built-in search functionality to work unhindered (for example, Chrome's built in Ctrl-F and Ctrl-G functionality), then there's no problem at all and I agree that you do not need to build a special find/replace in the source editor. For example, your 3.5.x wiki syntax editor worked perfectly fine with browser's built-in Ctrl-F and Ctrl-G.
    • However, if your source editor requires a bunch of custom javascript similar to Arsenale's Invisible Ink plug in and therefore the browser's inherent Ctrl-F/Ctrl-G functionality is overridden will not actually find things inside the editing field, then Houston, we have a HUGE problem!

    Bottom line is that you must enable users of the source editor to at least search for specific strings directly within your editing field. It will be far too cumbersome to be required to Ctrl-A > paste into external editor > search/scan/replace in external editor > Ctrl-A > paste back into your source editor. Why? Because much of the time real-world workflow requires us to simply scan the source for errors that probably don't exist but we're checking to make sure. The most obvious use case here is checking for spacekey strings when we've moved a set of pages from one space to another space, which creates all manner of broken links in the moved pages (based on spacekeys being auto-written by Confluence during the moves). I might scan 10 such pages for the original spacekey and find instances of such in only 2 of the set that was moved. Please don't make me copy/paste 10 separate pages into external editors to check for this sort of thing. I just want to open your source editor, do Ctrl-F and let Chrome's native Ctrl-F functionality tell me the info I need to know.

    1. I've added searching and replacing as a should have.

      I've been implementing a spike of the source editor over the last couple of days, and this should be able to be included in the initial release of the plugin.

  5. Syntax highlighting

    Again with the "HTML" option. No, this is not an option. We do not want to edit the HTML used to render the RTE editor. We want to edit the storage format directly. Really, please stop barking up the wrong tree. Source source source source source.

    What does syntax highlighting have to do with HTML? I think (Hope) they are referring to XML syntax highlighting as well as such:

    I mean that is what it in essence is to me, the Indentation and colors.

    1. My quoted comment is based on the highlighted bit from their Implementation Notes column for the Syntax highlighting story

      As discussed if this is a quick win from reusing some code formatting from speakeasy, we can do this. Not sure if it works with HTML, or only well-formed XML.

      1. The editor will be for the storage format, not the editor format (the html used to render the editor content in the browser).

        The storage format is in xml.

  6. Hurrah! Very glad to see this.

  7. Anonymous

    The "Find/replace in source" is important in the case where procedures are being updated and references need bulk changes.  We have to do these kind of changes in SQL today.  Virtually any text editor allows find/replace.  This should have that functionality as well.

    1. This has been added as a "should have" requirement. Unless there are some bugs that prevent that from being included, I'd expect this feature to be in the initial release.

  8. There's one user story that is missing entirely. This story should be a release requirement for the first push of the new Advanced Editor feature set. Your "strategic fit" for 3. Offline content editing is glad-handing the user requirements for "bulk edit link properties" and "bulk edit macro properties" by limiting the scope of these requirements to single-page editing scenarios. In fact, the scope for these two requirements is much larger than that. Hence, the following story you should add to the set above:

    Story title: Search storage format in global page search and {pagetreesearch} macros

    Story description: AS AN advanced content author I WANT to be able to use the global search control and {pagetreesearch} macros to search for specific storage syntax across all multiple pages in one operation SO THAT I can find pages that might need to be fixed in a bulk cleanup operation when I am unsure how many pages, or exactly which pages, might contain some "broken" storage format syntax.

    Priority order: 1. Must have

    Implementation notes: A simple one-off page editing solution does not solve typical use cases where reference links might require a bulk change operation, or where a bulk page move operation from one space to another results in incorrect spacekey links, etc. Users must be able to find all potentially broken pages by inputting a string from the XML storage format syntax that they know is problematic. The simplest way to achieve this goal (from a UI point of view), is to use a search prefix similar to the current all: search prefix that can be used in the search control of a space that has been restricted to search only within that space (via, for example, the limit search results to this space checkbox in the Documenation Theme). For example, a search prefix of xml: or storage: could be used to denote that the following string should be searched for within the storage format of all pages instead of in the rendered rich text of all pages. Such a search prefix should be able to be combined with other search prefixes. For example, if a space uses the Documentation theme and is constraining searches to that space alone, you should be able to use all: xml: searchstring to expand to the scope of the search to the entire Confluence instance.

     

    I'll put it this way to emphasis the importance of this story: Even if every other story listed above is in place, and even if the full storage format language reference is finished and is a masterpiece of technical writing, my company would still not be able to migrate from 3.5.x until this particular story is delivered to the field. This is critical functionality. It's one of the legs you cut out from under us with the new 4.0 editor. In 3.5.x and earlier, these use cases were supported because you could search the underlying wiki markup across all pages in a spacetree/space/instance by using quoted strings in the search controls.

    1. This is a perfectly legitimate use case but outside the scope of a single-page source editor. I've created an issue at http://jira.atlassian.com for this CONF-24568 - Confluence search is broken for data within content elements, such as URLs in links, parameters in macros, image names on pages Open where we can discuss further.

      1. Thanks, Bill. That works.

  9. excellent - this is why I trust Atlassian." be the change you seek" indeed

  10. Which existing validating XML editors are you looking at as examples?

    I have played with many, but I mostly use the following three:

    • Arbortext Editor (costs money)
    • Altova XMLSpy (costs money)
    • jEdit with XML plugin (free)

    For "developer"-type XML editing purposes (such as creating XML schemas, and creating small-ish XML document instances for, say, web service testing), I prefer jEdit, because it retains everything that I like about text editors, such as regular expression-based search and replace (across files) that you can use to edit the markup just as easily as the content, with dedicated XML features such as schema-based auto-completion. It just feels a little less restrictive to me.

    For writing documentation (specifically, in DITA), I typically use Arbortext. The pseudo-wysiwyg display - still showing markup, to a greater or lesser degree, but where the markup is protected from being edited like text content - makes for quicker authoring. But I often still revert to jEdit when I want to make bulk source changes.

    I know, these are all desktop applications, not browser-based.

    I think the following browser-based editors are at least worth looking at (I've only tried their demos):

    • WYMeditor (open source, and integrated in MediaWiki, according to the WYMeditor website... I suspect someone at Atlassian will be rolling their eyes at this "tip", because it seems like an obvious starting point)
    • Xopus

    I have not yet played with DITA Storm.

  11. A suggestion of a 'nice-to-have' would be something equivalent to the 'Insert Wiki' feature in the standard RTE for 'Insert Storage Format'. Would probably follow the same rules as the main Source Edit view with regards to permissions. This would make it simple to pass around snippets of how to use our more complicated macros without sending more novice users to the full Source view. It would validate the content and insert it directly.

  12. If I use the advanced (source) editor to insert a <p> element into an <li> element, then I hope that the rich text editor will:

    • Correctly render the nested paragraph
    • Allow similar markup structures to be created
    • Not replace the nested paragraph with two line breaks!

    This is just an example. The same applies to any other markup created using the advanced editor. The rich text editor should handle any valid markup (that is, valid according to the - published! - schema), and not change the markup. Even - at the risk of sounding like I'm contradicting myself - if the markup inserted in the advanced editor is non-semantic. That is, I would not want the rich text editor to change "<br/><br/>..." into "<p>...</p>".

    1. Yes. +1

      And this is another reason for providing both your full XSD and also for providing a full language reference for your schema. Doing so will keep you at Atlassian "honest" and ensure that your own devs and QA are testing for conformance with the rules of your schema, and it will enable users of your Advanced Editor to clearly understand what they can and cannot do when editing the storage format directly.

  13. Bravo! Thank you for re-establishing text-based edits!

  14. Now that I've had a chance to read up on this markup issue, and to spend some time looking at the storage format, I have to say that this proposal is a hilariously terrible solution to the many problems caused by your removal of the wiki markup option — not least of which is forcing alien workflow styles on people and wasting their time to retrain on something that had once been straightforward.  The entire point of a wiki is to not have to edit *ML languages.

    I've been the point person on Confluence at my place of work, and I cannot tell you what sort of grief I've been getting since this change.  Making people edit custom XML is not going to make them any happier.  I've already been snorted at when discussing this proposed solution with a co-worker.

    If you are really committed to this appalling solution, allow me to at least suggest for the "Validation" story that it should refuse bad mark-up, not strip things out.  We already have more than enough mystifying behaviors from the RTE.  Also, if we're to be forced to look at XML, syntax highlighting is a must-have, not a should-have.

    1. allow me to at least suggest for the "Validation" story that it should refuse bad mark-up, not strip things out.

      I've added this as a story to consider for future versions. V1 will strip out bad markup.

      1. Bill, I strongly feel this should not be a "nice to have" and "consider for future versions". Instead, this should be "must have in next v2". Frankly, this omission will probably make my enterprise wait to migrate from 3.5.x to 4.x until this refuse bad markup feature is in your new source editor.

        It's crazy to strip out bad markup in any XML-oriented editing environment. CRAZY. WRONG.

        What any XML-oriented environment should do is:

        1. Validate what you attempt to check in against an XSD or DTD.
        2. If your stuff won't validate, then refuse to check it in and certainly do NOT modify what was attempted to be checked in.
        3. Display informative validation error messages including the line numbers that contain the errors.

        If your V1 editor does not do all three of these things, then it is a nice stepping stone but it is not production ready. I mean, really. Do you really need us to spell out exactly why a V1 editor that can potentially destroy work you've done is NOT production-ready?

        Again, the Arsenale guys figured this out. Why is it so hard for Atlassian? If you need a shortcut, buy their IP and incorporate what they've done into your core product code.

        1. Well, here. I'll do Atlassian (and all your users) a favor and explain exactly why your V1 will probably be completely unacceptable for production use. This is a copy/paste of one section from my enterprise's own internal "green light" checklist for migration to Confluence 4.x. Please read it closely and think carefully about what you're planning to release in V1. I feel you're going to really piss off a lot of users if you release 
          V1 prematurely.

          Advanced editor for underlying storage format

          Atlassian has provided an Advanced Editor plugin that will enable us to directly modify the underlying XML storage format of a page, to fix/work around problems we cannot solve in the new 4.x rich text editor (RTE).

          Ideally this advanced editor should validate against their proprietary XSD and should refuse bad markup (providing linenumbers and validation error messages) instead of stripping and autocorrecting bad markup. However, V1 of this advanced editor plugin will probably not provide validation errors and refuse bad markup; it will only strip bad markup.

          Important: V1 of the editor will probably not be acceptable; we'll have to test it thoroughly to see how badly you can paint yourself into a corner and accidentally destroy preexisting work that had nothing to do with the reason you opened the advanced editor in the first place. We will probably need to wait for a V2 or later that will not destructively modify a page that you attempt to save.

           

  15. Anonymous

    I like to add another user story  / strategic fit which I'm missing from the specification definition:

    While the above description about copy/paste talks about pasting into 3rd party editors for further processing, I frequently copy/paste from an existing Confluence page into a new (or existing) one. You may call it "code reuse", "program by sniffing code" or what-ever, anyway it's an effective way to create pages by looking at examples from other pages (maybe even on different confluence databases) created by Confluence users which are smarter than myself. "Reusing" great examples is easier than to study documentation (wink)

    So in 3.x, I used to view the Wiki markup, copy and paste into my page using Wiki markup editor.

    I tried to do the same in 4.0 (haven't tried it in 4.1 yet).  The problems I was facing:

    1) View Source does not reveil which macro was used (ok, some do meanwhile) neither which parameters were provided

    2) When I pasted into the 4.0 editor, various macros were not interpreted correctly (see related reports on the 4.0 editor feedback page). You may classify this as bug-fix requirement to the 4.0 editor, but in the meantime being able to edit the source is the only way to fix such errors.

    Being able to view the macro parameters beyond what "View Source" provides helps to add them to my pages (learning about them, and how to use them).

    The above does not add anything to the requirements, but may add to the test scenarios and to the reasons for the development of the advanced editor.

     

    1. Anonymous

      +1

      Wiki format snippet cut and paste is a valuable technique from Confluence 3.X, especially for learning, which is simply gone in 4.0. (sad)

       

       

      1. Wiki format snippet cut and paste is a valuable technique from Confluence 3.X, especially for learning, which is simply gone in 4.0.

        You can still copy/paste from the View Source option in the tools dropdown in 4.x. It allows you to see what macros are being used on the page and copy them into another page, macros, parameters and all.

    2. 1) View Source does not reveil which macro was used

      The view source in 4.x will show you which macros are used ... it's essentially the same view you get when editing a page.

      2) When I pasted into the 4.0 editor, various macros were not interpreted correctly

      That shouldn't happen, that is unless you're copy/pasting across different instances which may have different macros enabled. Please let me know with which macros you encountered this problem.

      The above does not add anything to the requirements, but may add to the test scenarios and to the reasons for the development of the advanced editor.

      Correct - but the recommended way of copying sections of a page is still the View Source option in the toolbar...users should not have to have the source editor plugin enabled in order to do that.

  16. Anonymous

    and here is one more user story (from 3.5)

    I had added a plugin from different source (I think it was table-plus from Bob Swift, but not sure). I faced the issue that the 3.5 RTE did not know about the added macro (maybe this was an error in the plugin registration?), so its parameter could not be edited in the RTE. If I recall correctly, the RTE did even escape the \\{ of the macro because it didn't understand that it was such.

    Using the Wiki Markup Editor was the only way to add this macro to my page.

    I'm not sure whether the same can happen in 4.0 ? And whether this would could be handled then through the Advanced Editor?

    1. Anonymous

      I just found a work-around in this comment:  Confluence 4.0 Editor - Customer Feedback

      referring to use the Add Wiki feature. I'd consider this as a temporary workaround.

      (oh, I cannot edit the link text anymore...  - I wanted to include "Comment" or something like this, but can't)

       

       

  17. Anonymous

    So when are we getting this?

    1. Anonymous

      (A Different Anonymous). Ditto - what's the ETA on this, please? We would like to install 4.1.x in a test server to evaluate before committing to the loss of wiki markup, but we want to trial the XHMTL editor at the same time.

    2. Somewhere along the way, it was stated to be available before the end of Q1. (I can't remember who said this, and of course target dates are always iffy in software development.)

    3. We'll have it ready this quarter. I stated that in an earlier comment but it's clearly not visible enough so I'll update the page now.

  18. I'm concerned about your use of the terms "valid", "invalid", "validation", and - others have also mentioned this last one - "XHTML".

    XHTML?

    In the body of this page, you write:

    With the switch from wiki markup to xhtml

    as if your source format (which includes proprietary markup) is XHTML.

    Validation?

    Also from the body of this page:

    Validation ... There should be at least some basic validation (e.g. that the xml is well formed). This validation is performed when applying the changes in the source editor and returning to the Rich Text Editor.

    ...

    Other markup, that is considered invalid,

    ...

    Validation ... editor currently strips out bad markup.

    ...

    visually parse and spot invalid markup

    I am concerned that you are mixing colloquial and technical definitions of the terms "valid", "invalid", and "validation". And that you are blurring the distinction between well-formedness and validity by coining the term "basic validation".

    By "technical definitions", I am referring specifically to the definitions in the XML W3C Recommendation:

    Definition: A data object is an XML document if it is well-formed, as defined in this specification. In addition, the XML document is valid if it meets certain further constraints.

    Definition: An XML document is valid if it has an associated document type declaration and if the document complies with the constraints expressed in it.

    Not XHTML, no validation

    I am concerned that the information that you (Atlassian) have provided in the body of this page might lead readers to think that your source format is XHTML, and that the advanced editor will (in v1, "1: Must have") validate that XHTML (according to the technical definition of this term).

    My understanding is that neither is true. I would be happy to be wrong in both cases.

    Questions

    I am hoping for single-word, "yes" or "no" answers from you. (I think the answer to both questions is "no".)

    1. Is the Confluence storage format XHTML?
    2. Will the Confluence advanced editor validate the XML source (according to the XML definition of the term "validate")?

    (If your answer to question 1 is "yes", please cite the specific type: for example, XHTML 1.0 Strict?)

  19. I miss the dynamic Reveal Codes/WYSIWYG "split window" view of WordPerfect, where you can edit content in a Reveal Codes view - effectively, a markup view - occupying the bottom half of the application window, and immediately see the effect in the WYSIWYG view in the top half of the application window, without having to hide (switch between) one view to see the other. A similar "XML source"/"preview" arrangement (with both views visible at the same time) would be nice to have in the new Confluence editor.

    Actually, "preview" is probably the wrong term, because, ideally, I would like that "preview" to be editable. So, ideally, I guess I really mean (to match WordPerfect) a split "XML source"/"rich text" editor arrangement. But a split "XML source"/"non-editable preview" arrangement (where the preview dynamically updates as you edit the XML source) would certainly be nice to have.

    1. I suggested this same split window mechanic elsewhere too. (Probably somewhere over in the 4.0 Feedback thread.) Anyone who has worked extensively in XML environments and used high-quality XML editors will be aware that such features are possible and extremely useful because we've seen it done in the high-quality SGML/XML editor tools before.

      Remember, Atlassian, you made the decision to make Confluence an XML environment. Part of the consequences of that decision is ensuring that your users have the tools they need to work in an XML environment. This stuff is not "nice to have"; it's required for production. Your users are not casual users. They spend lots of time doing heads-down editing and revision. When you cannot see what your WYSIsortaWYG cursor is actually doing in the real underlying source code, you are flying blind.

      1. Your users are not casual users.

        This, 100 times.

      2. hate when people do "me too" comments, I'd much rather +1 a comment (Content Voting would fix that, btw (smile)), but I have to here.  I fully expect this comment to be deleted, if I don't do it myself, if I can't add anything to the conversation...but this is absolutely true.  Confluence is used by experienced users as well as laymen (but even our laymen use wiki markup).  The fact that we had wiki markup and loved it was not by accident.  There is a reason the largest wiki in the world still uses a text-based "wiki markup" language... granted its partially for technical reasons (and I give Atlassian high kudos for attempting and pretty much succeeding in making the transition to XHTML) but also its very powerful and actually lends itself very well to writing/copying documentation.  Honestly, it really needed to be phased out and not cut so fast so users could get their hands dirty until the WYSIWYG was perfect.

        I am looking forward to the editor.  As much as you may have thought users would never want to work directly with XML, I do it every day working with various tools and often times prefer the "Code View" over the "Design" view.

        1. hate when people do "me too" comments, I'd much rather +1 a comment (Content Voting would fix that, btw (smile))

          There might be something like that coming in 4.2. Maybe (wink)

    2. Live, editable preview does sound nice but, just to set expectations, it won't be in v1. We want to get a beta out very soon with the core features. We'll also open-source the plugin so that you can fork and add features yourself.

  20. Guys,

    whatever you do, do not do anything that will require anything more complicated than a plain old HTML <textarea>.

    It will be a huge maintenance burden and a pain to use if it’s not 100% perfect, and we all know it never will.

    If you want to give users syntax highlighting for XML/HTML/XHTML source, do it in a way that does not require implementing a full-blown text editor in Javascript and will benefit users of other web-based applications as well — 1) have HTML5 extended with a syntax highlighting specification for textarea and pre/code, 2) contribute an XML/HTML/XHTML syntax highlighter to Firefox and Chrome, 3) use that in Confluence/JIRA/whatever.

    Or, if you are not prepared to do that, make it just plain old textarea without syntax highlighting. Those who need highlighting can copy-paste the markup into an external editor, do the necessary changes, and copy-paste new text back. Advanced users will just use a browser extension, such as It’s All Text, that adds an external editor to all textareas.

  21. Anonymous

    Will this Plugin go into the OnDemand system as a matter of course or will someone have to request it do you know?

    1. Hi Anonymous,

      Having the plugin installed on OnDemand is not something we've planned for immediately.

      Thanks,
      John 

  22. Bill, John: When will the next version be ready? Not trying to be harsh, but this "beta" version is not ready for even testing, let alone production use. I've already tried to help you guys understand why stripping bad markup instead of simply refusing bad markup and displaying validation messages to help find and correct the bad markup is a very bad idea. In response: crickets.

    And regardless of whether you think that's an arguable point or not, without search and replace within the editor, again, this is simply not even worth testing let alone trying to use in a production context.

    I'd really appreciate a rough estimate on when you think you'll have the remaining features ready.

    1. Hi Shannon, we probably miscommunicated how the source editor does validation. I've updated the page to clarify how it works and will summaries here: It does do validation for bad markup and won't let you post malformed XHTML. So if you forget to close a tag it won't let you post and will move the cursor to the place where the error occurred. Also, if you define a macro parameter incorrectly, it simply get ignored by Confluence - Note this is exactly how the wiki markup editor worked so folks should be familiar with this behaviour. The only place the stripping occurs is if you paste object parameters in a rich text field outside of their intended place. So if you paste  <ac:parameter ac:name="bacon">cheese</ac:parameter> outside of a macro tag, it will replace the </ac:parameter> tag with a <p> because it doesn't expect to see the parameter there. Most wiki markup editor users would expect it to behave like the wiki markup editor and post the entire bit as text. We'll see if that's something we can address in future but don't have an exact date for it. Track that issue here SOURCE-50 - Source editor screen is blank with cursor flashing at line 1 Closed

      We had a search/replace feature in an early version but it was really poor quality so we deferred on it in order to get the beta out so we can get feedback ASAP. We'll see what the priorities are once the feedback is in. We hope you download the beta and give us your feedback...we've gotten some good feedback thus far.

      1. Thanks for the elaboration, Bill. Alright, that stripping behavior is less egregious than I'd feared, but still not really what I expect from an XML editor. Moving the cursor to the point of ONE error is far less useful than either REPORTING all line numbers where errors occurred and the nature of the error, or at least doing so for the FIRST such error encountered.

        But more importantly, you have not yet adequately documented the language reference. So why should I bother to test anything? Please see my newest comment at the bottom of the Confluence Storage Format page.

  23. Anonymous

    This seems to be missing my user story.

    I would like to be able to extract the contents of the page as wiki-markup so that my script which updates information on the page can modify the page, allowing me to paste the updated page back into Confluence.

    1. I'm afraid your user story doesn't have a happy ending, Anonymous. See

      Apparently we can have anything we want (big grin) except for wiki markup.

       

  24. Please install the beta plugin in the Confluence 4.x sandbox.

  25. When is the ability to search for underlying storage format syntax across all pages in a space (or across the entire confluence instance) in the global search control AND in any {pagetreesearch} search control coming?

    1. This is already working with webdav: Just map a drive via webdav, then use an editor (like notepad++) to make a "find in files/subdirectories and replace" action. I'm using this very often it works very well (wink)

      1. Can you please elaborate or point to specific Confluence doc pages that can do so? I have no idea what you're talking about with "just map a drive via webdav" nor how using Notepad++ has any bearing on editing a Confluence page or using a global Confluence instance search.

        1. I'm going to rain on the parade...but mainly with the hope of getting some love for one of my bugs. The procedure Sven mentioned will not work reliably across an entire space because pages that use fancy characters in their titles are invisible to WebDAV (see CONF-24077 - Page names with special characters break WebDAV clients Open ). Even if you're not using 👽 Unicode Space Aliens 👽 in your page titles, this is still a concern because common characters (like "/" or  "\") also provoke this behavior.

          (Shannon, WebDAV allows you to view your Confluence space/page tree as a filesystem. See  https://studio.plugins.atlassian.com/wiki/display/WBDV/WebDAV+Client+Setup for reference.)

          1. Thanks for the explanation, Scott. Unfortunately, we're all 64-bit Win 7 at my enterprise, and it appears that the Confluence WebDav plugin has some problems with Win7. We can't be dealing with registry changes and a lot of of fiddling and troubleshooting just to get access to a core functionality that Atlassian should be providing natively in their global search control and in {pagetreesearch} macros and the like.

            1. This is by no means a complete solution, but it might be of some interest... a copy of a comment I've just added to another page:

              Do you want to find out which pages on your Confluence site use a particular macro, or XML element, or combination of content and markup?

              For example, do you want to search all of the pages on your Confluence site for the following XML source?

              If so, then you might be interested in the "Searching Confluence storage format XML for content or markup" page in the Advanced Confluence Tips space of the Confluence, Tech Comm, Chocolate wiki. That page presents a Run macro with a nested SQL query that searches your Confluence database.

              Acknowledgment: Back in February 2010, Betsy Walker published an "Identify pages using particular string in content body" SQL query that did much the same thing.

  26. Anonymous

    using a code macro appears to crash the plugin (beta 1).  when inserted into the editor, it will hang after clicking the apply button.  when editing a page where the macro already exists, the <> button does not appear at all.  not sure if its the code macro itself, or the text I'm using inside it, this is what shows up when I retrieve the source format through the tools button:

     

     

    1. Anonymous - this is working fine for me in beta 2 (though I did need to remove the whitespace in the ac:parameter to get it to render.

      Can you update to beta 2, and see if the behaviour is any different?

  27. "With the switch from wiki markup to the new XHTML-based storage format..."

    Where is the discussion/documentation on the role and future of wiki markup? Are you saying here that wiki markup has essentially been replaced with XHTML? If so, where is this switch discussed?

    1. Hallo Brian

      The discussion is happening on this page: Confluence 4.0 Editor - Customer Feedback.

      Cheers, Sarah

       

  28. Atlassian,

    Regarding validation of XML in the advanced editor (Source Editor plugin): you might be interested in looking at the hurriedly cobbled together JavaScript in Wikifier (see my recent announcement on the parent "Confluence 4.0 Editor - Customer feedback" page), which uses the browser's (IE9, or current production version of Chrome, Firefox, or Safari) built-in schema-based validation to provide more than just the completeness checking that is currently provided by the advanced editor.

    If I were going to further develop Wikifier (probably not), I would write more code to further "parse the (XML) parsing errors", and so offer users better feedback about XML errors, without requiring any server-side processing. In case it's an issue: I would personally have no qualms specifying different (newer/later) minimum browser levels for the advanced editor than the rich text editor (but I'd want to check the browser level in code, and either gracefully "degrade" the experience - for example, no schema-based validation - or require users to upgrade).

    I realize that this suggestion contradicts another suggestion I've just made elsewhere (wink) :

    Rather than investing more development effort right now in the Source Editor plugin, spend some time making the data more accessible to external XML tools and methods.

     

  29. How can I copy/past from one wiki page into another one using only the RTE in Confluence On Demand? When I copy/paste content, all the formatting in the source is lost. My intent was to retain the formatting. If this is not possible, are there plans to implement this? I looked at the confluence Jira tickets on copy/paste and could not find this specific issue. If there is a ticket already, can you point me to the issue? Thanks.

    Sorry for bringing this up if this has already been covered.

     

    1. If you're using the RTE to copy and pasting between two pages in the same instance, then the formatting should just carry over. If for some reason you don't have access to edit the page you want to copy (i.e. can't get to the RTE), then you can go to the "view source" option in the tools menu and copy from there. Please let me know if that doesn't work for you.

      1. Hm. I created a new page (for testing) containing just this text/formatting (storage format shown here):

        <h1>Hello</h1>
        <ul>
        <li>Copy this</li></ul>
        <h1>World</h1>
        <ul>
        <li>and this</li></ul>

        Then, I saved the page. Re-editing the page, I select all then paste at the bottom of the page and get this (bold portion results from paste):

        <h1>Hello</h1>
        <ul>
        <li>Copy this</li></ul>
        <h1>World</h1>
        <ul>
        <li>and this</li></ul>
        <p>Hello</p>
        <p>&nbsp;</p>
        <p>&nbsp;&nbsp;&nbsp; Copy this</p>
        <p>&nbsp;</p>
        <p>World</p>
        <p>&nbsp;</p>
        <p>&nbsp;&nbsp;&nbsp; and this</p>


        Again, this is in an instance of Confluence On Demand. Am I doing something incorrect? Does on demand confluence support copy/paste with formatting retention?

        1. Brian, does this happen on other pages too, or just this one? It might be worth checking to see if the line you are pasting into has Bold applied before you paste.

          1. Matt, it appears that all pages fail to retain formatting on paste (I tried several). The only RTE entity that appears to want to copy/paste are macros and that, only if I click on them and "copy". If they are included in a text selection the macros don't retain their macro-ness either. For example, the attachments macro pastes just as the word 'attachments' when copy/pasted using text selection. Bold formatting does not appear to be part of the causality.

            I've tried copy/paste of the following, none of which retain formatting:

            • h1 heading
            • Bold text
            • 2x2 table with text in each cell
            • Numbered list
            • Bullet list
            1. Brian, it's entirely possible this is a bug with a specific browser or operating system.

              Can you confirm some details?

              • Browser version
              • Operating system and version.
              Ignore that I saw you answered Mark already.
        2. Matt, I don't think bold is the problem, it is the unwanted <p> tags and loss of existing formatting.

          Brian, when I follow your steps, also using an OnDemand instance, I get the results:

          <h1>Hello</h1>
          <ul>
          <li>copy this</li></ul>
          <h1>Hello</h1>
          <ul>
          <li>and this</li></ul>
          <h1>Hello</h1>
          <ul>
          <li>copy this</li></ul>
          <h1>Hello</h1>
          <ul>
          <li>and this</li></ul>

           

          This seems to be working as expected. What browser are you using?

           

          1. Mark, I'm using Firefox 12.0 under Windows 7.

            I've tried the paste in IE 9.0.6 and I get this on paste:

            <p>&nbsp;Hello<br />&bull;Copy this</p>
            <p>World<br />&bull;and this</p>
            <p>&nbsp;</p>
            <p>&nbsp;</p>
            <p>&nbsp;</p>

            OK. It must be something on my system. Formatting does not copy/paste in Microsoft Word either!

          2. OK, Now that I've uninstalled Skype click-to-call copy/paste works! Wow. Thanks for chiming in. Your comments got me to think about the larger context.

            1. Glad to help. I've created a Knowledge Base article to help out anyone else that comes across this problem.

               

  30. As a developer i cannot appreciate that wiki markup is not accessible anymore. If I want to document technical information the editor is not at all user friendly.

    i.e.:

    If I write a code block and after that a new h1 header there is no way to add simple content after the code block anymore unless I set my cursor before the header, type enter and then alter the empty header line to paragraph font. This is exactly one of the reasons wiki-markup should be available for exclusive issues.

    {code}
      some code
    {code}

    ??? content here afterwards ???

    h1. Some header

    I solve this issue by editing my wiki-markup in a text-editor .... and when updated I delete all content on the page, press ctrl + shift + D and paste my new markup in there. This gives me the slightest feeling of going back to the stone age.

     

    I really do understand why making it more available to the non-it personnel, but blocking this functionality just isn't the solution. Wiki-markup goes very fast for people who know how to use it. Right now you chose to be just another rich text editor. And that is not a hole in the market. It saddens me that you chose this easy path instead of making a very nice hybrid for a greater public.

    1. It saddens me that you chose this easy path instead of making a very nice hybrid for a greater public.

      You are not alone. Welcome to the multitudinous minority who share your grief. Better get over it or get out, though, because Atlassian has remained steadfast in its position despite similar feedback from many others. Unless you have been watching this and related pages for the last few months, you might not be aware of the number of negative responses, as Atlassian periodically culls comments (with some good reason: pages with many comments can become unusable due to performance issues).

    2. Hi David, you can us the CMD/CTRL+0 keyboard shortcut to instantly turn the selected row from an <h1> into a <p>. That's a lot faster than the steps you're currently undertaking.