ststeiger / wikimodel Goto Github PK

    @Test
    public void testBug1() throws IOException, WikiParserException {
        String in = "The link can also be a direct URL starting 
with {{http:}}, {{ftp:}}, {{mailto:}}, {{https:}}, or {{news:}}, in which 
case the link points to an external entity. For example, to point at the 
java.sun.com home page, use {{[[http://java.sun.com]}}, which becomes 
[http://java.sun.com/] or {{[[Java home page|http://java.sun.com]}}, which 
becomes [Java home page|http://java.sun.com].";
        IWikiParser wikiParser = new CommonWikiParser();

        IWemListener listener = new PrintListener(new 
NullPrinter());
        wikiParser.parse(new StringReader(in), listener);
    }

    class NullPrinter implements IWikiPrinter {

        public void print(String str) {
            // TODO Auto-generated method stub

        }

        public void println(String str) {
            // TODO Auto-generated method stub

        }

    }

Input:

<p>This is &lt;a href=&quot;url&quot;&gt;some HTML&lt;/a&gt;.</p>

Expected events:

beginDocument
beginParagraph
onWord: [This]
onSpace
onWord: [is]
onSpace
onSpecialSymbol: [<]
onWord: [a]
onSpace
onWord: [href]
onSpecialSymbol: [=]
onSpecialSymbol: ["]
onWord: [url]
onSpecialSymbol: ["]
onSpecialSymbol: [>]
onWord: [some]
onSpace
onWord: [HTML]
onSpecialSymbol: [<]
onSpecialSymbol: [/]
onWord: [a]
onSpecialSymbol: [>]
onSpecialSymbol: [.]
endParagraph
endDocument

Got:

beginDocument
beginParagraph
onWord: [This]
onSpace
onWord: [is]
onSpace
onSpecialSymbol: [<]
onWord: [a]
onSpace
onWord: [href]
onSpecialSymbol: [=]
onWord: ["url"]
onSpecialSymbol: [>]
onWord: [some]
onSpace
onWord: [HTML]
onSpecialSymbol: [</]
onWord: [a]
onSpecialSymbol: [>.]
endParagraph
endDocument

Hi,

I'm passing the following to the xhtml parser:

<p>hello <em class="italic">beautiful</em> <strong>world</strong></p>

And I get the following events (XWiki events but they're mapped to
wikimodel events):

beginParagraph
onSpace
onSpace
beginItalic
onWord: [beautiful]
endItalic
onSpace
beginBold
onWord: [world]
endBold
endParagraph

It should be:

beginParagraph
onWord: [hello]
onSpace
beginItalic
onWord: [beautiful]
endItalic
onSpace
beginBold
onWord: [world]
endBold
endParagraph

As you can see the "hello" world has disappeared.

Any idea?

Thanks
-Vincent

For example having this as an input source:

{nowiki}
* this is not a list and this is not *bold*
[not a link]
\\ <-- not a newline
{/nowiki}

Generates an onMacro event with the following content:

* this is not a list and this is not *bold*
[not a link]
 <-- not a newline

(note the missing \\)

I can convert my existing wiki pages to XHTML, but then I cannot use 
WikiModel (yet) - I need a converter to CommonWikiSyntax.

Storing content as XHTML is also more convenient than storing as wiki 
syntax.

In XWiki land we've decided to have a syntax for tables rather than a table
macro.

PrintListener.onMacroBlock does not xmlEncode name and params, this causes 
ill-formed XML from some wiki input.

What steps will reproduce the problem?
Create a "\n**bold" sequence and try to parse it using the CommonWikiParser

What is the expected output? What do you see instead?
The sequence above is recognized as a list of the second level, and not as
a strong formatted text in a paragraph.

All sequences "\n** xxx" should be recognized as item markups and
"\n**strong" as a bold formatting  switchers.

The wiki text below produces the XHTML underneath when parsed by
MediaWikiParser and serialized by the default Printlistener. Not sure about
how the reference should look like when it contains another reference, but
the reference identified by the parser looks wrong in any case, doesn't it?

Wiki text:

[[Image:Jerry Yang and David Filo.jpg|thumb|right|250px|Yahoo! co-founders
[[Jerry Yang (entrepreneur)|Jerry Yang]] (left) and [[David Filo]] (right)]]

Resulting XHTML:

<a href='[Image:Jerr'>[Image:Jerr</a> Yang and David
Filo.jpg|thumb|right|250px|Yahoo! co-founders <a href='Jerry Yang
(entrepreneur)|Jerry Yang'>Jerry Yang (entrepreneur)|Jerry Yang</a> (left)
and <a href='David Filo'>David Filo</a> (right)]]

Stéphane

The grammar for lists is wrong.  Wiki Creole specifies lists like:
{{{
* item
** nested item
* next item
}}}
With leading whitespace being ignored.

It looks like your current Creole grammar requires at least one initial
blank and nesting is done like this:
{{{
 * item
  * nested item
 * next item
}}}

See http://wikicreole.org/wiki/Creole1.0

## for a nested ordered list item is incorrectly interpreted as monospace
which currently isn't even in the Creole 1.0 spec

See http://wikicreole.org/wiki/Creole1.0

Hi,

On the xwiki lists we have voted to change the quotation symbol from space
(the current wikimodel symbol for the xwiki parser) to ">".

Could this be modified please?

Vote thread: 
http://xwiki.markmail.org/search/?q=[Rendering]%20Wiki%20syntax%20for%20quotes%2
0and%20block%20quote%3F#query:[Rendering]%20Wiki%20syntax%20for%20quotes%20and%2
0block%20quote%3F+page:1+mid:pl3vtrg2l2wbkqqh+state:results

Thanks

The following content isn't parsed as a list but as some bold content by
wikimodel:

(space)(space)* item1
(space)(space)* item2

Input <html><a href="url" ref="_blank">link</a>
The output of the parser only handle the href and label, but miss the rel.

This because in XhtmlHander, the code about <a> only handle the href and
label. Even though in WikiReference, there is 
private final WikiParameters fParameters to store the rel or target
information, but this is never used in parser and ReferenceHandler.

Current version, I cannot find the "image" implementation in creole wiki,
in the parsers and tests.

Creole image label is: {{myimag.png|This is my image alt text}}
ref to:  http://www.wikicreole.org/wiki/Creole1.0#section-Creole1.0-ImageInline

thanks.

Hi,

In XWiki land we have voted to use {{{ }}} (same as creole) for escaping a
portion of text.

See http://tinyurl.com/6l5mxx

Thanks

Google search for "wikimodel" currently shows the WikiModel SourceForge
project page as first result. I assume, SourceForge is the old home of
WikiModel and the new one is Google Code.

I suggest to put a noticeable comment on the SourceForge project page,
which states that the project has moved to Google Code. This might avoid a
lot of confusion.

The XhtmlHandler does not track what type of list it is in (i.e. ordered vs
unordered) so it always uses '*' for list items.  It needs to be changed to
use '*' for unordered lists and '#' for ordered lists.  XhtmlHandler should
maintain a stack of list types to track which character to use for list items.

I parsed the following XML with XhtmlParser:
<table><tbody>
  <tr><th>header col1</th><th>header col2</th></tr>
  <tr><td>col1</td><td>col2</td></tr>
  <tr><td>you         </td><td>can         </td></tr>
  <tr><td>also        </td><td>align<br /> it. </td></tr>
</tbody></table>

When the iWemListener gets called at beginTableCell, it should get passed
true for the boolean for the <th> elements.  This parameter is always false.

It's really hard to guess what events in IWemListener correspond to and
there's not a single line of javadoc...

Right now if I enter "**bold**" wikimodel thinks it's a list with a content
of "*bold**" whereas it should be recognized as a bold element.

If I send the following:

<a href="/xwiki/bin/view/Space/ExistingPage">Space.ExistingPage</a>

the onReference(String) method is passed
"/xwiki/bin/view/Space/ExistingPage Space.ExistingPage" which looks
strange: it has both the URL and the label separated by a space.

Since you're introducing a new WikiReference class I guess in the future
"/xwiki/bin/view/Space/ExistingPage" will be set as the link value and
"Space.ExistingPage" as the label, right?

Thanks

WikiEntityUtil translates the "'" (apostrophe) character to the entity
"’".  This entity is not recognized by the SAX parser if you feed it
the HTML generated by WikiModel.  I believe that WikiEntityUtil should be
changed to map this character to "'" which the SAX parser handles
correctly.

With the latest code, I get 2 test failures when I try to run mvn install
from the org.wikimodel.wem directory.  If I am interpreting the output
correctly the failures occur when running
org.wikimodel.wem.test.CreoleWikiParserTest and
org.wikimodel.wem.test.CommonWikiParserTest

If there's something I can do to provide more meaningful output, please let
me know.

I've attached the output from the Maven build.

See http://dev.xwiki.org/xwiki/bin/view/Drafts/XWikiSyntax20#HHorizontalLine

Patch:
Replace the following in XWikiScanner.jj:

    | <#HORLINE: "---" ("-")+ >

by

    | <#HORLINE: "----" ("-")+ >

What steps will reproduce the problem?
1. Parse a text containing ".... foo:bar ..."

What is the expected output? What do you see instead?
I would expect no interpretation as wiki syntax. But it becomes a link.

Input:

<?xml version="1.0" encoding="UTF-8"?>
<html><head /><body>one two three</body></html>

Output:

beginDocument
beginParagraph
onWord: [one]
onSpace
onWord: [t]
onWord: [wo]
onSpace
onWord: [three]
endParagraph
endDocument

Expected:

beginDocument
beginParagraph
onWord: [one]
onSpace
onWord: [two]
onSpace
onWord: [three]
endParagraph
endDocument

If I remove the <xml> declaration it works just fine.

Thanks

Input:

<html><p>this is one<br/>paragraph</p><p>this is another one</p></html>

Expected events:

beginDocument
beginParagraph
onWord: [this]
onSpace
onWord: [is]
onSpace
onWord: [one]
onLineBreak
onWord: [paragraph]
endParagraph
onEmptyLines: [1]
beginParagraph
onWord: [this]
onSpace
onWord: [is]
onSpace
onWord: [another]
onSpace
onWord: [one]
endParagraph
endDocument

Got:

beginDocument
beginParagraph
onWord: [this]
onSpace
onWord: [is]
onSpace
onWord: [one]
onLineBreak
onWord: [paragraph]
endParagraph
beginParagraph
onWord: [this]
onSpace
onWord: [is]
onSpace
onWord: [another]
onSpace
onWord: [one]
endParagraph
endDocument

As you can see it's missing the onEmptyLines event.

For example my use case is to transform the following URL:

<a href="/xwiki/bin/view/Main/WebHome" rel="target">...</a>

Right now I don't the "rel" attribute passed as a parameter inside
WikiReference.

I attempt to parse the following html with the XhtmlParser:
<ul>
  <li>item a</li>
  <li>item b</li>
  <li>item c</li>
  <ul>
      <li>item c.a</li>
  </ul>
</ul>

I would expect the listener calls to be something like:
beginList();
beginListItem(); endListItem();
beginListItem(); endListItem();
beginListItem(); endListItem();
beginList(); // start of nested list
beginListItem(); endListItem();
endList();   // end of nested list
endList();

I do not see the nested beginList() and endList() calls.  I do not know if
this problem is isolated to the XhtmlParser, or if it is more of a general
problem with the way wem handles nested lists.  

Once I get the updated jars (either built or sent to me) that include fixes
for handling the Creole markup for nested lists, I can test nesting from
that direction.  For example in Creole, the above list should be:

* item a
* item b
* item c
** item c.a

If you have:

{nowiki}
* this is not a list and this is not *bold*
[not a link]
\\ <-- not a newline
{/nowiki}
{nowiki}*{/nowiki}

wikimodel doesn't generate a line break event between the 2 on Macro events.

Right now wikimodel doesn't parse the content of links (references). It's a
pity since it makes using the different wiki syntaxes hard.

For example xwiki's links can be [label>link] or [label|link].

The link syntax is part of the wiki's syntax so it would make sense to me
if wikimodel also supported parsing the links.

Right now we're going to have to write 4 new link parsers in the xwiki
project in order to support all syntaxes (wiki, creole, jspwiki, mediawiki).

Thanks
-Vincent

The XWiki project has voted a change for its XWiki Syntax v2.0 for bold.
The new syntax is 2 stars.

For example: This is **bold**.

Wikimodel should be modified to reflect this.

Thanks

input: <del>strikedout</del>

The del tags are ignored and it simply generates an on word event

There is no TagHandler added for the <br /> element, so the
WikiScannerContext is never called at onLineBreak() when the html contains
this element.  I added a handler in my copy of XhtmlHandler right after the
handler for <hr />, and it seems to fix the problem.

I use XhtmlParser + EventDumpListener. 
the input is 
"<html><body><img src=\"http://xwiki.org/logo.png\" /></body></html>"

the output is 
beginDocument()
endDocument()

There is nothing about <img> event.

What steps will reproduce the problem?
To test inline nowiki, I feed the parser the following lines

The following line will test inline nowiki
before the nowiki {{{ during the nowiki }}} after the nowiki
done

What is the expected output? What do you see instead?
When this gets rendered to HTML by the PrintListener, it looks like:

<div class='doc'>
<p>The following line will test inline nowiki
before the nowiki <code>{{{ during the nowiki }}}</code> after the nowiki
done </p>

The nowiki markup strings of "{{{" and "}}}" should not show up in the
rendered HTML.  The output should look like:

<div class='doc'>
<p>The following line will test inline nowiki
before the nowiki <code> during the nowiki </code> after the nowiki
done </p>

What version of the product are you using? On what operating system?
wikimodel-0.1.jar on Windows XP.


Please provide any additional information below.

links like:
{{{
[[link]]
[[link|linkText}}
}}}
aren't working properly

If you enter this input:

This is ***** bold

You get this output:

beginDocument
beginParagraph
onWord: [This]
onSpace
onWord: [is]
onSpace
onSpecialSymbol: [*]
onSpace
onWord: [bold]
endParagraph
endDocument

This doesn't look correct since reserializing this in wiki format will
generate: "This is * bold"

I would have expected either:
- an empty beginBold/endBold block followed by a special symbol (*)
- 3 special symbols (*) followed by a bold block

Thanks

For this input I get onEmptyLines(1):

-------------
this is one
paragraph

this is another one
-------------

However for this one I get only onEmptyLines(2):

-------------
{{macro/}}



{{macro/}}
-------------

Some more examples:

-------------
{{macro/}}



This is a para
-------------

generates onEmptyLines(3)

And

-------------
This is a para



{{macro/}}
-------------

generates onEmptyLines(2)

Hi,

I have several problems with the XHMTL parser and lists. Take the following
input:

<html><ol><li>Item 1<ol><li>Item 2<ul class="star"><li>Item
3</li></ul></li><li>Item 4</li></ol></li><li>Item 5</li></ol><ul
class="star"><li>Item 1<ul class="star"><li>Item 2<ul class="star"><li>Item
3</li></ul></li><li>Item 4</li></ul></li><li>Item 5</li><li>Item
6</li></ul></html>

It generates the following events:

beginList: [BULLETED]
beginListItem
onSpace
onWord: [ ]
onWord: [1]
beginList: [BULLETED]
beginListItem
onSpace
onWord: [ ]
onWord: [2]
beginList: [BULLETED]
beginListItem
onSpace
onWord: [ ]
onWord: [3]
endListItem
endList: [BULLETED]
endListItem
beginListItem
onSpace
onWord: [ ]
onWord: [4]
endListItem
endList: [BULLETED]
endListItem
beginListItem
onSpace
onWord: [ ]
onWord: [5]
endListItem
endList: [BULLETED]
beginList: [BULLETED]
beginListItem
onSpace
onWord: [ ]
onWord: [1]
beginList: [BULLETED]
beginListItem
onSpace
onWord: [ ]
onWord: [2]
beginList: [BULLETED]
beginListItem
onSpace
onWord: [ ]
onWord: [3]
endListItem
endList: [BULLETED]
endListItem
beginListItem
onSpace
onWord: [ ]
onWord: [4]
endListItem
endList: [BULLETED]
endListItem
beginListItem
onSpace
onWord: [ ]
onWord: [5]
endListItem
beginListItem
onSpace
onWord: [ ]
onWord: [6]
endListItem
endList: [BULLETED]

However this is not correct. It should be:

beginList: [NUMBERED]
beginListItem
onWord: [Item]
onSpace
onWord: [1]
beginList: [NUMBERED]
beginListItem
onWord: [Item]
onSpace
onWord: [2]
beginList: [BULLETED]
beginListItem
onWord: [Item]
onSpace
onWord: [3]
endListItem
endList: [BULLETED]
endListItem
beginListItem
onWord: [Item]
onSpace
onWord: [4]
endListItem
endList: [NUMBERED]
endListItem
beginListItem
onWord: [Item]
onSpace
onWord: [5]
endListItem
endList: [NUMBERED]
beginList: [BULLETED]
beginListItem
onWord: [Item]
onSpace
onWord: [1]
beginList: [BULLETED]
beginListItem
onWord: [Item]
onSpace
onWord: [2]
beginList: [BULLETED]
beginListItem
onWord: [Item]
onSpace
onWord: [3]
endListItem
endList: [BULLETED]
endListItem
beginListItem
onWord: [Item]
onSpace
onWord: [4]
endListItem
endList: [BULLETED]
endListItem
beginListItem
onWord: [Item]
onSpace
onWord: [5]
endListItem
beginListItem
onWord: [Item]
onSpace
onWord: [6]
endListItem
endList: [BULLETED]

There are 2 problems:
* the "Item" word is swallowed
* it doesn't support ordered lists

Thanks

Currently there's a problem since the xwiki parser in wikimodel recognizes
{table} but since we've now doubled the curly braces for macro syntax the
new {{table}} macro isn't recognized.

On a related topic we're rewriting the table macro so there'll probably be
some differences from what exists now. I'll update this issue when we've
progressed more.

For example XWiki generates the following XHTML for escapes:

<html><p><pre><![CDATA[*]]></pre><pre><![CDATA[*]]></pre>hello<pre><![CDATA[*]]>
</pre><pre><![CDATA[*]]></pre></p></html>

This should generate the following events:

beginDocument
beginParagraph
onEscape: [*]
onEscape: [*]
onWord: [hello]
onEscape: [*]
onEscape: [*]
endParagraph
endDocument

We can discuss the XHTML syntax we want to use.

Thanks

Right now the following inputs generate the same events:

1Title
1 Title

I'm proposing to force requiring at least one space before the header's
text so that 1Title doesn't generate a header.

Input:

<html><p><a class="wikicreatelink"
href="/xwiki/bin/view/Space/UnknownPage"><span
class="wikicreatelinktext">label</span><span
class="wikicreatelinkqm">?</span></a></p></html>

Output:

beginDocument
beginParagraph
onWord: [label]
onSpecialSymbol: [?]
endParagraph
endDocument

As you see it's missing the onReference event.

For example the XHTMLParser should understand:

<span style="font-weight: bold;">foo</span>

See
http://groups.google.com/group/wikimodel-users/browse_thread/thread/57dfa59ee40d
6e65

Steps to reproduce the problem: parse the following syntax with the XWiki
parser: "before{code}\n1 Not a header\n\n* Not a list{code}after text text
text{code} again{code}"

Only one big "{code}" section will be identified, instead of two as expcted.

The XWiki project has voted a change for its XWiki Syntax v2.0 for links.
The new syntax is 2 brackets.

For example: This is a [[links]].

Wikimodel should be modified to reflect this.

Thanks

It doesn't suport the beginFormat() event at all actually.

Wikimodel should support underline. It's pretty common in wikis, see
http://www.wikimatrix.org/syntax.php?i=25

In addition right now in XWikiScanner.jj there's:

            t = getFORMAT_SYMBOL()
            {
                str = t.image.trim();
                WikiStyle style = null;
                if ("__".equals(str) || "**".equals(str)) {
                    style  = IWikiScannerContext.STRONG;

This is not correct since "__" is used for underline in XWiki and not for Bold.

Thanks a lot
-Vincent