nricciar / wikicloth

Ruby implementation of the MediaWiki markup language.

WikiCloth

Ruby implementation of the MediaWiki markup language.

Supports

• Variables, Templates {{ … }}
• Links
  • External Links [ … ]
  • Internal Links, Images [[ … ]]
• Markup
  • == Headings ==
  • Lists (*#;:)
  • bold ('''), italic ('') or both (''''')
  • Horizontal rule (——)
  • Tables
  • Table of Contents [__NOTOC__, __FORCETOC__, __TOC__]
• <code>,<nowiki>,<pre> (disable wiki markup)
• <ref> and <references/> support
• html sanitization

For more information about the MediaWiki markup see https://www.mediawiki.org/wiki/Markup_spec

Install

  git clone git://github.com/nricciar/wikicloth.git
  cd wikicloth/
  rake install

Usage

  @wiki = WikiCloth::Parser.new({
    :data => "<nowiki>{{test}}</nowiki> ''Hello {{test}}!''\n",
    :params => { "test" => "World" } })

  @wiki.to_html => "<p>&#123;&#123;test&#125;&#125;  <i>Hello World!</i></p>"

Advanced Usage

Most features of WikiCloth can be overriden as needed…

  class WikiParser < WikiCloth::Parser

    url_for do |page|
      "javascript:alert('You clicked on: #{page}');"
    end

    link_attributes_for do |page|
      { :href => url_for(page) }
    end

    template do |template|
      "Hello {{{1}}}" if template == "hello"
    end

    external_link do |url,text|
      "<a href=\"#{url}\" target=\"_blank\" class=\"exlink\">#{text.blank? ? url : text}</a>"
    end

  end

  @wiki = WikiParser.new({ 
    :params => { "PAGENAME" => "Testing123" }, 
    :data => "{{hello|world}} From {{ PAGENAME }} -- [www.google.com]"; 
  })

  @wiki.to_html =>
  <p>
  Hello world From Testing123 -- <a href="http://www.google.com" target="_blank" class="exlink">http://www.google.com</a>
  </p>

wikicloth's Issues

Invalid characters frequently in Wikipedia documents break parsing

Wikipedia articles often have '[' characters in {{Cite}} template params (mostly the "title" param). Wikipedia replaces ']' with ']', as it should, but for some reason does not apply this transformation to the left brace. This results in invalid syntax, but consistently invalid syntax across the entire of Wikipedia.

For example, see the "title" param in this citation from the current revision of the Wikipedia article:

{{cite web
|url=http://lists.wikimedia.org/pipermail/wikitech-l/2005-April/016297.html
|title=[Wikitech-l] Lucene search|author=Brion Vibber|accessdate=2009-02-26}}

I am doing some work on the latest dump of Wikipedia (Sept 2010) and this occurs on many large articles. Other examples are the World War II and Vietnam War articles, though these references have been removed in the latest revisions, by pure chance it seems.

I am not sure if it is a goal of wikicloth to match MediaWiki as a reference implementation, but if so then this case should be handled.

I am trying and failing to do a fix myself.

Dependency on rinku breaks JRuby-compatibility

WikiCloth is no longer compatible with JRuby after adding rinku as a dependency in this commit. Is this something you would consider fixing?

First item in list not created when list is preceded by a heading

When a heading immediately precedes a list, the first element in the list is not rendered correctly. If there are two newlines after the heading, the error does not occur.

Input
=Heading=
* One
* Two
* Three

Expected Output

[edit] Heading

• One
  
• Two
  
• Three
  

  Output
  

  
  

  [edit] Heading

  * One
  
  • Two
    
  • Three
    

Preformatting fails for empty lines

If one of the lines in a preformatted block is just a space (or a series of spaces), the previous part of the block is not rendered. Only happens once per block.

Example:

Some text

 code line
 another line
 
 some code
 some more code

 more code after a space

More text

Is rendered as:

Some text

 some code
 some more code
 
 more code after a space

More text

templates

Probably an easy question but I couldn't find an example in the docs, how can I load a series of custom templates? tried:

template do |template|
    "<references/>" if template == "reflist"
    "<a href='{{{url}}}'>\"{{{title}}}\"</a> <i>{{{publisher}}}.</i> {{{date}}} {{{accessdate}}}" if template == "cite web"
end

it only applies the last one. I would like to create a few.

parsing errors on rezeptewiki.org

I am trying to parse the data from rezeptewiki, a german wiki for recipes. I found two errors that crash the parser on 43 documents of that wiki.

Template in table definition

Unknown error on line 11 row 4: WikiCloth::WikiBuffer-->width="100%"-->border="2" cellspacing="1" cellpadding="2" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #d6d6d6; border-collapse:collapse; background-color:#fffbf5; empty-cells:show; width="500px"
DAMAGE : 16452 - Zubereitung:Saisonkalender Obst
undefined method `[]' for nil:NilClass
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth/wiki_buffer/table.rb:108:in `new_char'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth/wiki_buffer.rb:158:in `add_char'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth/wiki_buffer.rb:159:in `add_char'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:138:in `add_current_char'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:86:in `block in render'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:86:in `each_char'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:86:in `render'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:107:in `to_html'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth/parser.rb:89:in `method_missing'

This can be found in this document: http://www.rezeptewiki.org/wiki/Zutat:Nova

Problems with sections

I don't know why, but on a few documents the section system gets messed up.

undefined method `<<' for nil:NilClass
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:44:in `block in load'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:39:in `each_line'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:39:in `load'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth.rb:28:in `initialize'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth/parser.rb:14:in `new'
/Users/bodo/.rvm/gems/ruby-1.9.3-p125/gems/wikicloth-0.8.0/lib/wikicloth/parser.rb:14:in `initialize'

This can be found in this document: http://www.rezeptewiki.org/wiki/Gebackene_Innviertler_Zwetschgenkn%C3%B6del

Documentation

This really need some.

Extreme slowness with a large page

I'm using wikicloth as part of Gollum and a page is taking a really long time to generate, as in, 16+ seconds:
gollum/gollum#680

The short version of that bug report is that there appears to be some exponential algorithm used, such that page generate times are fine (~2 sec) for half the page, but then increase sharply to the full page, where they are 16+ seconds.

I have wikicloth installed locally from a gem, but I don't know enough ruby to make a simple file to use wikicloth to convert a file, or to profile it, which would be the logical next step to fix this.

Do you have any suggestions? The wikicloth output looks great, but it's too slow to be usable. Thanks!

Table Spanning Templates Causes Problems

If you have three templates, say {{{table}}} = "

", {{{tablemid}}} = "" and {{{tableend}}} = "

test

" and you include these in a page right in a row like "RAWR {{{table}}}{{{tablemid}}}{{{tablemid}}}{{tableend}}} something" the WikiParser returns only "RAWR".

remove unnecessary/unused files

e.g. uninstall.rb and install.rb, are not used and should be removed.

Buffer chokes on multibyte characters

This wiki page on GitHub causes wikicloth to hang. It appears to be caused by the buffer's mishandling of multibyte strings.

The following will just hang indefinitely:

require 'wikicloth'
require 'open-uri'

url = "https://raw.githubusercontent.com/wiki/CA-CST-SII/Software-Standards/Demo:-C%23-Coding-Standards.mediawiki"

puts WikiCloth::Parser.new(:data => open(url).read).to_html

Error when running bundle install

In my gemspec
gem 'wikicloth', :git => 'git://github.com/nricciar/wikicloth.git'

when running bundle install

bundle install
Fetching git://github.com/nricciar/wikicloth.git
There was a LoadError while evaluating wikicloth.gemspec:
no such file to load -- expression_parser from
/Library/Ruby/Gems/1.8/bundler/gems/wikicloth-7f789f75cdf4/wikicloth.gemspec:2

WikiCloth's i18n module is broken

Executing WikiCloth using the supplied wikicloth/i18n does not work.
Using the i18n gem works as expected

Steps to reproduce (using default wikicloth)

irb(main):001:0> require 'wikicloth'
=> true
irb(main):002:0> WikiCloth::WikiCloth.new(:data => "__TOC__\n\n= One =\n\nwth").to_html(:noedit => true)
NoMethodError: undefined method `has_key?' for nil:NilClass
from /usr/lib/ruby/vendor_ruby/wikicloth/i18n.rb:25:in `t'
from /usr/lib/ruby/vendor_ruby/wikicloth.rb:96:in `rescue in render'
from /usr/lib/ruby/vendor_ruby/wikicloth.rb:78:in `render'
from /usr/lib/ruby/vendor_ruby/wikicloth.rb:109:in `to_html'
from (irb):2
from /usr/bin/irb:12:in `<main>'

Expected result (performing require 'i18n' before I18n check in lib/wikicloth.rb):

irb(main):001:0> require 'wikicloth'
=> true
irb(main):002:0> WikiCloth::WikiCloth.new(:data => "__TOC__\n\n= One =\n\nwth").to_html(:noedit => true)
=> "\n\n\n\n<h1><a name=\"One\"></a><span class=\"mw-headline\" id=\"One\">One</span></h1>\n\n\n\n<p>wth</p>"

wikicloth with JRuby 1.7.10?

Is it somehow possible to use wikicloth with JRuby 1.7.10? I just noticed the dependency on rinku which I don't know how to build for JRuby.

Abbr doesn't work

Testing the following:

An '''HTML element''' is an individual component of an [[HTML]] document.

Taken from here: http://en.wikipedia.org/wiki/HTML_element#General_phrase_elements

I think it must be allowed, as mentioned in the specifications: http://www.mediawiki.org/wiki/Markup_spec#Anywhere

"Plus most non-dangerous HTML semantic inline elements: 'br', 'abbr', 'cite', 'del', 'ins', 'sub', 'sup', 'em', 'strong', 'var', 'code', 'ruby', 'rb', 'rt', 'rp' "

image link attributes

There's not an example of this, and I couldn't figure how to access if from the source, but how can I modify the way images are sourced and linked back to?

Ie. I have an image [[Image:/uploads/image/image/9/daniel_ek_and_martin_lorentzon-spotify.jpg]]

too long and complicated...

I would rather have the just the name "daniel_ek_and_martin_lorentzon-spotify.jpg" and then shorten it using the image_tag helper method in rails. How can I override this similar to the way you overwrite links?

link_attributes_for do |page|
  Post.exists?(page.parameterize) ? { :href => url_for(page) } : { :href => url_for(page), :class => 'new' }
end

Nested lists are not rendered properly

This wikitext fragment

# one
#* eee
#* eee
# two
# three

is rendered as

<ol><li>one
<ul>
<li>eee
</li>
<li>eee
</li>
</ul>
</li></ol>
<li>two
</li>
<li>three</li>

But the following is expected

<ol><li>one
<ul>
<li>eee
</li>
<li>eee
</li>
</ul>
</li>
<li>two
</li>
<li>three</li>
</ol>

Paragraphs around headings

It seems WikiCloth adds a paragraph around headings, which causes browsers to add extra paragraphs.

# == A second level heading ==
# 
# A paragraph
#
# Another paragraph
WikiParser.new(:data => "== A second level heading ==\n\nA paragraph\n\nAnother paragraph").to_html

leads to

 => "\n<p><h2><span class=\"editsection\">&#91;<a href=\"?section=A_second_level_heading\" title=\"Edit section: A second level heading\">edit</a>&#93;</span> <span class=\"mw-headline\" id=\"A_second_level_heading\"><a name=\"A_second_level_heading\">A second level heading</a></span></h2>\n</p>\n<p>A paragraph\n</p>\n<p>Another paragraph</p>"

This is invalid HTML and causes browsers (at least webkit-based ones) to convert both the <p> before the <h2> and the </p> after the <h2> into a complete paragraph (<p></p>), making the DOM look like this:

<p></p>
<h2>A second level heading</h2>
<p></p>
<p>A paragraph</p>

These empty paragraphs (typically having margins set in the CSS) take up space on the HTML page, messing up the visuals.

If you use a single newline after the heading, instead of a double like above, the first paragraph is "eaten" by the paragraph around the h2:

# == A second level heading ==
# A paragraph
#
# Another paragraph

WikiParser.new(:data => "== A second level heading ==\nA paragraph\n\nAnother paragraph").to_html

This returns:

=> "\n<p><h2><span class=\"editsection\">&#91;<a href=\"?section=A_second_level_heading\" title=\"Edit section: A second level heading\">edit</a>&#93;</span> <span class=\"mw-headline\" id=\"A_second_level_heading\"><a name=\"A_second_level_heading\">A second level heading</a></span></h2>\nA paragraph\n</p>\n<p>Another paragraph</p>"

Which is turned into DOM like this:

<p></p>
<h2>A second level heading</h2>
"A paragraph"
<p></p>
<p>Another paragraph</p>

I believe both these behaviors are wrong. In both cases I would expect something like the following:

=> "\n<h2><span class=\"editsection\">&#91;<a href=\"?section=A_second_level_heading\" title=\"Edit section: A second level heading\">edit</a>&#93;</span> <span class=\"mw-headline\" id=\"A_second_level_heading\"><a name=\"A_second_level_heading\">A second level heading</a></span></h2>\n<p>A paragraph</p>\n<p>Another paragraph</p>"

Resulting in this DOM:

<h2>A second level heading</h2>
<p>A paragraph</p>
<p>Another paragraph</p>

Some internal links are not rendered properly

Consider for example this

[[Script:bla_bla_bla_bla | Foo]]

should be rendered as

<a href="Script:bla_bla_bla_bla">Foo</a>

but renders as

<a href="Script:bla_bla_bla_bla"><span class="resource-prefix">Script</span>:<span class="resource-postfix">bla_bla_bla_bla</span></a>

External links are broken with several multilines

According to http://www.mediawiki.org/wiki/Help:Links#External_links
http://mediawiki.org should be rendered as a

<a href="http://mediawiki.org">"http://mediawiki.org"</a>

but it renders as

<p>http://mediawiki.org</p> which is a plain text.

Parsing the wiki data of Klaus_Schulze leads to exception

When trying to parse the wiki text from http://de.wikipedia.org/wiki/Klaus_Schulze , wikicloth throws an exception:

The error occurred while evaluating nil.[]
from /.rvm/gems/ruby-1.9.2-p0/gems/wikicloth-0.6.3/lib/wikicloth/wiki_buffer.rb:133:in `add_char'
from /.rvm/gems/ruby-1.9.2-p0/gems/wikicloth-0.6.3/lib/wikicloth/wiki_buffer.rb:134:in `add_char'
from /.rvm/gems/ruby-1.9.2-p0/gems/wikicloth-0.6.3/lib/wikicloth.rb:64:in `block in render'
from /.rvm/gems/ruby-1.9.2-p0/gems/wikicloth-0.6.3/lib/wikicloth.rb:64:in `each_char'
from /.rvm/gems/ruby-1.9.2-p0/gems/wikicloth-0.6.3/lib/wikicloth.rb:64:in `render'
from /.rvm/gems/ruby-1.9.2-p0/gems/wikicloth-0.6.3/lib/wikicloth.rb:69:in `to_html'
from /.rvm/gems/ruby-1.9.2-p0/gems/wikicloth-0.6.3/lib/wikicloth/parser.rb:97:in `to_html'

Possibility to disable Edit-Links

I want to remove the Edit-Links completely with an additional option. I only want to show content from wikipedia.

Subset of Wiki markup

Is it possible to only allow a subset of the MediaWiki markup language? Maybe by using a white list?

trouble with put_section

I've been having difficulty with the put_section() method.

get_section works fine.

@page = Page.find(params[:id])

if params[:section]
   section = @page.wiki.get_section(params[:section])
   @page.section_content = section
end

But when I try to do:

@page = Page.find(params[:id])

if params[:section]
  wiki = @page.wiki
  wiki.put_section(params[:section],params[:section_content])
  @page.content = wiki.to_wiki
end

#normal rails save here...

It does nothing. Also tried to get it to work in the console with no luck. ;-(

Wiki method in model

  def wiki
    data = WikiParser.new(:data => content)
  end

Can't render preformated text

combination of tags break it (not working nether preview/view):

wont show last paragaraph:

https://github.com/axet/gzproject/wiki/WebDesigner-Environment

same as here, wont show last paragraph:

https://github.com/axet/gzproject/wiki/GZone-GetResources

Alternate callback styles...

Here are some ideas on how to pass callbacks to the WikiCloth parser: http://pastie.org/1078454

toc

I was trying to change the default behavior for the table of contents, but I got a wrong number of arguments error. That can be fixed by patching parser.rb

      def toc(&block)
        self.send :define_method, 'toc' do |sections, numbered|
          self.instance_exec(sections, numbered, &block)
        end
      end

Parsing wiki text inside extension

I have an extension, which handles a special tag, which itself might contain other custom tags (for which I have extensions as well)

Example:

<tag>
 <sub>
   foo
 <sub>
</tag>

In my extension I want to invoke parser such as it recursively handles sub-tags

module WikiCloth
class TagExtension < Extension

 # <tag>
 # ....
 # </tag>
 element 'tag', :skip_html => true, :run_globals => false do |buffer|
  content = buffer.element_content     
  wiki = Parser.new(:data => content)
  wiki.to_html 

However, it doesn't invoke the handler for 'sub' and renders a normal html

I tested a "sub" extension separately (being a top-level tag) -- it works. It only doesn't work inside other extension.

What I am doing wrong?

colon in transclude breaks things

require 'wikicloth'

class Foo < WikiCloth::Parser
  template do |template|
    p template
  end
end

Foo.new(:data => '{{:Foo bar bat baz}}').to_html
# "Foo bar bat baz"
# => "\n<p>Foo bar bat baz</p>"

Foo.new(:data => '{{:Foo bar:bat baz}}').to_html                                                                                                                  
# => "\n"

As you can see, a colon in the title breaks the handling of the template. Yes, this is valid.

Would patch it myself if I could figure out where the problem was. :/

mailto links in mediawiki format don't work

Hi,
I'm trying to add some wiki pages to https://github.com/immon/mailredirect, but I can't seem to add mailto links as specified in http://www.mediawiki.org/wiki/Help:Links#External_links. I add [mailto:[email protected] Onno Ekker], but this is rendered as [mailto:[email protected]] , instead of Onno Ekker with a link to my emailaddress...
Onno

Error when parsing lists

Example:
https://github.com/lucaswerkmeister/JFractalizer/wiki/GitHub-MediaWiki-bug-demonstration
The lists part should be rendered to:

<ul><li>Level 1<ul>
<li>Level 2<ul><li>Level 3, and these two closing angle brackets are not in the source code:</li></ul>
</li></ul>
</li>
<li>Level 1, not Level 2<ul><li>Level 2, not Level 3</li></ul>
</li>
</ul>
Numbered, and totally not indented:
<ol><li>Level 1<ol>
<li>Level 2<ol><li>Level 3</li></ol>
</li></ol>
</li>
<li>Level 1<ol><li>Level 2</li></ol>
</li>
</ol>

but is

<ul><li>Level 1<ul>
<li>Level 2<ul><li>Level 3, and these two closing angle brackets are not in the source code:&gt;&gt;</li></ul>
</li>
<li>Level 1, not Level 2<ul><li>Level 2, not Level 3</li></ul>
</li>
</ul>
Numbered, and totally not indented:




<ol><li>Level 1<ol>
<li>Level 2<ol><li>Level 3&gt;&gt;</li></ol>
</li>
<li>Level 1<ol><li>Level 2</li></ol>
</li>
</ol>
</li></ol>
</li></ul>

The main error is the </li></ol></li></ul> being closed too late (probably not by wikicloth at all but instead by some sanitizer in Gollum), and the >>s (always as many as list levels are closed) where the closing list tags should be are also wrong (but might also be inserted by the sanitizer).
I tried identifying the problem, but I don't know Ruby, and I don't know your code, so my analysis might be wrong:

• I'm quite sure the problem lies in wiki_buffer.rb (https://github.com/nricciar/wikicloth/blob/master/lib/wikicloth/wiki_buffer.rb), probably in render_list_data().
• The part that should close the list tags is apparently
  close.each_char do |e| ret << "</#{list_inner_tag_for(e)}></#{list_tag_for(e)}>" (line 408), so the problem seems to be that close is empty. I can't find why.

no output when an internal link terminates the data string

I'm using wikicloth 0.7.1 in a local rails application. When I give a :data parameter containing an internal link that sits as the last character of the string I receive a zero length string as the output of to_html

Eg:
irb(main):013:0> WikiCloth::Parser.new({ :data => '[[test]]' }).to_html
=> ""

However if I ensure there is at least one trailing character then it gives the expected output

Eg:
irb(main):014:0> WikiCloth::Parser.new({ :data => '[[test]] ' }).to_html
=> "\n

<a href="test">test

"

Unexpected I18n.locale overwrite

An odd bug in one of our projects came from wikicloth overwriting I18n.locale to the default locale if no locale was explicitly passed in at parser construction time. While it is possible to be sure to pass in the correct locale, it seems like wikicloth shouldn't overwrite this value to system default when no option is passed. Sending a PR that changes the behavior, if you agree.

Contents within <math> tag can cause exception

The following text can cause an exception at table.rb:108 as wikicloth understands the "{|" to be the start of a table:
1-\frac{k}{|E(G_j)|}

This particular error can occur many times on a page with content such as Wikipedia's Randomized algorithm page.

Perhaps content should be passed to the link_handler, with default behavior of ignoring it?

External links in ToC are not rendered as expected

This is how links in headings are being rendered in ToC on mediawiki.org:

And this is how the same markup is rendered on github.com:

The markup is:

== [https://github.com/nricciar/wikicloth Heading 2] ==

== [https://github.com/nricciar/wikicloth Heading 2] ==

=== [https://github.com/nricciar/wikicloth Heading 3] ===

==== [https://github.com/nricciar/wikicloth Heading 4] ====

===== [https://github.com/nricciar/wikicloth Heading 5] ===== 

Table of contents not working

Table of contents is not rendered.

Steps to reproduce

irb(main):001:0> require 'wikicloth'
=> true
irb(main):002:0> WikiCloth::WikiCloth.new(:data => "__TOC__\n\n= One =\nabcd").to_html(:noedit => true)
=> "\n\n\n\n<h1><a name=\"One\"></a><span class=\"mw-headline\" id=\"One\">One</span></h1>\n\n\n<p>abcd</p>"

Lines starting with space followed by a line with single space are missing in the HTML result.

UPDATE: I further reduced the failing example text.

This:

test = <<DOC
 missing text

DOC
# Note there are one or more whitespaces in line 2 of the text above, but GitHub deletes them.

p WikiCloth::Parser.new(:data => test).to_html

Causes that:

\n\n<p><pre></pre>\n</p>

latex handling

It looks like wikicloth can handle even handle latex code inside "math" tags.

The problem is there it makes some assumptions:
the conversions need to be done by a programm called "blatex",
it need to be on the same server,
and it need to called by unix shell.

The logic for all this is hidden in the html_element.to_s method
To make it easier to customize it, the whole handling should at least go into an extra method.

use tags to mark released versions

Cannot get the list of sections

Assuming I have this markup

== Head1 ==
foo

== Head2 ==
bar

wiki = WikiCloth::Parser.new(:data => data, :noedit => true) 
wiki.to_html
puts "Sections: #{wiki.sections}" 

returns me an array with one element [""]

Hoverer the to_html method returns a valid HTML

Rendering of lists

As mentioned in another issue, there are problems rendering lists. I forked your code and rewrote render_list_data() to fix it. I can't prove it works in all cases, but it does work for these:

== 1 ==
* a
** b
*#* c
*** x

==2==
# one
#* eee
#* eee
# two
# three

==3==
# a
# b
#* c
# d
# e
#* f
#* g
#* h
#* i
# j
#* k
# l
#* m

==4==
:# a
:# b
:#: c
:# d

The above lists generate element for element the same results as MediaWiki. Feel free to incorporate the changes into your branch.

no whitespace after link

if i have in markup

  [[SomePage]] next

i get

  <a href="SomePage">SomePage</a>next

so no whitespace placed between link and text

Convert single newlines to space (and newline)?

Should single newlines at the end of lines be converted to a space, or maybe to a space and a newline?

I tend to break the lines in my wiki markup with newlines for readability, but when these files are rendered with WikiCloth, the last word of a line gets joined to the first word of the next line, unless I explicitly add a space at the end of every line.

This is not a problem for manually edited HTML files, so I think WikiCloth does something not quite right here.

release it please

You've made lots of fixes, even a prerelease would help :)

0.5.0 is totally broken with large documents, master works fine

Formatting in code blocks not rendered

Formatting in code blocks is not working as expected:

1.9.3-p362 :002 > str = " code block ''this should be italic'' '''this should be bold'''"
 => " code block ''this should be italic'' '''this should be bold'''" 
1.9.3-p362 :003 > WikiCloth::Parser.new({:data => str}).to_html
 => "\n\n<p><pre> code block &#39;&#39;this should be italic&#39;&#39; &#39;&#39;&#39;this should be bold&#39;&#39;&#39;\n</pre>\n</p>" 

Two things that need doing.

1. Table of contents.
2. Some method of setting how section edit links work.

Resource links don't work for other languages

In English Wikipedia pages, resources are labelled "file:something" but in other languages they're localised.
For example on the Korean page for Cake "file" is "파일".

The list of prefixes in wiki_link_handler won't handle these, as far as I can tell.

Hardcoded "Table of Contents"

The automatic "Table of content" text is hardcoded inside the method WikiLinkHandler#toc , this makes it difficult to change it,.
For example to translate it, Or change the structure of the html, like use css for the style or get rid of the table.

One solution is to create a method to emits the string, so that subclasses can override it.

wiki_buffer.rb:141:in `+': no implicit conversion of nil into String.

It happens when I try to parse wikipedia page "Lawrence Joseph Sarsfield Daly" https://gist.github.com/Zhomart/9e7c6f3735bc7221f874

Regex issue in wikicloth.rb line 70

If you meet a string like this:

"<!--" + " " * 41 =~ /<!--(.|\s)*?-->/

It will run forever.

maybe /<!--.*?-->/m is a good replacement