Understanding your intent about commonmark-hs

[mb21]

@jgm Maybe it would be useful to add a few sentences to commonmark/README.md explaining some design decisions? Like:

• what's the pipeline: input string -> Tokens.hs#tokenize: token stream -> Parser.hs#parseCommonmark -> renderer typeclass ?
• why formatting specs instead of just code (the old configuration vs. code debate)? is this for extensibility using custom extensions, or do you feel it even helps in writing the normal commonmark parser?
• ...?

Looking forward to see this progressing!

[jgm]

Andrew Cowie <[email protected]> writes:

Would you be willing to write a short para about your intent here? Is this package meant to ultimately replace the Pandoc markdown parser? Or are you using this as a place to trial extensions that might land in the CommonMark spec? Or... ?

Yes, I aim to replace pandoc's markdown parser with this library, eventually. This would resolve some longstanding bugs (e.g., jgm/pandoc#3903, jgm/pandoc#1735). It would also significantly increase markdown parsing speed. The first step would be to use this library, instead of cmark-gfm, for pandoc's 'commonmark' input, and gradually add extensions until most of pandoc's major markdown extensions are supported. At that point we might consider making 'commonmark' the default input format for pandoc, instead of 'markdown'. We'd probably need to keep the old pandoc markdown parser around, though, for conversions from/to other markdown flavors. My other aim is to provide a robust, conformant, flexible, fast, low-dependency, liberally licensed, pure Haskell commonmark parser for the Haskell community to use. One possible application is adding commonmark support to haddock. Having a pure Haskell parser, instead of a wrapper around libcmark, is a big advantage for two reasons: (1) safety, (2) ability to compile to JavaScript with ghcjs or to an eventual webassembly target, when ghc supports that. (So, this library would fill the role for which a number of people are currently using my unmaintained cheapskate library.)

I've been reading your code and trying to get my head around whether I should be attempting to contribute here, or to mmark, or somehow using cmark-gfm, or... Mostly I need one of the table extensions; my target is LaTeX fragments but I'm mostly interested in seeing what comes of a newer AST as compared to using pandoc-types' "native" AST.

Contributions here would be welcome! (And suggestions: The API is still evolving, and the code is probably less perspicuous than it should be.) mmark is really its own thing: it departs significantly from standard markdown syntax and from commonmark. cmark-gfm wraps a C library, so any parsing changes, or changes to the data structures, would have to be made in the C library.

[jgm]

@mb21 good idea, I've done that.

[mb21]

Thanks, exactly the info I was looking for!

And tokenize splits the input string on everything that's a space-like or alpha-numeric character.