Parse and format markdown-ish syntax about athens

[tomisme]

https://hazel.org/ are doing some interesting work on running/analyzing incomplete programs using 'holes', could be a (longer term) direction to go in

[jeroenvandijk]

I've commented per sub topic:

Parsing

Instaparse is a way better option than regexp. It takes a bit to get used to, but eventually combining regexp's will just become a terrible fragile mess. So +1 on that.

Incomplete AST

What options do we have? In the backend I can imagine we would treat the incomplete AST just as normal next. On the front end we might have a special parser that has spelling-checker like behaviour, saying this like "you might want to fix this". We could also annotate (potentially) incomplete ASTs in the database I guess. This would allow one to fix incorrect AST's every now and then and improve the whole database.

Transformation

Like you said, Instaparse still needs a post-processing step to put it in the right data structure. I've used clojure.spec for this in the past and it can work, but it also a bit tricky. I wouldn't say it's ideal.

For the json import (#31) I was already thinking of experimenting with https://github.com/noprompt/meander to support the transformation from nested maps to datoms. I still need to validate this approach, but I can imagine it can be useful for transforming the ASTs as well. Either way clojure.spec should definitely be used to validate the shape of the end result. In the past I've worked with different layers of specs, like compiled form and dsl form. It doesn't have to be one spec layer that does everything.

[jeroenvandijk]

In addition to my comment about Meander, I just found this demo of an implementation of a hiccup parser here. Pretty impressive

[teodorlu]

Either way clojure.spec should definitely be used to validate the shape of the end result. In the past I've worked with different layers of specs, like compiled form and dsl form. It doesn't have to be one spec layer that does everything.

Interesting. Gave me some ideas about incomplete ASTs:

• Base spec for valid ASTs or strings
• "Extended frontend spec" that is (s/or :valid-ast ::valid-ast :ast-with-spell-check-suggest ::ast-with-spell-check-suggest) (TODO better names)

[roryokane]

In the current implementation of the parser, I’ve handled incomplete syntax not with a special “incomplete” type of AST, but by interpreting the syntax as valid plaintext. This is the same way Markdown (at least CommonMark) handles syntax errors.

I can explain how it works with a simplified version of the parser:

block = ( block-link / any-char )*
block-link = <'[['> any-chars <']]'>

Within the * repetition, block uses ordered alternation / to attempt to match the following characters to a block-link – [[ and ]] with character between them – first. If block-link doesn’t match, the parser consumes one more character with any-char and repeats.

That way, a string like [[foo] is ultimately not parsed as a special unterminated link, which I imagine might look like this:

[:block [:error-unterminated [:block-link "foo]"]]]

Rather, it’s interpreted as the literal text “[[foo]”, like this:

[:block "[[foo]"]

Accepting the syntax as plain text is much easier to implement than trying to detect and correct errors. Accepting the syntax as plain text also ensures that the author’s punctuation will never be hidden by being misinterpreted as incomplete syntax. The downside is that this parser can’t be used to provide error messages in the case that the user truly wanted to use the link syntax.

[teodorlu]

I'm currently playing around this this: https://github.com/teodorlu/athens/tree/parser-work/src/cljc/athens

[teodorlu]

Generally agree with all your points. Interesting application of incomplete AST to provide user error messages.

Also curious about what you have in mind for meander. I haven't put much thought to AST transformations.