Document-level rendering? about jupyterlab-markup

[agoose77]

The big issue here in the MyST context is a hybrid of JupyterLab constraints and markdown-it / MyST constraints

The big "missing feature" is that references are document-level, but right now MarkdownIt is handled per-cell. If we have a per-document renderer context, then we fix the per-cell problem, but then have state invalidation to deal with.

Ultimately, as a UX, I'd like:

• references to work (at the document level)
• partial renders to work (updating a single cell doesn't cause a total reflow)

In technical terms, this would mean:

• the ability to parse each cell into an AST.
• this AST can then be transformed by extensions
• one extension can declare a dependency upon a part of the AST
• this dependency tree is used to perform partial re-renders.
• rendering is stateless; all contextual information handled by AST transforms.

In practical terms, I think this probably means ultimately moving away from markdown-it in favour of something like the unifiedjs ecosystem. We'd also need to modify how JLab invokes the rendering mechanism, namely that it cannot be entirely stateless from the markdown perspective - the mdast renderer could be, though. This might be as simple as bypassing the markdown rendering mechanism 🤮 and registering an mdast renderer. Then the JLab integration can also speak mdast, but we take charge at the notebook level of getting that mdast from the markdown source.

I wish this were trivial 😆

[bollwyvl]

I think to get document-level features to work, the whole authoring experience needs to be rethought, not (just) the parsing/rendering one. For a few years, I've thought ProseMirror is the still the likeliest candidate, and JupyterLab 4 will probably be the right timeline, for CodeMirror 6 and simplified focus for RTC (just yjs).

For example, in a notebook, Instead of JSON cells that sometimes have markdown, the top-level document would be a prosemirror node (nbformat:notebook) that only allowed nbformat:metadata and nbformat:cells at the top, with subsequent, extensible schema structures underneath, each with a dedicated UI.

This would mean most of the time, editing happens against directly properties on nodes, and round-tripping to nbformat JSON (and markdown source) would happen only at hard save points.

One side effect of this might be a custom collaborative editing schema rather than moving the nbformat json schema, or raw text in the editor case, over CRDT.

The UI/DX of a notebook would play out where, much like a google doc, a cherry-picked metadata (title, author, etc) would optionally appear at the top, and unlike the current notebook ux, would start with an empty markdown cell, where tab would change it to a code cell, starting an embedded CM6 UI.

None of this changes #13, however... so there's still very much a need for a robust way to document the intent of the document owner vs whatever opinions their tool author had today. But taking over the whole document, it will be a lot more natural to own the metadata as well.

Done right, the user input and output result is quite pleasing, though:

• https://github.com/benrbray/prosemirror-math (leaving aside the whole katex/mathjax issue for now)

And composable with "big rocks" already being adopting in JupyterLab 4

• https://github.com/sibiraj-s/prosemirror-codemirror-6