Parsed syntax tree cannot be modified (note: will not be implemented) about marked

[UziTech]

Sure there are many ways you can do that, it just depends on how much you want to recreate what marked.parse does for you.

One way you can do it is to use Lexer.blockTokens to create block tokens then update links before creating inline tokens.

As you can see that isn't quite as simple as just appending reflinks to the markdown.

[UziTech]

That doesn't work because it is not supposed to work like that. The url is resolved when creating the link tokens. It would be redundant, and a waste of time, to resolve it again on parsing. I'm going to close this since there is a way to do this even if it isn't the way that you want.

[calculuschild]

Defined where externally? Just on a different page?

[tipiirai]

I tried to modify tokens.links property returned by the lexer before rendering the page, but couldn't get it to work. Just trying to find any means to provide the links before rendering. Could be on a different page or given as a configuration option.

[UziTech]

You can add the reflinks to the markdown in a preprocess hook.

Something like:

marked.use({
  hooks: {
    preprocess(markdown) {
      return markdown + '\n\n[mylink]: https://example.com/mylink';
    }
  }
});

[tipiirai]

The hook is not called with marked.parser(tokens) where the tokens are generated with marked.lexer()

[UziTech]

No. It is called by marked.parse(). If you are using marked.lexer you can just append the reflinks to the markdown that you pass in

[tipiirai]

Gotcha. So there is no way to parse Markdown into an abstract syntax tree, then modify the tree (links specifically), and finally render the tree to HTML. Like parsing & rendering usually works.

[tipiirai]

Lexer.blockTokens sounds good! How exactly I can use that for updating the links? Some sample code would be helpful. Doesn't have to work. I can work my way out from there. Thanks!

[UziTech]

You will have to essentially copy the code from Lexer.lex and add in whatever changes you want.

marked/src/Lexer.ts

Lines 86 to 98 in 0d3e9e7

	lex(src: string) {
	src = src
	.replace(/\r\n|\r/g, '\n');
	this.blockTokens(src, this.tokens);
	let next;
	while (next = this.inlineQueue.shift()) {
	this.inlineTokens(next.src, next.tokens);
	}
	return this.tokens;
	}

[tipiirai]

Sorry. I have no idea how to use the blockTokens() function. Is it available as an API method at all?

[UziTech]

You can access the Lexer API by creating a new Lexer

import { Lexer, marked } from 'marked';

const lexer = new Lexer();
const tokens = lexer.blockTokens(markdown);
// update tokens.links
...

[tipiirai]

Thanks. But the issue is that updating tokens.links has no effect. For example by setting:

tokens.links.foo = { href: 'bar.com }

And [test][foo] is still rendered as [test][foo]

[UziTech]

Can you share your whole code? It should work since that is exactly what marked does.

[tipiirai]

Here's the code:

import { marked } from 'marked'

const md = `
[This works][works]

[This doesn't][test]

[works]: works.com
`
const tokens = marked.lexer(md)

tokens.links.test = { href: 'doesnt.com'}

console.info(marked.parser(tokens))

And the output is

<p><a href="works.com">This works</a></p>
<p>[This doesn’t][test]</p>

[UziTech]

You aren't calling blockTokens at all?

Marked.lexer creates block tokens and inline tokens. You have to add the link before getting the inline tokens.

[tipiirai]

I have no idea how to use the undocumented blockTokens method. I tried this now:

import { marked, Lexer } from 'marked'

const md = `
[This works][works]

[This doesn't][test]

[works]: works.com
`

const lexer = new Lexer()
const tokens = lexer.blockTokens(md)

tokens.links.test = { href: 'doesnt.com'}

But the last sentence threw an error:

TypeError: undefined is not an object (evaluating 'tokens.links.test = { href: "doesnt.com" }')

Can you share a code that correctly updates the links before rendering? Thanks.

[UziTech]

It seems like just appending reflinks to the markdown would be the best way to do it. If you get it working with blockTokens it could be hard to update it if we ever change the Lexer methods.

[tipiirai]

it seems appending reflinks to the markdown is the only way to do this, because there is no way to modify the abstract syntax tree before rendering as I originally stated.

[UziTech]

Adding a hook between block tokens and inline tokens is something we were thinking about doing. If you wanted to create a PR for that it would be very helpful.

[UziTech]

This is one way you can currently do it with Lexer and Parser

import { Marked, Lexer, Parser } from 'marked';

const marked = new Marked();

marked.use(
  // extensions
);

const markdown = `
[This works][works]

[This does too][test]

[works]: works.com
`;

const lexer = new Lexer(marked.defaults);

lexer.tokens.links.test = { href: 'doesToo.com' };

const tokens = lexer.lex(markdown);

const parser = new Parser(marked.defaults);

console.log(parser.parse(tokens));

But this also works and is a lot simpler:

console.log(marked.parse(markdown + '\n\n[test]: doesAlso.com'));

[tipiirai]

Cool. I got that working.

I agree, the JavaScript API for providing links in the AST phase is quite messy and not too user-friendly. Thre is no clear parse & render phases in Marked, which is unfortunate. I would like to do something like this (pseudocode):

import { parse, render } from 'marked'

// build AST
const ast = marked.parse(markdown)

// modify it
ast.doSomething()
ast.doSomethingElse()

// render with rendering options
const html = render(ast, { renderers, hooks, ... })

[tipiirai]

Actually the issue on the title still remains. There is no way to update the links after parsing the AST, but before rendering. ie. the following does not work

// parse tokens (AST)
const tokens = lexer.lex(markdown);

// update tokens
tokens.links.test = { href: 'doesToo.com' };

// this doesn't work
console.log(parser.parse(tokens));

[UziTech]

Also this does work

import { marked } from 'marked'

// build AST
const ast = marked.lex(markdown)

// modify it
doSomething(ast)
doSomethingElse(ast)

// render with rendering options
const html = marked.parser(ast)

we just don't provide the doSomething and doSomethingElse functions.

If you would want to create a package that provides convenient methods for stuff to do with the ast you can.

[tipiirai]

I was about to make such convenient methods but then realized that it's impossible with the current design of Marked, which does not have a clear and modifiable syntax tree. The parsed tree is somehow "resolved" and cannot be altered.