smbkr / mappa Goto Github PK

We're not currently outputting valid JSON. We're just spitting out a POJO to the console. We need to call JSON.stringify first.

Instead of awaiting the sitemap, we could open a stream, write to that from the mapper, and output the contents to stdout as they are received.

As a mappa user
I would like to see the output as soon as it's ready
So that I know it's working

Add a max-depth config flag. If set, this should limit the depth to which we crawl. For example, given a max-depth of 2, we should only fetch and map links that are 2 segments deep relative to the hostname.

https://www.example.org/foo ✅
https://www.example.org/foo/bar ✅
https://www.example.org/foo/bar/baz ❌

This should probably be an environment varible. For example, MAPPA_MAX_DEPTH=2 npm run start 'https://www.example.org/'