[question] large sitemap, check last sitemap item and continue generate next file only? about sitemap.js

[huntharo]

Yeah, you don't want to run SitemapAndIndexStream again.

Although... the DB index you should be using should have all of the sitemap records sequentially in an index, so the actual cost of the query should be very low in that it should not use a lot of DB I/O and should not use a lot of DB CPU.

100k records is not a lot. That would only need to write a 2-3 record sitemap index and 2-4 sitemap xml files. Regenerating all of that would only take a minute or so with the XML serialization being the longest part.

The alternative is much more complex. You'll need to pull back the index file, get the latest sitemap XML file out of it, fetch that file, parse the latest sitemap into items, add to the items collection, rotate into a new file (and add it to the index) when the last file fills up, then put back the old latest file, any newly added files, and the index file.

It's... not trivial. I have a system that I'm trying to open source that does all of that, but I can't make any promises.

[huntharo]

Here is a project that utilizes this project and uses Kinesis streams to allow millions of items to be written to sitemaps (and updated / maintained):

https://github.com/shutterstock/streaming-sitemaps

That project may meet your needs or give you some ideas on how to approach the problem.