sromexs / get-sitemap-links

This package get, fetch, crawl, sitemap pages recursively and fetch all links in between <loc> tag.

I am Sadra 👋

Available for a remote job:

My main skills:

Intermediate skills

Low skills

get-sitemap-links's Issues

The links should parse XML data format link <CData /> when parsing sitemaps

First of all, thanks for your work here :)

I am using this library for an internal tool and realized that this fails to extract URLs correctly when the loc data is in CDAta format like below

<sitemap>
  <loc><![CDATA[https://example.com/post-sitemap.xml]]></loc>
  <lastmod><![CDATA[2020-11-16T18:13:33+00:00]]></lastmod>
</sitemap>

In this case, the expected return value is https://example.com/post-sitemap.xm but instead we get <![CDATA[https://example.com/post-sitemap.xml]]>

We perhaps need to add a regex somewhere to extract data between CData section

Something wrong with regex pattern

Regex rules may miss-match sometimes
for example code below may find next url https://www.orangemarketing.com</loc><lastmod>2023-01-30</lastmod></url><url><loc>https://www.orangemarketing.com/555

Its contains part of XML sitemap, looks like issue with missing trailing slash

const GetSitemapLinks = require("get-sitemap-links").default;

(async () => {
  const array = await GetSitemapLinks(
    "https://www.orangemarketing.com/sitemap.xml"
  );
  
  console.table(array);
})();