Feature request: RSS feed crawler about ambar

[sochix]

Hi @jenkin . Can you please describe this crawler a bit? Should it fetch all linked documents? And if the document an html page should it fetch it too?

[jenkin]

RSS specifications require the mime type as attribute of enclosure tag: "It has three required attributes: url says where the enclosure is located, length says how big it is in bytes, and type says what its type is, a standard MIME type." So the crawler should fetch only enclosures with a supported (by Ambar) mime type (ie. application/pdf). Other types, such as mp3 in podcast feeds, will be ignored.

[sochix]

@jenkin got it! Can you please also describe use case for the RSS crawler? What kind of data you want to crawl with it? Newsletters or what?

[sochix]

Check our support options