Large seed URL offer for CC News Dataset about news-crawl

[john-hewitt]

Awesome. My advisor and I would prefer to email these as opposed to posting them on Github directly. We scraped them from a site, and for the site's sake would prefer they not be 100% public.

[sebastian-nagel]

Hi @john-hewitt,
we are definitely interested. Thanks! The crawler currently requires that a news site provides a RSS feed which is not a redirect. But we hope to remove this limitations, see also #3 and #8. To have a large and curated list of news sites available is of course great, the data from DMOZ contains some noise.
-Sebastian

[sebastian-nagel]

Hi @john-hewitt,
thanks I'll contact you per mail. Of course, we would prefer an open seed list to be transparent and ensure a certain amount of reproducibility of the crawls. That's why DMOZ is an ideal resource. We need to discuss internally how we proceed and whether or not we use it as seed list, or alternatively as a reference to check the coverage of DMOZ links.
-Sebastian

[sebastian-nagel]

Closing. The seeds for the news crawl have been improved significantly over the last 2 years. Thanks, @john-hewitt!