bernardodesousa / its-crewed Goto Github PK

FYI here's a really good overview on writing a crawler.

http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/

To be honest the majority of your TODOs is creating copy which the third point being "write the crawler"

Why do you think you need headless chrome to crawl? One case I can see using something like that for is those types of blogs/sites that render all content via javascript meaning a standard GET request won't return response data we're interested. That's something I wouldn't even begin to think about until after creating the first MVP.

Frankly, I think the crawler itself is the "easy" part of the project (made even simpler if you use something off the shelf like Nutch), you then need to process all that data you just fetched. My first impression that this will be the harder part because the number of edge cases that can exist (especially as the number of corrections/transforms increases) is quite a lot. This first case of "manned" -> "crewed" is probably uncommon enough that the rate of false positives would be low but other such corrections/transformations would need more context / semantic analysis / natural language processing.