phymucs / cobweb Goto Github PK
View Code? Open in Web Editor NEWThis project forked from stewartmckee/cobweb
Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
License: MIT License