drbenway / siteresearch Goto Github PK

export the crawl result to sitemap.xml file

When setting up the crawler, provide an option to populate the url table with all known urls from google.
(google search "site:www.yourdomain.com")

Add support for Gephi export

provide the option to import an xml sitemap file into the crawler tables

plugin to export crawler data to xgmml support (https://gephi.org/)

write unit tests for crawler.php

add phpdoc comments to setup script

Option to define the maximum crawle depth

Allow for a random amount of time between two page fetches

Allow the export of the crawler table for later use.

phpdoc on crawler.php contains lots of errors.

add the option to run multiple scripts at the same time.

update wiki or project description to point to the api docs and designs

option to export the crawler table to csv file

regard settings in the robot.txt file when crawling urls

create a class to support httpwatch based on http://www.phpied.com/automating-httpwatch-with-php/
windows only

do implementation tests on mamp, wamp & xampp

do tests with different php and mysql versions and settings do define a set of minimum requirements

Allow the export of the crawler table for later use.

option to export the crawler table to an excel or open office document.