abayer / commoncrawl Goto Github PK

The start of the shared commoncrawl code repository.

Please set hadoop.version and hadoop.path in build.properties to point to your version of 
hadoop. 

Once commoncrawl.jar has been built, you can execute a job/sample via the bin/launcher.sh script.

For example, to run the BasicArcFileReaderSample against the ARC file 2010/01/07/18/1262876244253_18.arc.gz 
in the main commoncrawl bucket, commoncrawl-crawl-002, you would run the following command line:

  bin/launcher.sh org.commoncrawl.samples.BasicArcFileReaderSample <<AWS ACCESS KEY>>  <<AWS SECRET KEY>> commoncrawl-crawl-002 2010/01/07/18/1262876244253_18.arc.gz

The luancher runs the command in the background. You can monitor progress via either ./logs/<<ClassName>>.log for LOG output, or ./logs/<<ClassName>>_run.log for stdout output.