Dockerized Map Reduce about pachyderm

[jdoliner]

This feature is underway and schedule to ship in v0.3. You can track progress here.

The API for this right now works (or is going to work) as follows:

# Creating a job
curl -XPOST pfs/job/foo -d '{"input" : ["inputDir", "parentJob"], "container" : "jdoliner/container", "command" : ["/usr/bin/webserver", "arg1", "arg2"]}'
# Checking the status of a job being run on <commit>
curl -XGET pfs/job/foo/status?commit=<commit>
# Accessing output from a job
curl -XGET pfs/job/foo/file/bar

Once jobs are created via a post to /job they will automatically be kicked off following the next commit to the branch they were created on.

After the initial commit jobs will be rerun any time new data is committed that changes the input to the job.

Jobs currently consist of a piece of JSON with 3 fields "input", "container" and "command". "container" and "command" are used in a call to docker run to spin up the webserver that documents are passed through. "input" defines which documents should be passed to the server. Input can either be directories from the raw filesystem or output from other jobs. Pfs will automatically schedule jobs based on their dependency on one another.