bduggan / clowder

Event-driven RESTful job processing

clowder

This is a prototype for a distributed event-driven job queue which uses HTTP as a transport and Redis as a backend.

Quick start :

Test :

prove t/01-process.t

Run :

JOBSERVER=http://localhost:3000
cd script
redis-server &
./jobserver.pl daemon --listen $JOBSERVER &
./subscriber.pl &
./minion.pl $JOBSERVER &
./submit_job.pl --app './app.pl'
./check_job.pl --id 1

System Architecture :

Start jobserver.pl, which waits for new jobs to do.
Start minion.pl which runs the jobs for you.

submit_job.pl will submit a job request
ingest_file.pl adds files (using REST)

subscriber.pl watches changes in state
so that as dependencies arrive, the minions don't have to wait.

Use check_job.pl in order to see
if your job is running or waiting for a key.

"Key"s represent files, or a granule of data.
We represent them with hashes; the content doesn't matter.

Contents :

• jobserver.pl : job server
• minion.pl : sample job processor (minion)
• submit_job.pl : client for submitting job requests
• ingest_file.pl : client for announcing file ingests
• subscriber.pl : backend process for event-driven state changes
• gui.pl : monitor files and jobs