zofuthan / dagobah

Simple DAG-based job scheduler in Python

Dagobah

Dagobah is a simple dependency-based job scheduler written in Python. Dagobah allows you to schedule periodic jobs using Cron syntax. Each job then kicks off a series of tasks (subprocesses) in an order defined by a dependency graph you can easily draw with click-and-drag in the web interface.

Dagobah lets you retry individual tasks from failure, sends you helpful email reports on job completion and failure, keeps track of your tasks' stdout and stderr, and persists its information in various backends so you don't have to worry about losing your data.

You can also [use Dagobah directly in Python.](../../wiki/Using Dagobah Directly in Python)

Installation

Dagobah works with Python 2.6 or 2.7.

pip install dagobah
dagobahd  # start the web interface on localhost:9000

On first start, dagobahd will create a config file at ~/.dagobahd.yml. You'll probably want to check that out to get your backend and email options set up before you start using Dagobah.

Dagobah does not require a backend, but unless you specify one, your jobs and tasks will be lost when the daemon exits. Each backend requires its own set of drivers. Once you've installed the drivers, you then need to specify any backend-specific options in the config. See the config file for details.

Available Backends

To use a backend, you need to install the drivers using the commands below and then tell Dagobah to use the backend in the config file (default location ~/.dagobahd.yml).

MongoDB

pip install pymongo

Deprecated Backends

• SQLite: Deprecated following version 0.3.1.

Features

Single-user auth

Manage multiple jobs scheduled with Cron syntax. Run times are shown in your local timezone.

Tasks can be anything you'd normally run at a shell prompt. Pipe and redirect your heart out.

Failed tasks don't break your entire job. Once you fix the task, the job picks up from where it left off.

On completion and failure, Dagobah sends you an email summary of the executed job (just set it up in the config file).

Tasks can even be run on remote machines (using your SSH config)

Contributors

• Travis Thieman
• Shon T. Urbas
• Utkarsh Sengar
• Stephanie Wei
• Ryan Clough

Get Started Contributing

• See the fledgling wiki or create a new issue to get started
• If you have any questions, go ahead and email or tweet at me, or go ahead and create a new issue in this repository.