Coder Social home page Coder Social logo

codepunkt / dispo Goto Github PK

View Code? Open in Web Editor NEW
9.0 5.0 7.0 120 KB

Dispo is a job and cronjob scheduler for Node

License: MIT License

JavaScript 100.00%
zeromq scheduler worker worker-management job job-scheduler cronjob-scheduler cronjob crontab-syntax

dispo's Introduction

Dispo

npm version Build Status Coverage Status js-standard-style

Dispo is a job and cronjob scheduler for Node.

It uses Kue as its job queue and Redis to store job data. Definition of recurring jobs is done using crontab syntax, one-off jobs can be queued using ZeroMQ requests. nodemailer and nodemailer-sendmail-transport are being used to send emails.

All Jobs, regardless if running automatically or queued on demand for single runs, have to be defined in a configuration file. Jobs defined as cronjobs with a recurring interval are scheduled and run automatically.

Requirements

Installation

> npm install dispo

Usage

Dispo provides a binary to start from the command line.

> node_modules/.bin/dispo -h

Usage: dispo [options]

Options:

  -h, --help               output usage information
  -V, --version            output the version number
  -C, --config <config>    config file path, default: `jobs.json`
  -B, --basedir <basedir>  directory used as a base for relative config and job paths

The --config or -C option is required and points to the relative path of the configuration file. If no configuration path is given, the configuration is expected to be in jobs.json relative to the current working directory.

Job configuration

Jobs are defined in the jobs property of the configuration file. Each job, identified using a name, must point its file property to a JavaScript file exporting a function that is run when the job is executed.

The following configuration example defines a job called logRandomNumber that can be queued on-demand and a recurring job called databaseCleanup that is defined to run on 01:00 am every day using crontab syntax.

The attempts property on the database cleanup job defines that the job is only attempted to run once. When the property is not explicitely set, it defaults to 3 so a job is retried twice on failure. When a recurringly scheduled job/cronjob reaches fails on each of its attempts, it is not automatically rescheduled.

The recipients property will override the default set in mailer.js. You can also use it to disable sending an email for a job when it is enabled globally. You can set the global configuration in index.js.

{
  "jobs": {
    "logRandomNumber": {
      "file": "jobs/logRandomNumber.js"
    },
    "databaseCleanup": {
      "file": "jobs/databaseCleanup.js",
      "cron": "0 1 * * * *",
      "attempts": 1,
      "recipients": "[email protected]"
    }
  }
}

Send email on job failure

Dispo supports sending an email when a job fails after all available retries, using nodemailer to send the mails. To do so, simply enable the mailer in the configuration file and add email addresses that should be notified whenever a job fails to notifyOnError on a per job basis.

{
  "options": {
    "mailer": true
  },
  "jobs": {
    "mightFail": {
      "file": "jobs/mightFail.js",
      "cron": "*/1 * * * *",
      "attempts": 3,
      "notifyOnError": "[email protected]"
    }
  }
}

By default, dispo will use nodemailer-sendmail-transport to send your emails, but you're free to add a different nodemailer transport such as smtp. You can also set mail options, such as from.

import nodemailer from 'nodemailer'

module.exports = {
  options: {
    mailer: {
      transport: nodemailer.createTransport('smtp://smtp.example.com')
      mail: {
        from: '[email protected]'
      }
    }
  },
  jobs: {
    mightFail: {
      file: 'jobs/mightFail.js',
      cron: '*/1 * * * *',
      attempts: 3,
      notifyOnError: '[email protected]'
    }
  }
}

Attempts with delay and backoff for failing jobs

Jobs that sometimes fail to execute correctly (or any job in general to be precise) can be configured to restart with a delay after they fail. You can use this feature via the backoff property. Provide backoff.type = 'fixed' and the backoff.delay = <Number> in milliseconds to set a fixed delay in milliseconds that will be waited after a failed attempt to execute the job. Provide backoff.type = 'exponential' and the backoff.delay = <Number> in milliseconds to set a exponential growing delay in milliseconds that will be waited after a failed attempt to execute the job. The base of the expenential growth will be your given delay.

The following configuration example defines a job called flakyService that is defined to run every minute on every day. flakyService will be executed a second, third and fourth time when it fails (attempts: 4), but the second, third and fourth try will each wait 3 seconds before re-executing.

{
  "flakyService": {
    "file": "jobs/flakyService.js",
    "cron": "*/1 * * * *",
    "attempts": 4,
    "backoff": {
      "delay": 3000,
      "type": "fixed"
    }
  }
}
incremental

The following configuration example defines a job called flakyServiceWithLongRegenerationTime that is defined to run every minute on every day. flakyServiceWithLongRegenerationTime will be executed a second, third, fourth, fifth and sixth time when it fails (attempts: 4), but:

  • the second try will wait 6 seconds,
  • the third try will wait 9 seconds,
  • the forth try will wait 12 seconds, before re-executing.
{
  "flakyServiceWithLongRegenerationTime": {
    "file": "jobs/flakyServiceWithLongRegenerationTime.js",
    "cron": "*/1 * * * *",
    "attempts": 4,
    "backoff": {
      "delay": 3000,
      "type": "incremental"
    }
  }
}
exponential

The following configuration example defines a job called anotherFlakyServiceWithLongRegenerationTime that is defined to run every minute on every day. anotherFlakyServiceWithLongRegenerationTime will be executed a second and third time when it fails (attempts: 4), but:

  • the second try will wait 4 seconds (= 2000 * 2000 milliseconds),
  • the third try will wait 16 seconds (= 4000 * 4000 milliseconds), before re-executing.
{
  "anotherFlakyServiceWithLongRegenerationTime": {
    "file": "jobs/anotherFlakyServiceWithLongRegenerationTime.js",
    "cron": "*/1 * * * *",
    "attempts": 3,
    "backoff": {
      "delay": 2000,
      "type": "exponential"
    }
  }
}

dispo's People

Contributors

codepunkt avatar draggha avatar kbariotis avatar lgeiger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

dispo's Issues

Send an email when a job failed

When a job fails after all available retries, send an email to selected recipients to inform them of the failure

Allow configuration of recipients in the configuration file on a per-job basis, but also allow to override it when queing jobs for single runs using ZMQ.

Fix travis installation of zmq

At the moment, CI checks on travis fail because there apparently is an issue with the npm installation of zmq.

Most likely zeromq is not installed on the virtual travisCI instance the tests are running in:
Failed at the [email protected] install script 'node-gyp rebuild'

Allow failure backoff configuration

kue already supports failure backoff configuration.

Allow backoff to be configured per-job in the configuration file. Also allow to override it when sending a ZMQ message to queue a job.

Extract a `parseJobs` method from the binary

Test that parsing the jobs from configuration file

  • fails when file property is missing for a job
  • fails when file property of a job doesnt point to an existing file
  • returns an array of job objects otherwise that consist of job name, number of attempts, fn to execute and optionally a string for scheduling in crontab syntax

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.