datamill-co / singer-runner

A CLI and library to run Singer Taps and Targets

singer-runner

Singer Runner manages tap and target processes, as well as metrics, state, and configuration.

Features

• Run a tap or target
• Pass run options via CLI paramters or JSON/YAML config file.
• Use local file system or S3 for piping the singer stream, storing state, and storing metrics.
• Metric storage, piping, and state storage can be extended / customized by inheriting from the base classes.

Usage

Requires python 3, tested with python 3.7

Install

pip install singer-runner

Run

$ singer-runner 
Usage: singer-runner [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  run-tap
  run-target

Concepts

• Pipes
  • Pipes move a stream of Singer messages from tap to target. A pipe could be as simple as a local file, a file in S3, or Kafka.
• State Storage
  • Loads/saves the state JSON.
• Metrics Storage
  • Accepts a stream of Singer Metrics

Programmatic Usage

Singer runner can be used within any python application. The primary functions are in singer_runner.runner including:

• run_tap runs a tap
• run_target runs a target

Classes in the singer_runner.metrics, singer_runner.pipes, and singer_runner.state can be used as arguemnts, along with catalog/config.