Coder Social home page Coder Social logo

loongson-cloud-community / alertsnitch Goto Github PK

View Code? Open in Web Editor NEW

This project forked from yakshaving-art/alertsnitch

0.0 0.0 0.0 113 KB

Ingests Prometheus alerts and poops them into a MySQL database for offline analysis

License: MIT License

Shell 3.46% Go 85.94% Makefile 10.60%

alertsnitch's Introduction

AlertSnitch

Captures Prometheus AlertManager alerts and writes them in a MySQL or Postgres database for future examination.

Because given a noisy enough alerting environment, offline querying capabilities of triggered alerts is extremely valuable.

How does it work

  1. You stand up one of these however you like (multi-arch Docker images provided)
  2. You setup AlertManager to point at it and propagate your alerts in.
  3. Every alert that gets triggered reaches your database.
  4. Profit.
graph TD
    A[alertmanager] -->|POST|B(AlertSnitch)
    B --> |Save|C(MySQL/PG Database)
    C -.-|Graph|G[Grafana]
    C -.-|Query|D[MySQL/PG Client]
    style B fill:#f9f,stroke:#333,stroke-width:1px
    style C fill:#00A0A0,stroke:#333,stroke-width:1px
    style D fill:#00C000
    style G fill:#00C000

Local install

Simply install to your $GOPATH using your GO tools

$ go get gitlab.com/yakshaving.art/alertsnitch`

Requirements

To run AlertSnitch requires a MySQL or Postgres database to write to.

The database must be initialized with AlertSnitch model.

AlertSnitch will not become online until the model is up to date with the expected one. Bootstrapping scripts are provided in the [scripts][./script.d] folder.

Configuration

MySQL

For specifics about how to set up the MySQL DSN refer to Go MySQL client driver

This is a sample of a DSN that would connect to the local host over a Unix socket

export ALERTSNITCH_BACKEND="mysql"
export ALERTSNITCH_DSN="${MYSQL_USER}:${MYSQL_PASSWORD}@/${MYSQL_DATABASE}"

Postgres

export ALERTSNITCH_BACKEND="postgres"
export ALERTSNITCH_DSN="sslmode=disable user=${PGUSER} password=${PGPASSWORD} host=${PGHOST} database=${PGDATABASE}"

How to run

Running with Docker

Run using docker in this very registry, for ex.

$ docker run --rm \
    -p 9567:9567 \
    -e ALERTSNITCH_DSN \
    -e ALERTSNITCH_BACKEND \
    registry.gitlab.com/yakshaving.art/alertsnitch

Running Manually

  1. Open a terminal and run the following
  2. Copy the AlertSnitch binary from your $GOPATH to /usr/local/bin with sudo cp ~/go/bin/alertsnitch /usr/local/bin
  3. Now run AlertSnitch as with just alertsnitch
    • To just see the alerts that are being received, use the null backend with ALERTSNITCH_BACKEND=null

Setting up in AlertManager

Once AlertSnitch is up and running, configure the Prometheus Alert Manager to forward every alert to it on the /webhooks path.

---
receivers:
- name: alertsnitch
  webhook_configs:
    - url: http://<alertsnitch-host-or-ip>:9567/webhook

Then add the route

# We want to send all alerts to alertsnitch and then continue to the
# appropiate handler.
route:
  routes:
  - receiver: alertsnitch
    continue: true

Command line arguments

  • -database-backend sets the database backend to connect to, supported are mysql, postgres and null
  • -debug dumps the received WebHook payloads to the log so you can understand what is going on
  • -listen.address string address in which to listen for HTTP requests (default ":9567")
  • -version prints the version and exit

Environment variables

  • ALERTSNITCH_DSN required database connection query string
  • ALERTSNITCH_ADDR same as -listen.address
  • ALERTSNITCH_BACKEND same as -database-backend

Readiness probe

AlertSnitch offers a /-/ready endpoint which will return 200 if the application is ready to accept WebHook posts.

During startup AlertSnitch will probe the MySQL database and the database model version. If everything is as expected it will set itself as ready.

In case of failure it will return a 500 and will write the error in the response payload.

Liveliness probe

AlertSnitch offers a /-/health endpoint which will return 200 as long as the MySQL/Postgres database is reachable.

In case of error it will return a 500 and will write the error in the response payload.

Metrics

AlertSnitch provides Prometheus metrics on /metrics as per Prometheus convention.

Security

There is no offering of security of any kind. AlertSnitch is not ment to be exposed to the internet but to be executed in an internal network reachable by the alert manager.

Grafana Compatibility

AlertSnitch writes alerts in such a way that they can be explored using Grafana's MySQL/Postgres Data Source plugin. Refer to Grafana documentation for further instructions.

Testing locally

We provide a couple of Makefile tasks to make it easy to run integration tests locally, to get a full coverage sample run:

make bootstrap_local_testing
make integration
go tool cover -html=coverage.out
make teardown_local_testing

alertsnitch's People

Contributors

adarsh096 avatar afirth avatar marcusteixeira avatar pcarranza avatar wsilva avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.