jeffwillette / docker-alertd

Monitor docker stats and send alerts

Docker-Alertd

What Does It Do?

docker-alertd monitors docker containers on a host machine and sends alerts via email when usage limits have been breached.

Currently, alerts can be sent based on:

1. Container existence (regardless of running state)
2. Running state (running or existed)
3. Memory usage (in MB)
4. CPU Usage (as a percentage)
5. Minimum Process running in container

Step 1: Install

Method: Build from source

Assuming that you already have go installed on your machine, you can just go get it.

go get github.com/deltaskelta/docker-alertd

If you would like to download a pre-compiled binary, head to the release downloads

latest release - precompiled binary download

Step 2: Make a Configuration File

Docker-Alertd needs a configuration file in order to run, it will search the directory which contains the binary, and the home directory for a config file automatically. A special config file location can be specified with the --config flag.

A base config file can be generated with the command:

docker-alertd initconfig -d /path/to/config/location/

Example .docker-alertd.yaml file

---
# The duration and interations settings, if omitted, have a default value of 100ms between
# docker API calls and an indefinite number of iterations which will run the monitor forever
#duration: 100				# duration in ms between docker API calls
#iterations: 0				# number of iterations to run

# 'containers' is an array of dictionaries that each contain the name of a container to
# monitor, and the metrics which it should be monitored by. If there are no metrics
# present, then it will just be monitored to make sure that is is currently up.

# This will monitor only that the container exists, running or not...
# containers:
#   - name: mycontainer

containers:
  - name: container1
    expectedRunning: true

  - name: container2
    expectedRunning: true
    maxCpu: 20
    maxMem: 20
    minProcs: 4

# If email settings are present and active, then email alerts will be sent when an alert
# is triggered.
emailSettings:
  active: true
  smtp: smtp.someserver.com
  password: s00p3rS33cret
  port: 587
  from: [email protected]
  subject: "DOCKER_ALERTD"
  to:
    - [email protected]

Configuration Variables

Containers

duration: the duration to wait between docker engine API calls.

iterations: the number of iterations that docker-alertd should run (0 = run forever)

name: the container name or ID

maxCpu: the maximum cpu usage threshold (as a percentage), if the container uses more CPU, an alert will be triggered.

maxMem: the maximum memory usage threshold (in MB). If the container uses more system memory than this, an alert will be triggered.

minProcs: the minimum number of running processes (PID's) in the container. If a the number of running processes dips below this level (when a process fails), an alert will be triggered.

Email Settings

active: whether email settings are active or not

smtp: the smtp server to connect to

password: the password to use for smtp authentication

port: the port to connect to the smtp server

from: the email address to send from

subject: the subject line of emails sent

to: an array of email addresses to send the alerts to

Slack Settings

webhookURL: the webhookURL provided by slack after you authorize an app on a slack channel. See slack apps

Step 3: Run the program

Assuming docker-alertd is in your system path, and the config file is in the home directory or the same directory as the binary...

$ docker-alertd

if the binary is not on your $PATH then you will have to pass absolute paths.

$ /path/to/binary/docker-alertd --config ~/path/to/configuration/file/.docker-alertd.yaml

This will start the program and log the output to stdout. It can be stopped with CTRL-C.

Example Output:

2017/09/18 12:11:44 starting docker-alertd
------------------------------
2017/09/18 12:11:44 ALERT:
2017/09/18 12:11:44 container1: Existence check failure: Error: No such container: container1
2017/09/18 12:11:44 container2: Existence check failure: Error: No such container: container2
...

Step 4. Set up as a background process (optional)

If you wish to have docker-alertd run as a background process, it needs to be setup as a background process as per your operating system.

As A Systemd Service (for Linux systems with systemd)

If you have a systemd based system then you can refer to docker-alertd.service.example the example systemd service file and this tutorial

With Launchd (MacOS)

Refer to the launchd plist example file file and the launchd reference

With Sys V Init (various Linux systems without systemd)

Refer to this Sys V Init tutorial

Within docker

If you want to run docker-alertd from within Docker, you can do that. This requires mounting the docker socket into the container which comes with security risks because the container has access to the docker socket and therefore if it was compromised, it would have root access on the system google - mounting docker socket security.

That being said if you decide it is something you want to do in development or in a controlled production environment, you can use the image on the docker registry.

pull the image frm the registry

docker pull deltaskelta/docker-alertd

#or if you want to build your own tagged image, clone the repo and run
#docker build -t [your-tag] .

run the command to get a config file printed to stdout. You must save it, modify it to fit your needs, and then mount it into the container that will run docker-alertd.

NOTE: go-wrapper run is equivalient to docker-alertd if the binary was installed normally, so all of docker-alertds normal commands will follow go-wrapper run

docker run --rm  deltaskelta/docker-alertd go-wrapper run initconfig --stdout

and then run the app with the mounted config file and the mounted docker socket.

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v ~/.docker-aled.yaml:/root/.docker-alertd.yaml deltaskelta/docker-alertd

Testing Alert Authentication

Docker-Alertd comes with a testalert command which will search for a nonexistant container name and send an alert to the appropriate places, use this for testing that smtp or other authentication settings are correct

docker-alertd's Issues

Prevent crash when container does not exist or is removed

Currently, the program exits/crashes when a container does not exist or is removed.
In my opinion the service should keep running and only send an alert about the circumstance.

What do you think?

Refactor the way alerts are stored

formatting should be up to the alerter, so the error should store things as agnostically as possible and leave the formatting for the sender of the message.

• emails should be formatted so that if there is one message in the email, a blurb is put into the subject, if there is more than one message, it should say "multiple alerts" or something of the sort.

• look into the way that alerts (for each individual container) and alerts (for all containers combined) are formed. These might need to be different structs.

needs to send an alert when a container does not exist

Get a working Dockerfile and registry entry

There should be an option pull from the registry and run it from docker instead of going through the trouble of installing the binary and running it as a background process.

• Are there any security concerns with mounting the docker.sock?

• Will the configuration come from the Dockerfile or from a normal config file passed to the container?

Differentiation and customization of alerts / e-mails

These are actually two separate features.

1. Differentiation between e-mails

It would help to be able to define different subjects for e-mail alerts depending on the type of alert (failure or recovery). Example: "container failure" vs. "container recovery"

2. Customization of alerts

It would also be nice to provide a custom text or even use a template for individual content in alerts/e-mails.

Both features are nice to have.

Make example systemd service

Container existence check

I want to check for the existence of a container (regardless of state).
In my case the container is initially stopped / not running.
As far as I understand the correct way to do this is the following:

containers:
  - name: xyz

However, with this I´m getting alerts when the container starts and recovery alerts when the container stops. It looks like the above configuration causes docker-alertd to expect the container not to be running.

Do you have any idea what the problem could be?

Is there any way to get an alert on DNS ping failure

I have two DNS mapped to a running container. Now sometimes one url fails but other works as expected.Need to know if there is any option to monitor the same.

add slack/twitter notifications

notification type should be optionally set in settings.

What happens if the container listed in conf doesn't exist?

Alternative installation method

Disclaimer: I do not have any experience with go(lang).

Is there any possible alternative method to install this service? go get takes forever to download docker (which is a dependency I assume).

remove vendor directory

flag parsing is still referring to YAML files

Mke "alerted" bool so that duplicate emails can be avoided

it should also notify when things return to normal for the container

Alert should avoid sending false positives for short spikes

A CPU usage might spike for some short lived process. It should take this into account since it is getting CPU usage info every second or so.

Breaking changes in v0.4.0?

I downloaded the latest version in order to be able to only check for container existence. However, it seems like alerting does not work at all anymore. Sample config:

---
duration: 100				# duration in ms between docker API calls
iterations: 0				# number of iterations to run (0 = run forever)

containers:
  - name: foo
    expectedRunning: true

  - name: bar
    expectedRunning: true

  - name: baz

email:
  active: true
  smtp: some.example.com
  password: ...
  port: 587
  from: [email protected]
  subject: "monitoring alert"
  to:
    - [email protected]
    - [email protected]

When stopping a container expected to be running, nothing happens.
Any idea what could be wrong?

Get cobra/viper integrated

• remove the old config parser and replace with yaml (parsed by cobra)

There should be some way to limit alert emails so quick spikes will be ignored

Currently if there is a CPU spike, it will send an alert, then a recovery alert, and then another alert if there is an intermittent spike.

I think there should probably be something like a delay between an alert and a recovery check to see if a recovery is really warranted.

Another possible way around this issue is to only send an alert after a certain amount of consecutive checks showed an alert is really necessary. That would cut down the amount of times that an alert would be sent for a short spike

Make metrics optional

Really like the idea of this service. It is very minimal and serves a very specific purpose. Thanks for the work!

My requirement is to only be alerted when a container is completely absent. I can achieve this by setting min-procs to 1. Therefore I´m not interested in any cpu oder memory usage warnings.

Would it be possible to only watch the metrics which are actually defined in the config file?

disable the docker api on the host to see if the go library still works

tests

There is a test failing with the error:

2017/02/11 01:12:33 key 'min-procs' not found

It is probably being cause by the JSON templates in the test file being out of date

container existence and container running should be separate checks

right now a check for existence means that it will check to see that the container exists, running or not. There needs to be another state check where it checks for whether or not the container is currently running.