makhomed / automon

/var/log/messages monitoring tool

automon

/var/log/messages monitoring tool

This tool will monitor /var/log/messages file and send alerts via Telegram if detects any anomalies.

Installation

• cd /opt
• git clone https://github.com/makhomed/automon.git automon

Upgrade

• cd /opt/automon
• git pull

Configuration

• vim /opt/automon/automon.conf
• write to config something like this:

host localhost
host example.com one-line description of this host

Configuration file allow comments, from symbol # to end of line.

Configuration file has only four directives: host, log, alert and delay.

host directive has syntax: host <hostname>[:port] [description]. <hostname> part is requred, it may be domain name or ip address. port is optional, by default used port 22. description also optional. If hostname is localhost or 127.0.0.1 - direct access to log files will be used, else log files will be acessed via ssh.

log directive has syntax log </path/to/logfile>. Default value of log directive is /var/log/messages. It can be redefined to any other value, for example, /var/log/syslog. Value of log directive will be used for all below host declarations. For example:

host centos1
host centos2

log var/log/syslog

host debian1
host debian2

alert directive defines path to alert program, default value is /opt/automon/bin/alert-via-telegram. Program /opt/automon/bin/alert-via-telegram included in automon and send alerts to Telegram via https://pypi.python.org/pypi/telegram-send script. alert program receive one argument - full name of file with generated alert text. See source of /opt/automon/bin/alert-via-telegram program for details. Using /opt/automon/bin/alert-via-telegram as example you can write own alert program for sending alerts via email or SMS or via any other way.

delay directive defines delay between two automon scans is daemon mode. By default delay is 600 seconds.

Global ignore patterns defined if files in directory /opt/automon/ignore.d. This directory included in automon repository. Local ignore patterns should be defined in files in directory /opt/automon/local-ignore.d. This directory is not included in automon repository and should be created manually. Host-specific ignore paterns should be defined in files in subdirectories named as host name + ".d". For example, for host localhost ignore patterns should be defined in files localed inside directory /opt/automon/local-ignore.d/localhost.d, for host example.com ignore patterns should be defined in directory /opt/automon/local-ignore.d/example.com.d.

Each line in ignore file should be python regular expression, symbols ^ at start and $ at end will be added automatically. If first non-whitespace symbol of line is # - such line considered as comment and will be ignored in pattern matching.

Command line arguments

automon [-c /path/to/configuration/file.conf] [mode]

automon has optional command line agrument -c </path/to/configuration/file.conf>. If agrument -c not defined - by default will be used config /opt/automon/automon.conf.

automon also has ohe optional positional argument mode. Allowed values are daemon, once and debug. daemon mode useful for running automon as systemd service. In this mode automon will be run forever with delay seconds delay between two scans of hosts defined in configuration. once mode is useful for running automon from cron. In once mode automon run once and exit. debug mode useful for debug, in this mode no alerts will be send and no logscan state will be readed or saved. In debug mode alert will be printed to stdout and automon will exit. In daemon and once modes alerts will be send to system administrator via alert program.

Before first run

Before first run you need to create Telegram bot and configure telegram-send script. Detalis see in https://pypi.python.org/pypi/telegram-send documentation.

Secure Shell

For work you need to generate private ssh key on automon server with comamnd ssh-keygen -t rsa and copy public key from /root/.ssh/id_rsa.pub to /root/.ssh/authorized_keys on monitored servers. Also you need to check connection with monitored server with command ssh example.com and answer yes to ssh question:

# ssh example.com
The authenticity of host 'example.com' can't be established.
ECDSA key fingerprint is SHA256:/cYI0bJzEX+CF3DhGEUQ+ZeGFmMzEJYAt3C15450zKs.
ECDSA key fingerprint is MD5:44:20:bd:f5:aa:a7:52:ac:c5:19:e5:e0:28:2b:90:49.
Are you sure you want to continue connecting (yes/no)? yes

Automation via cron

Create configuration file /opt/automon/cron.conf and define hosts to check inside it. After it configure cron job, for example, in file /etc/cron.d/automon:

0 * * * * root /opt/automon/automon -c /opt/automon/cron.conf once

Automation via systemd service

Create configuration file /opt/automon/automon-daemon.conf and define hosts to check inside it. After it create systemd service, for example, in file /etc/systemd/system/automon.service:

[Unit]
Description=automon
After=network-online.target

[Service]
ExecStart=/opt/automon/automon -c /opt/automon/automon-daemon.conf daemon
Restart=always
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

Note: in new versions of systemd StartLimitInterval renamed to StartLimitIntervalSec and moved from [Service] to [Unit] section. See details at https://selivan.github.io/2017/12/30/systemd-serice-always-restart.html

After this you need to start service:

• systemctl daemon-reload
• systemctl enable automon
• systemctl start automon
• systemctl status automon

If all ok you will see what service is enabled and running.

Automation via multiple systemd services

Create multiple configuration file /opt/automon/service1.conf, /opt/automon/service2.conf, ... and define hosts to check inside it. After it create systemd service, for example, in file /etc/systemd/system/[email protected]:

[Unit]
Description=automon %I
After=network-online.target

[Service]
ExecStart=/opt/automon/automon -c /opt/automon/%i.conf daemon
Restart=always
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

Note: in new versions of systemd StartLimitInterval renamed to StartLimitIntervalSec and moved from [Service] to [Unit] section. See details at https://selivan.github.io/2017/12/30/systemd-serice-always-restart.html

After this you need to start services:

• systemctl daemon-reload
• systemctl enable automon@service1
• systemctl enable automon@service2
• ...
• systemctl start automon@service1
• systemctl start automon@service2
• ...
• systemctl status automon@service1
• systemctl status automon@service2
• ...

If all ok you will see what automon services are enabled and running.