josef-friedrich / check_unattended_upgrades

Monitoring plugin to check automatic updates (unattended-upgrades) on Debian / Ubuntu.

check_unattended_upgrades

Note: The monitoring plugin is currently being rewritten from Shell (check_unattended_upgrades) to Python (check_unattended_upgrades.py). Use the Shell version until further notice.

Summary / Short description

Monitoring plugin to check automatic updates (unattended-upgrades) on Debian / Ubuntu.

Usage

check_unattended_upgrades v1.4
Copyright (c) 2015-2019 Josef Friedrich <[email protected]>

Monitoring plugin to check automatic updates (unattended-upgrades) on Debian / Ubuntu.


Usage: check_unattended_upgrades <options>

Options:
 -A, --anacron
	Check if the package 'anacron' is installed.
 -a, --autoclean
	Check if the configuration 'APT::Periodic::AutocleanInterval' is set
	properly.
 -c, --critical
        Time interval since the last execution to result in a
        critical state (time units depending on '--format').
 -D, --short-description
	Show a short description of this check plugin.
 -d, --download
	Check if the configuration 'APT::Periodic:Download-Upgradeable-Packages'
	is set properly.
 -e, --enable
	Check if the configuration 'APT::Periodic::Enable' is set properly.
 -f UNIT, --format UNIT
        Defines the unit for the numbers of '--warning' and '--
        critical', also the output of 'last-run'. Allowed values
        are: 'seconds', 'minutes', 'hours' and 'days', default:
        'seconds'.
 -h, --help
	Show this help message.
 -l, --lists
	Check if the configuration 'APT::Periodic::Update-Package-Lists' is set
	properly.
 -m, --mail
	Check if the configuration 'Unattended-Upgrade::Mail' is set properly.
 -n, --dry-run
	Check if 'unattended-upgrades --dry-run' is working. Warning: If you use
	this option the performance data last_ago is always 0 or near to 0.
 -p, --repo
	Check if 'Unattended-upgrades' is configured to include the specified
	custom repository.
 -R, --reboot
	Check if the machine needs a reboot.
 -r, --remove
	Check if the configuration 'Unattended-Upgrade::Remove-Unused-
	Dependencies' is set properly.
 -S, --security
	Check if 'Unattended-upgrades' is configured to handle security updates.
 -s, --sleep
	Check if the configuration 'APT::Periodic::RandomSleep' is set properly.
 -t, --systemd-timers
	Check if the appropriate Systemd timers are enabled ( apt-daily-upgrade.timer, apt-daily.timer ).
 -u, --unattended
	Check if the configuration 'APT::Periodic::Unattended-Upgrade' is set
	properly.
 -v, --version
	Show the version number.
 -w, --warning
        Time interval since the last execution to result in a
        warning state (time units depending on '--format').

Performance data:
  - last_ago
	  Time interval in seconds for last unattended-upgrades execution.
  - warning
	  Interval in seconds.
  - critical
	  Interval in seconds.

About file system permissions:
	The user which executes this plugin must have read permissions to this
	log file:

		/var/log/unattended-upgrades/unattended-upgrades.log

	To allow every user on your system to read the mentioned log file this
	permissions are recommended:

		751 (drwxr-x--x) /var/log/unattended-upgrades
		644 (-rw-r--r--) /var/log/unattended-upgrades/unattended-upgrades.log

Project pages

• https://github.com/Josef-Friedrich/check_unattended_upgrades
• https://exchange.icinga.com/joseffriedrich/check_unattended_upgrades
• https://exchange.nagios.org/directory/Plugins/Software/check_unattended_upgrades/details

Testing

make test

check_unattended_upgrades's Issues

Why do you only check the last line in the log?

Hi
First thanks for your work. I haven't really looked into the log file yet, but I'm wondering why you're only evaluating the last line of the log file. Is there a specific reason for this?

In my use case, I have a test instance in addition to a production instance. On the productive instance only security updates are installed automatically but on the test instance all updates are always installed. There I now had the case that the last line is only an INFO but in between there was a warning.

INFO Starting unattended upgrades script
INFO Allowed origins are: origin=Debian,codename=bullseye,label=Debian, origin=Debian,codename=bullseye,label=Debian-Security, origin=Debian,codename=bullseye-security,label=Debian-Security, o=jitsi.org, a=stable
INFO Initial blacklist: 
INFO Initial whitelist (not strict): 
WARNING Package jicofo has conffile prompt and needs to be upgraded manually
INFO package jicofo not upgraded
INFO package jitsi-meet not upgraded
INFO package jitsi-meet-prosody not upgraded
INFO package jitsi-meet-web not upgraded
INFO package jitsi-meet-web-config not upgraded
INFO package jitsi-videobridge2 not upgraded
INFO No packages found that can be upgraded unattended and no pending auto-removals
INFO Package jicofo is blacklisted.
INFO Package jitsi-meet is kept back because a related package is kept back or due to local apt_preferences(5).
INFO Package jitsi-meet-prosody is kept back because a related package is kept back or due to local apt_preferences(5).
INFO Package jitsi-meet-web is kept back because a related package is kept back or due to local apt_preferences(5).
INFO Package jitsi-meet-web-config is kept back because a related package is kept back or due to local apt_preferences(5).
INFO Package jitsi-videobridge2 is kept back because a related package is kept back or due to local apt_preferences(5).

Therefore I changed your script to check all lines between the last "Starting unattended upgrades script" and the end of the file.

while IFS= read -r line; do
    if [ ! -z "$(echo "$line" | grep -E ' (WARNING|ERROR) ')" ]; then
        # added -s check because it returns error on an empty file
        if [ ! -z "$(echo "$line" | grep ' ERROR ')" ]; then
            printf "CRITICAL - In the log file is an ERROR message.\n${line}"
            exit $STATE_CRITICAL
        fi

        # added -s check because it returns error on an empty file
        if [ ! -z "$(echo "$line" | grep ' WARNING ')" ]; then
            printf "WARNING - In the log file is a WARNING message.\n${line}"
            exit $STATE_WARNING
        fi
    fi
done < <(tail -n "+$(cat $LOG_FILE | grep -n 'Starting unattended upgrades script' | tail -n1 | cut -d: -f1)" $LOG_FILE)

(This needs bash instead of sh)

Are there any side effects there? Haven't tested it extensively yet. If you are interested I can also prepare a PR for you.

Systemd timers are not checked

On Ubuntu 16.04+ those APT::Conf variables

APT::Periodic::Enable
APT::Periodic::Unattended-Upgrade

are required but not enough. It is also essential that appropriate Systemd timers are started:

systemctl list-timers  | grep apt-daily.timer
systemctl list-timers  | grep apt-daily-upgrade.timer

And are enabled on startup:

systemctl is-enabled apt-daily.timer
systemctl is-enabled apt-daily-upgrade.timer

Also you can get their last execution times either from systemctl list-timers or with
ls -la /var/lib/systemd/timers/stamp-apt-daily*

WARNING issued for pending reboot

check_unattended_upgrades issues a WARNING if the last log line reads

2022-11-17 04:05:01,255 WARNING Shutdown msg: b"Shutdown scheduled for Thu 2022-11-17 05:05:00 UTC, use 'shutdown -c' to cancel."

This line should be ignored, right?

Ubuntu 18.04 "CRITICAL - Log file does not exist."

last weeks i started to add the check_unattended_upgrades check to my icinga. yesterday i had one working client till it got rebooted. now i have the same error:

CRITICAL - Log file (/var/log/unattended-upgrades/unattended-upgrades.log) does not exist.

on every machine in icinga. the log-file exists and can be read by everyone (permissions:644). i played with the permissions and found out that the folder /var/log/unattended-upgrades needs execute for everyone (751 instead of 750) to get read by the script and make it work again.

Ubuntu 16.04 log file reading bug

On Ubuntu 16.04, the plugin fails to read unattended-upgrades.log. The output suggests the file does not exist, however it has been confirmed to exist.

Error:
CRITICAL - Log file (/var/log/unattended-upgrades/unattended-upgrades.log) does not exist.

-rw-r--r-- 1 root root 6012 Apr 12 01:58 /var/log/unattended-upgrades/unattended-upgrades.log

Last parameter value applied to all in the configuration checks

In object ChecksCollection in function check_config, from line 820:

If the check is called with different parameters addressing the configuration checks somehow the last one is always passed to all.

So on the command line then for example results

/check_unattended_upgrades.py --mail root --lists 1 --remove true --verbose
CRITICAL: Configuration value for "APT::Periodic::Update-Package-Lists" unexpected! actual: 1 expected: true
CRITICAL: Configuration value for "Unattended-Upgrade::Mail" unexpected! actual: root expected: true
OK: Configuration value for "Unattended-Upgrade::Remove-Unused-Dependencies": true

and
/check_unattended_upgrades.py --mail root --lists 1 --verbose
CRITICAL: Configuration value for "APT::Periodic::Update-Package-Lists" unexpected! actual: 1 expected: root
OK: Configuration value for "Unattended-Upgrade::Mail": root

as well as
/check_unattended_upgrades.py --mail root
OK: Configuration value for "Unattended-Upgrade::Mail": root

just depending on the order in ChecksCollection - the function call in line 851 takes precedence over the call in line 850 and 849, the call in 850 takes precedence over the call in line 849...

Actual status is not checked.

It returns OK while last execution of unattended-upgrades wasn't successful.

# /usr/local/bin/check_unattended_upgrades  
OK - The last execution of “unattended-upgrades” was at 2019-01-12 14:35:58. | last_ago=307 warning=93600 critical=187200

 # tail /var/log/unattended-upgrades/unattended-upgrades.log -n3
2019-01-12 14:35:26,083 INFO Packages that will be upgraded: apparmor curl dirmngr distro-info-data gettext-base git git-man gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm gpgv initramfs-tools initramfs-tools-bin initramfs-tools-core libapache2-mod-php7.2 libapparmor1 libcurl3-gnutls libcurl4 libgd3 libmspack0 libmysqlclient20 libnss-systemd libnss3 libpam-systemd libperl5.26 libpython2.7 libpython2.7-minimal libpython2.7-stdlib libssl1.0.0 libssl1.1 libsystemd0 libudev1 libx11-6 libx11-data linux-gcp linux-headers-gcp linux-image-gcp mysql-client-5.7 mysql-client-core-5.7 mysql-server mysql-server-5.7 mysql-server-core-5.7 netplan.io nplan openssh-client openssh-server openssh-sftp-server openssl perl perl-base perl-modules-5.26 php7.2 php7.2-bz2 php7.2-cli php7.2-common php7.2-curl php7.2-gd php7.2-imap php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-recode php7.2-tidy php7.2-xml php7.2-xmlrpc php7.2-zip python-requests python2.7 python2.7-minimal python3-requests systemd systemd-sysv tzdata udev
2019-01-12 14:35:26,100 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2019-01-12 14:35:58,860 ERROR Cache has broken packages, exiting