rackerlabs / auter Goto Github PK

View Code? Open in Web Editor NEW

64.0 64.0 18.0 328 KB

Automatic updates for RHEL, Debian, and their derivatives, with the ability to run pre/post hooks & reboot afterwards.

License: Apache License 2.0

Makefile 4.50% Shell 85.53% Roff 9.98%

patching shell-scripts

auter's People

Contributors

Stargazers

Watchers

Forkers

reaperzn beerey piersc rjmco mark-hyde ruthealee popindavibe pombredanne samjsharpe costingalan alexxnica kryndex pingue haho16 hardxstyle im2kl bengtfredh abotelho-cbn

auter's Issues

change help exit code

Currently any auter errors that call the print_help function (All 2 of them) have a subsequent exit call which returns exit code 0:

paolo@paolo-Desktop:~/Scripts/auter$ grep -n "print_help " auter
151:([[ ! $1 ]] || [[ ! $1 =~ "-" ]]) && print_help && exit 0
164:    -h|--help) print_help ; exit 0; shift;;

Is it not standard that when calling the help function most applications use exit with non-zero? Also does it not make more sense to move the exit call to the print_help function?

Bring back the transaction functionality

Hey guys, I will be working on an alternative of yum transactions, which will have essentially the same effect. The idea here will be a new variable in the config "$USETRANSACTIONS"

During the prep phase, if the $USETRANSACTIONS variable is set to yes then the downloadonly option will download packages to a specific auter location
During the install phase, if the $USETRANSACTIONS variable is set to yes then auter will install all rpms from that location

Log name is package manager specific

The log file name /var/lib/auter/last-prep-yum-output-default (https://github.com/rackerlabs/auter/blob/master/auter#L122) is package manager specific and hard coded. It should probably just not have the PM name in it at all.

Auter errors when saved patches are already installed

If we have the ONLYINSTALLFROMPREP set to yes we can get into the following situation:

auter --prep is run which downloads the rpm files locally
auter --apply is run or a user manually updates the system which includes all the downloaded packages
auter --apply is run a second time or after the manual update, you will get the following error:

[root@paolo-auter-test7 ~]# auter --apply
INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
INFO: Applying updates
ERROR: Updates failed. Exiting.

There are a few issues here:

We are relying on a directory listing to deturmin if there are updates to be applied:

  if [[ "${ONLYINSTALLFROMPREP}" == "yes" ]]; then
    RC=0
    if [[ $(ls -1 ${DOWNLOADDIR}/${CONFIGSET}/*.rpm | wc -l) -gt 0 ]]; then
      RPMS="${DOWNLOADDIR}/${CONFIGSET}/*.rpm"
      RC=100
    fi
    # When passing RPMs to dnf/yum, the update verb won't install any that aren't already
    # installed (i.e. dependencies of other packages). Instead we need to use install.
    UPDATEACTION="install"
  else
    local RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} &>/dev/null; echo $?)
    UPDATEACTION="update"
  fi

The actual error is because of the following lines of code in the apply_updates function:

    local HISTORY_BEFORE=$(${PACKAGE_MANAGER} history list)
...
...
    local HISTORY_AFTER=$(${PACKAGE_MANAGER} history list)

    if [[ "${HISTORY_BEFORE}" == "${HISTORY_AFTER}" ]]; then
      logit "ERROR: Updates failed. Exiting."
      quit 3
    fi

Example:

INFO: Running with: /usr/bin/auter --prep
INFO: Running in an interactive shell, disabling all random sleeps
INFO: Updates downloaded to /var/cache/auter/default


[root@paolo-auter-test7 ~]# ll /var/cache/auter/default
total 1536
-rw-r--r-- 1 root root 448108 Jan 18 12:57 openssh-6.6.1p1-33.el7_3.x86_64.rpm
-rw-r--r-- 1 root root 657552 Jan 18 12:58 openssh-clients-6.6.1p1-33.el7_3.x86_64.rpm
-rw-r--r-- 1 root root 450388 Jan 18 12:58 openssh-server-6.6.1p1-33.el7_3.x86_64.rpm

[root@paolo-auter-test7 ~]# auter --apply
INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
INFO: Applying updates
INFO: Updates complete (yum Transaction ID : 20). You may need to reboot for some updates to take effect


[root@paolo-auter-test7 ~]# auter --apply
INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
INFO: Applying updates
ERROR: Updates failed. Exiting.

To remedy this in the following way:

if [[ "${ONLYINSTALLFROMPREP}" == "yes" ]]; then
  RC=0
    if [[ $(ls -1 ${DOWNLOADDIR}/${CONFIGSET}/*.rpm | wc -l) -gt 0 ]]; then
      RPMS="${DOWNLOADDIR}/${CONFIGSET}/*.rpm"
      RC=$(${PACKAGE_MANAGER} check-update $(find ${DOWNLOADDIR}/${CONFIGSET}/*.rpm -printf "%f\n" | sed 's/.rpm$//g' ))
    fi
    # When passing RPMs to dnf/yum, the update verb won't install any that aren't already
    # installed (i.e. dependencies of other packages). Instead we need to use install.
    UPDATEACTION="install"
  else
    local RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} &>/dev/null; echo $?)
    UPDATEACTION="update"
fi

Any thoughts or alternative suggestions?

Create contrib scripts for configsnap

Please create the following scripts for configsnap to be used by auter:

configsnap.pre-apply
configsnap.post-apply
configsnap.pre-reboot
configsnap.post-reboot

These scripts should be placed in a new directory contrib/configsnap

auter pid file

if auter was running and for some reason is manually killed with kill -9 then auter status still shows auter is currently enabled and running

We should get the print_status function to assess if the pid file exists and if that pid is still running.

Create a crashdump type option for failures

Auter overwrites some useful "log" files in /var/lib/auter each time it runs. It may be worth adding a crashdump option which will copy the files from /var/lib/auter and also a separate directory or at least rename them to include a timestamp.

duplicate reboot issue

If we have AUTOREBOOT set to "yes" and the --reboot option is also specified in the command line then the reboot function seems to be called twice. This should be avoided as the pre-reboot scripts will also be executed twice followed by an error stating that another reboot is already in progress.

Add README.md for the tests directory

This should include a brief overview of the automated testing as well as a short description of each test script that is executed.

Add more cron commented examples.

It would be nice if we could have a commented out cronjob in the auter.cron file to apply updates every day - just makes it nice and easy to uncomment for the "default" schedule. Something like this:

# 30 2 * * *    root /usr/bin/auter --apply                             # Every day at 2.30am

Maybe also an example to force a reboot with the --reboot flag since that's not in the example cron either?

Document when autoreboot will occur

If updates are not applied successfully, the server will not be rebooted by design. This isn't explicit in the documentation anywhere so we should make this more clear for end users.

Clean up prep cache post apply

After an apply operation the rpm files in /var/cache/auter/default are left. Is there a method to clean up these files? This would make my reporting of which files are pending more accurate (rather than having to scrape and parse the log output) to do a directory listing of each device.

Auter waits forever for yum locks

If there's a yum lock, auter will wait forever until the lock is removed. This could have the unintended consequence of an update out of schedule, worse an automatic post update reboot out of schedule (if enabled).

Steps to reproduce:

Create a yum lock by starting a yum upgrade and leave it at the prompt
Let Auter to run via cron
Auter will wait for the lock to be removed
Exit out of the yum upgrade some time later
Auter will apply all updates (and reboot if set) out of schedule

Pre and post prep scripts

I finally remembered the purpose for #17 and #19

So can we please add at least a post-prep script but preferably a pre-prep script too. The main purpose of this would be to take an action based on the patches that were downloaded.

Example:
Patches are downloaded on Monday:

post-prep script does a directory listing of /var/cache/auter/{{PROFILE}}/ and emails the list of patches to be applied to the administrator.

Standardization of error logging categories

We should adjust all logs that are sent to the logit function to begin with one of the following:
INFO:
WARNING:
ERROR:

Example:
INFO: "Running script ${SCRIPT}" <-- custom pre/post auter script running
INFO: "Updates downloaded"
INFO: "No updates are available to be downloaded."
INFO: "Applying updates"
INFO: "Updates complete."
INFO: "No updates are available to be applied."
INFO: "Adding post-reboot-hook."
INFO: "Rebooting server"
INFO: "Removed post-reboot hook"
INFO: "auter is currently.""
INFO: "auter enabled"
INFO: "auter disabled"
INFO: "auter disabled. Please run auter --enable to enable automatic updates."
INFO: "Running with: ."
INFO: "Running in an interactive shell.*"

ERROR: "Updates failed. Exiting."
ERROR: "Exit status._Exiting." <-- Error while downloading/checking for updates
ERROR: "Cannot find yum or dnf" <-- As it states
ERROR: "exited with non-zero exit code" <-- Error with custom pre/post auter script
ERROR: "FATAL ERROR: auter DATADIR._does not exist."
ERROR: "Custom config file.*does not exist"
ERROR: "exists but the execute bit is not set." <-- Exexution bit missing from custom pre/post auter script

WARNING: "downloadonly option is not available"
WARNING: "auter is already running
WARNING: "Using default config values."
WARNING: "You must specify the CONFIGSET variable.*"

Auter --apply tries to install previous prepared rpms

If auter is set to only install from prep, after a prep has completed and an apply is run, if a second apply is attempted, in some cases, auter will fail:

[root@auter-test7 ~]# auter --apply
INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
INFO: Running Pre-Apply script /etc/auter/pre-apply.d/01-configsnap-pre
INFO: Applying updates
ERROR: Updates failed. Exiting.

In this case there was a long list of updates from a previous prep:

[root@auter-test7 ~]# ls -lah /var/cache/auter/default/ | wc -l
82

By setting the set -x option I found that this was the command that was being run:

[root@auter-test7 ~]# yum install -y --disableexcludes=all /var/cache/auter/default/bind-libs-9.9.4-50.el7_3.1.x86_64.rpm /var/cache/auter/default/bind-libs-lite-9.9.4-50.el7_3.1.x86_64.rpm /var/cache/auter/default/bind-license-9.9.4-50.el7_3.1.noarch.rpm /var/cache/auter/default/bind-utils-9.9.4-50.el7_3.1.x86_64.rpm /var/cache/auter/default/ca-certificates-2017.2.14-70.1.el7_3.noarch.rpm /var/cache/auter/default/chkconfig-1.7.2-1.el7_3.1.x86_64.rpm /var/cache/auter/default/configsnap-0.13-1.el7.noarch.rpm /var/cache/auter/default/device-mapper-1.02.135-1.el7_3.5.x86_64.rpm /var/cache/auter/default/device-mapper-event-1.02.135-1.el7_3.5.x86_64.rpm /var/cache/auter/default/device-mapper-event-libs-1.02.135-1.el7_3.5.x86_64.rpm /var/cache/auter/default/device-mapper-libs-1.02.135-1.el7_3.5.x86_64.rpm /var/cache/auter/default/dmidecode-3.0-2.1.el7_3.x86_64.rpm /var/cache/auter/default/dracut-033-463.el7_3.2.x86_64.rpm /var/cache/auter/default/dracut-config-rescue-033-463.el7_3.2.x86_64.rpm /var/cache/auter/default/dracut-network-033-463.el7_3.2.x86_64.rpm /var/cache/auter/default/epel-release-7-10.noarch.rpm /var/cache/auter/default/firefox-52.2.0-1.el7.centos.x86_64.rpm /var/cache/auter/default/firewalld-0.4.3.2-8.1.el7_3.3.noarch.rpm /var/cache/auter/default/firewalld-filesystem-0.4.3.2-8.1.el7_3.3.noarch.rpm /var/cache/auter/default/gawk-4.0.2-4.el7_3.1.x86_64.rpm /var/cache/auter/default/glibc-2.17-157.el7_3.5.x86_64.rpm /var/cache/auter/default/glibc-common-2.17-157.el7_3.5.x86_64.rpm /var/cache/auter/default/graphite2-1.3.10-1.el7_3.x86_64.rpm /var/cache/auter/default/grubby-8.28-21.el7_3.x86_64.rpm /var/cache/auter/default/gtk3-3.14.13-20.el7_3.1.x86_64.rpm /var/cache/auter/default/initscripts-9.49.37-1.el7_3.1.x86_64.rpm /var/cache/auter/default/irqbalance-1.0.7-6.el7_3.1.x86_64.rpm /var/cache/auter/default/jasper-libs-1.900.1-30.el7_3.x86_64.rpm /var/cache/auter/default/kernel-3.10.0-514.26.2.el7.x86_64.rpm /var/cache/auter/default/kernel-tools-3.10.0-514.26.2.el7.x86_64.rpm /var/cache/auter/default/kernel-tools-libs-3.10.0-514.26.2.el7.x86_64.rpm /var/cache/auter/default/kpartx-0.4.9-99.el7_3.3.x86_64.rpm /var/cache/auter/default/libblkid-2.23.2-33.el7_3.2.x86_64.rpm /var/cache/auter/default/libc-client-2007f-16.el7.x86_64.rpm /var/cache/auter/default/libgudev1-219-30.el7_3.9.x86_64.rpm /var/cache/auter/default/libmount-2.23.2-33.el7_3.2.x86_64.rpm /var/cache/auter/default/libnetfilter_conntrack-1.0.6-1.el7_3.x86_64.rpm /var/cache/auter/default/libtirpc-0.2.4-0.8.el7_3.x86_64.rpm /var/cache/auter/default/libtomcrypt-1.17-25.el7.x86_64.rpm /var/cache/auter/default/libtommath-0.42.0-5.el7.x86_64.rpm /var/cache/auter/default/libtool-ltdl-2.4.2-22.el7_3.x86_64.rpm /var/cache/auter/default/libuuid-2.23.2-33.el7_3.2.x86_64.rpm /var/cache/auter/default/lvm2-2.02.166-1.el7_3.5.x86_64.rpm /var/cache/auter/default/lvm2-libs-2.02.166-1.el7_3.5.x86_64.rpm /var/cache/auter/default/NetworkManager-1.4.0-20.el7_3.x86_64.rpm /var/cache/auter/default/NetworkManager-libnm-1.4.0-20.el7_3.x86_64.rpm /var/cache/auter/default/NetworkManager-team-1.4.0-20.el7_3.x86_64.rpm /var/cache/auter/default/NetworkManager-tui-1.4.0-20.el7_3.x86_64.rpm /var/cache/auter/default/nss-3.28.4-1.2.el7_3.x86_64.rpm /var/cache/auter/default/nss-sysinit-3.28.4-1.2.el7_3.x86_64.rpm /var/cache/auter/default/nss-tools-3.28.4-1.2.el7_3.x86_64.rpm /var/cache/auter/default/nss-util-3.28.4-1.0.el7_3.x86_64.rpm /var/cache/auter/default/ntpdate-4.2.6p5-25.el7.centos.2.x86_64.rpm /var/cache/auter/default/ntsysv-1.7.2-1.el7_3.1.x86_64.rpm /var/cache/auter/default/openssh-6.6.1p1-35.el7_3.x86_64.rpm /var/cache/auter/default/openssh-clients-6.6.1p1-35.el7_3.x86_64.rpm /var/cache/auter/default/openssh-server-6.6.1p1-35.el7_3.x86_64.rpm /var/cache/auter/default/php-PHPMailer-5.2.24-1.el7.noarch.rpm /var/cache/auter/default/polkit-0.112-12.el7_3.x86_64.rpm /var/cache/auter/default/pulseaudio-libs-6.0-9.el7_3.x86_64.rpm /var/cache/auter/default/python2-boto-2.45.0-3.el7.noarch.rpm /var/cache/auter/default/python-dmidecode-3.10.13-12.el7_3.x86_64.rpm /var/cache/auter/default/python-firewall-0.4.3.2-8.1.el7_3.3.noarch.rpm /var/cache/auter/default/python-perf-3.10.0-514.26.2.el7.x86_64.rpm /var/cache/auter/default/qemu-img-1.5.3-126.el7_3.10.x86_64.rpm /var/cache/auter/default/rdma-7.3_4.7_rc2-6.el7_3.noarch.rpm /var/cache/auter/default/rpcbind-0.2.0-38.el7_3.1.x86_64.rpm /var/cache/auter/default/selinux-policy-3.13.1-102.el7_3.16.noarch.rpm /var/cache/auter/default/selinux-policy-targeted-3.13.1-102.el7_3.16.noarch.rpm /var/cache/auter/default/sudo-1.8.6p7-23.el7_3.x86_64.rpm /var/cache/auter/default/systemd-219-30.el7_3.9.x86_64.rpm /var/cache/auter/default/systemd-libs-219-30.el7_3.9.x86_64.rpm /var/cache/auter/default/systemd-python-219-30.el7_3.9.x86_64.rpm /var/cache/auter/default/systemd-sysv-219-30.el7_3.9.x86_64.rpm /var/cache/auter/default/tuned-2.7.1-3.el7_3.2.noarch.rpm /var/cache/auter/default/tzdata-2017b-1.el7.noarch.rpm /var/cache/auter/default/util-linux-2.23.2-33.el7_3.2.x86_64.rpm /var/cache/auter/default/wordpress-4.8.1-1.el7.noarch.rpm /var/cache/auter/default/xfsprogs-4.5.0-10.el7_3.x86_64.rpm

The summary of that yum command is:

Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:3.10.0-514.26.2.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================
 Package          Arch             Version                          Repository                                    Size
=======================================================================================================================
Reinstalling:
 kernel           x86_64           3.10.0-514.26.2.el7              /kernel-3.10.0-514.26.2.el7.x86_64           148 M

Transaction Summary
=======================================================================================================================
Reinstall  1 Package

Total size: 148 M
Installed size: 148 M
Downloading packages:
Running transaction check
Running transaction test


Transaction check error:
  package kernel-3.10.0-514.26.2.el7.x86_64 is already installed

Error Summary
-------------

This is essentially because in the apply_updates function we have the following logic:

  if [[ "${ONLYINSTALLFROMPREP}" == "yes" ]]; then
    RC=0
    if [[ $(ls -1 ${DOWNLOADDIR}/${CONFIGSET}/*.rpm | wc -l) -gt 0 ]]; then
      RPMS="${DOWNLOADDIR}/${CONFIGSET}/*.rpm"
      RC=100
    fi 
    # When passing RPMs to dnf/yum, the update verb won't install any that aren't already
    # installed (i.e. dependencies of other packages). Instead we need to use install.
    UPDATEACTION="install"
  else
    local RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} &>/dev/null; echo $?)
    UPDATEACTION="update"
  fi

We are statically setting the RC to 100 (return code of yum check-update when there are updates to be applied) if there are files in the auter rpm cache. There is no test to see if those packages require updating. We should look into adjusting the line to replicate the same actions taken if ONLYINSTALLFROMPREP is set to no:

proposed code:

  if [[ "${ONLYINSTALLFROMPREP}" == "yes" ]]; then
    RC=0
    if [[ $(ls -1 ${DOWNLOADDIR}/${CONFIGSET}/*.rpm | wc -l) -gt 0 ]]; then
      RPMS="${DOWNLOADDIR}/${CONFIGSET}/*.rpm"
      RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} $(rpm -qp ${RPMS})&>/dev/null; echo $?)
    fi
    # When passing RPMs to dnf/yum, the update verb won't install any that aren't already
    # installed (i.e. dependencies of other packages). Instead we need to use install.
    UPDATEACTION="install"
  else
    local RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} &>/dev/null; echo $?)
    UPDATEACTION="update"
  fi

With that change I get the following:

[root@auter-test7 ~]# auter --apply
INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
INFO: No updates are available to be applied.

However if I downgrade a single package in that list I still get the same problem:

INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
INFO: Running Pre-Apply script /etc/auter/pre-apply.d/01-configsnap-pre
INFO: Applying updates
ERROR: Updates failed. Exiting.

Manual retry using yum:

Dependencies Resolved

=======================================================================================================================
 Package            Arch             Version                        Repository                                    Size
=======================================================================================================================
Updating:
 xfsprogs           x86_64           4.5.0-10.el7_3                 /xfsprogs-4.5.0-10.el7_3.x86_64              3.9 M
Reinstalling:
 kernel             x86_64           3.10.0-514.26.2.el7            /kernel-3.10.0-514.26.2.el7.x86_64           148 M

Transaction Summary
=======================================================================================================================
Upgrade    1 Package
Reinstall  1 Package

Total size: 152 M
Downloading packages:
Running transaction check
Running transaction test


Transaction check error:
  package kernel-3.10.0-514.26.2.el7.x86_64 is already installed

Any ideas for a proper fix for this?

Script phases in log messages

We should include the script phase in the log message in case the user changes the directory in the config file.

Currently:

auter: Running script /var/lib/auter/post-apply.d/getData-post

Should be:

auter: Running post-apply script /var/lib/auter/post-apply.d/getData-post

Unintended repo connection in apply_updates function

In the apply_updates function there is a code block to check if the local rpms are still valid updates. This should work even if the repos are not available. Currently there is no issue with the actual "yum update" command however during the check part, a "yum check-update" is executed which does query the repos.

Pre/Post prep scripts

Not sure why this was missed but it might be useful to have pre/post prep script hoots too.

The stderr output from the ls should be suppressed

~]# auter --apply
INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
ls: cannot access /var/cache/auter/default/*.rpm: No such file or directory
INFO: No updates are available to be applied.

find command needs correcting

I will look into this at a later stage but we need to quote the find "*.rpm" string to avoid bash interpreting this when running manually

Create contrib scripts for email notifications

Please create the following scripts for auter to send email notifications:

notify.post-prep

Should send an email to a pre-configured address with the pending updates and the scheduled install date and time

notify.pre-apply

should send an email to notify that updates are being applied

notify.pre-reboot

should send an email to notify that the server is about to be rebooted

notify.post-reboot

should send an email to notify that the server is back up

These scripts should be placed in a new directory contrib/email-notifications

Add text to "enable" file.

The "enable" file in /var/lib/auter/enable is 0kb. This file should have some text to ensure that it's not deleted by accident.

DOWNLOADOPTION in auter.aptModule needs to change to array

auter/auter.aptModule

Line 38 in a240f29

DOWNLOADOPTION="-o dir::cache::archives=${DOWNLOADDIR}/${CONFIGSET}"

Be --quiet option for auter

I have auter in a cron job, including auto reboot. Every time it installs updates, I receive an email like

Shutdown scheduled for Mon 2018-04-16 05:00:11 CEST, use 'shutdown -c' to cancel.

Silencing this via > /dev/null stdout redirection does not work. Silencing this via 2> /dev/null stderr redirection works but would also suppress real errors I would like to see. Maybe auter could have a --quiet flag or configuration option and silences shutdown when given.

conditionalReboot does not detect 'outdated' running kernel

On centos/rhel 6. if the kernel has been updated outside of an auter run then the conditionalReboot script in the devel branch will not detect that a reboot is required.

https://github.com/rackerlabs/auter/blob/develop/contrib/conditionalReboot/99-conditionalReboot.sh

perhaps add some logic like:

if [[ "$(grubby --default-kernel |sed 's#.*vmlinuz-##g')" != "$(uname -r)" ]]
then
  echo reboot needed
else
  echo "we're OK"
fi

problem when no rpms are found

[root@auter-training-7 ~]# auter --apply
INFO: Running with: /usr/bin/auter --apply
INFO: Running in an interactive shell, disabling all random sleeps
ls: cannot access /var/cache/auter/default/*.rpm: No such file or directory
INFO: No updates are available to be applied.

If no updates during a prep, log it

This caused a bit of confusion. If there's no updates available in the prep operation, can this be logged?

Capturing errors from package manager

During prep, last-prep-config is created containing only stdout from the yum/dnf run. stderr is not captured - it'll currently get printed to screen when running on a tty, or use cron's behaviour, which by default is to send an email. This is the kind of message that could be on stderr, but won't prevent the pkgmanager command from succeeding:

There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).

Apply behaves differently - all output is discarded from the update command, but the yum history command output is saved, which includes any errors.

whitespace in cron file issue

It appears that tabs are not being properly handled in the /etc/cron.d/auter file. When the cron job tries to run, the command seems to be appending the full line as an option when tabs are used. This causes the job to fail. Here is an example of the cron log entries.

Dec 13 11:00:01 staging CROND[24583]: (root) CMD (/usr/bin/auter --prep^I^I^I^I # Prep Every Wednesday at 11:00)
Dec 13 12:00:01 staging CROND[25482]: (root) CMD (/usr/bin/auter --apply^I^I^I # Apply Every Wednesday at 12:00 (AUTOREBOOT is set in /etc/auter/auter.conf))

I checked the cache, logs and uptime to confirm that neither the --prep or --apply succeeded.

conditionalReboot issues

First of all, I'm still fan of the conditinalReboot-script created in #152 and #144.

There is still one bug in the script, it cannot find the binary lsof on my system (OracleLinux/CentOS/RedHat 7) as the lsof-binary resides in /usr/sbin and that directory isn't in the crontab-path. So either the PATH should be temporarily fixed to include /usr/sbin or the lsof-binary should be called directly from /usr/sbin/lsof.

But at the end of the script it complains about the reboot-call:

May 30 09:09:13 <server> auter: INFO: Running Post-Apply script /etc/auter/post-apply.d/conditional-reboot.sh
May 30 09:09:14 <server> auter: /etc/auter/post-apply.d/conditional-reboot.sh assessed that the server needs to be rebooted. The assessments that triggered this requirement are:
May 30 09:09:14 <server> auter: Rebooting because detected deleted libraries
May 30 09:09:14 <server> auter: Rebooting because package kernel-uek-4.1.12-124.15.2.el7uek.x86_64 was updated and is in the /etc/auter/post-apply.d/conditional-reboot.sh APPLIST config
May 30 09:09:14 <server> auter: Rebooting server
May 30 09:09:14 <server> auter: ERROR: auter is already running or /var/run/auter/auter.pid exists.
May 30 09:09:14 <server> auter: ERROR: Post-Apply script /etc/auter/post-apply.d/conditional-reboot.sh exited with non-zero exit code 6. Aborting auter run.

And that's correct because auter is still running the post-script and it want's to run itself (again) with the reboot-call. Maybe it's a good idea to just call the reboot-function from the conditional reboot script but that would make it harder to debug it of course if ran standalone and not part of auter itself.

dnf issue with ${DOWNLOADDIR}/${CONFIGSET}

When using the following options in combination we have a slight issue (This is the default options at install time)

PREDOWNLOADUPDATES="yes"
ONLYINSTALLFROMPREP="no"

The problem is that the ${DOWNLOADDIR}/${CONFIGSET} directory is only created if ONLYINSTALLFROMPREP is set to yes (See line 333)

if [[ "${ONLYINSTALLFROMPREP}" == "yes" ]]; then
  if [[ ! -d "${DOWNLOADDIR}/${CONFIGSET}" ]]; then
    install -m 755 -o root -g root -d ${DOWNLOADDIR}/${CONFIGSET}
  elif [[ $(stat -c %G%U%a "${DOWNLOADDIR}") != rootroot[0-9][0-9][0,1,4,5] ]]; then
    logit "ERROR: ${DOWNLOADDIR}/${CONFIGSET} does not have the correct permissions."
    quit 3
  fi
fi

However in the prep function, if PREDOWNLOADUPDATES==yes and PACKAGE_MANAGER == dnf we have the following line:

[[ "${PACKAGE_MANAGER}" == "dnf" ]] && find /var/cache/dnf -name *.rpm -exec mv {} ${DOWNLOADDIR}/${CONFIGSET} \;

In the default config case we are moving all rpms to ${DOWNLOADDIR}/${CONFIGSET} however that directory is never created so we land up with the following situation:

Before I run prep:

# dnf check-update 
Last metadata expiration check: 2:29:07 ago on Tue Jan 24 08:39:37 2017.

openssl.x86_64                                                      1:1.0.2j-3.fc24                                                 updates
openssl-libs.x86_64                                                 1:1.0.2j-3.fc24                                                 updates

# ll /var/cache/auter/
total 0

I then run prep:

# auter --prep
INFO: Running with: /usr/bin/auter --prep
INFO: Running in an interactive shell, disabling all random sleeps
INFO: Updates downloaded

At this point the /var/cache/auter/default directory should have been created and the rmps moved to that directory however because the directory was not created we have the following file that was created:

# ll /var/cache/auter/default 
-rw-r--r--. 1 root root 1258970 Jan 24 11:09 /var/cache/auter/default

# file /var/cache/auter/default
/var/cache/auter/default: RPM v3.0 bin i386/x86_64 openssl-libs-1:1.0.2j-3.fc24

The update still runs as expected however this is not the expected behavior and should be fixed.

Auter post script doesn't run if updates fail

Per title. There's a quit if updates weren't applied successfully which gets triggered before the post script is able to run:

    if [[ "${HISTORY_BEFORE}" == "${HISTORY_AFTER}" ]]; then
      logit "ERROR: Updates failed. Exiting."
      quit 3
fi

<post script code>

There's no indication that this would happen in the doco and logically doesn't make much sense - if updates fail for some reason, but you've disabled monitoring / stopped services in the pre scripts, you're in a bad shape.

The naive solution is to just re-order the code to run the post script before the success of the update is checked, but maybe a potential improvement would be to pass the status of the update into the post script as $1 so the script can then potentially do different things depending on whether the update failed or not.

Rotate should delete any left-over high-number files

It could be relevant to include all files numbered above ROTATE, to avoid left-overs when modifying the value.

Ex:

if [[ -e "${OUTPUT_FILE}.${ROTATE}" ]]; then
  ROTATE_MAX_FOUND=$(ls -1 ${OUTPUT_FILE}.* | tail -n 1 | sed 's/.*\([0-9]\{1,\}\)$/\1/' )
  for i in `seq ${ROTATE} 1 ${ROTATE_MAX_COUNT}`; do
    [[ -e  "${OUTPUT_FILE}.$i" ]] && rm -f "${OUTPUT_FILE}.$i"
  done
fi

ie if ROTATE is initially set to 10 and then decreased to 5, we should delete the 6th-though-10th files, so we don't have old runs lying around

Central management functionality

centrally manage schedule
centrally manage configs

Missing check if updates are pending when ONLYINSTALLFROMPREP set to yes

Currently we are relying on the fact that there are files in ${DOWNLOADDIR}/${CONFIGSET}/*.rpm but if there are files, we dont check if there are outstanding updates from those files. This becomes a problem when you you have completed a prep and apply run then try apply again.

Currently this is the code snippet:

  if [[ "${ONLYINSTALLFROMPREP}" == "yes" ]]; then
    echo "ONLYINSTALLFROMPREP = YES"
    RC=0
    if [[ $(ls -1 ${DOWNLOADDIR}/${CONFIGSET}/*.rpm | wc -l) -gt 0 ]]; then
      echo "There are files in rmp ${DOWNLOADDIR}/${CONFIGSET}"
      RPMS="${DOWNLOADDIR}/${CONFIGSET}/*.rpm"
      RC=100
    fi
    # When passing RPMs to dnf/yum, the update verb won't install any that aren't already
    # installed (i.e. dependencies of other packages). Instead we need to use install.
    UPDATEACTION="install"
  else
...
...
...

This should be changed to:

  if [[ "${ONLYINSTALLFROMPREP}" == "yes" ]]; then
    echo "ONLYINSTALLFROMPREP = YES"
    RC=0
    if [[ $(ls -1 ${DOWNLOADDIR}/${CONFIGSET}/*.rpm | wc -l) -gt 0 ]]; then
      echo "There are files in rmp ${DOWNLOADDIR}/${CONFIGSET}"
      RPMS="${DOWNLOADDIR}/${CONFIGSET}/*.rpm"
      RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} "$(rpm -qp ${RPMS})" &>/dev/null; echo $?)
    fi
    # When passing RPMs to dnf/yum, the update verb won't install any that aren't already
    # installed (i.e. dependencies of other packages). Instead we need to use install.
    UPDATEACTION="install"
  else
...
...
...

Location for pre/post scripts shouldn't be under /var/lib

From https://bugzilla.redhat.com/show_bug.cgi?id=1359234

"/var/lib's hierarchy holds state information pertaining to an
application or the system. State information is data that programs
modify while they run, and that pertains to one specific host. Users
must never need to modify files in /var/lib to configure a package's
operation."

/usr/share also doesn't work for this since content there should not be expected to be modifed. A better location compliant with FHS is under /etc.

Check permissions on @reboot cron file

In most cases this should not be a problem however we should add a check to confirm that the auter-postreboot cron file has the correct permissions before rebooting the server. If the permissions are not correct they should be set.

From what I have found, owner and group should be set to root:root and permissions should be set to 0644 should be sufficient.

As per the man pages:
RHEL6:

CAVEATS
       All crontab files have to be regular files or symlinks to regular files, they must not be  executable  or
       writable for anyone else but the owner.

RHEL7:

CAVEATS
       The crontab files have to be regular files or symlinks to regular files, they must not be  executable
       or writable by anyone else than the owner.

Ubuntu 16.04: (In the NOTES Section)

/etc/crontab and the files in /etc/cron.d must be owned by root, and must not be group-  or  other-
       writable.

Debian 9: (In the NOTES Section)

/etc/crontab and the files in /etc/cron.d must be owned by root, and must not be group- or other-writable.

duplicate reboot call

If AUTOREBOOT is set to yes and auter is called with the --reboot option, currently the reboot_server function is called twice.

Auter should report the date/time that it completed

Tools, including auter-audit.sh would like to report the date that Auter last run. It would be much easier to do this in a standard way if Auter reported (in ISO8601 format) the date/time it completed.

e.g. instead of this:

# zgrep -h "auter.*complete" /var/log/messages-20170723.gz
Jul 20 00:09:18 test-server auter: INFO: Updates complete (yum Transaction ID : 44). You may need to reboot for some updates to take effect

We should have something like this:

Jul 20 00:09:18 test-server auter: INFO: Updates complete (yum Transaction ID : 44). You may need to reboot for some updates to take effect. DATETIME=2017-07-20T00:09:18

That way tools can extract the DATETIME from the log, rather than relying on what logging format the server has set in syslog for its date.

Don't allow running with no options

auter will not detect invalid positional args, e.g.:

# auter test
INFO: Running with: /usr/bin/auter test
INFO: Running in an interactive shell, disabling all random sleeps

Instead, it should display the help text.

Add option to skip scripts

It might be useful to add an option to skip some scripts.

We may look into the following options:

--skip-all-scripts
--skip-phase-scripts [ pre-prep | post-prep | pre-apply | post-apply | pre-reboot | post-reboot ]
--skip-script-by-name SCRIPTNAME

The third option would ideally be a glob match although it might be easier to set it as a literal match

Create contrib scripts for lvm snapshots

Please create the following scripts for auter to create an lvm snapshot:

lvm-snap.pre-apply

Should take the following actions:

Check if lvm is in use for the root filesystem
Check if there is a reasonable amount of free unallocated space in the VG
Check if there are already snapshots created (We probably don't want to allow multiple snapshots to be automatically created)
If all checks are passed then create a new snapshot with a descriptive name associating it with auter

Ubuntu support

Heya,

It'd be super good if this supported ubuntu as well!

Review shellcheck default exclusions in 05-shellcheck.sh

I was wondering why a couple of the default exclusions are in place:

SC2102: does not exist
SC2123: seems quite legitimate considering how much we use arrays for command arguments etc
SC2155: I think this one can be left in, however prior to submitting pull requests this should be checked via a manual run of shellcheck
SC2148: why don't we include shebangs in auter.yumdnfModule and auter.aptModule? I know they are not stand-alone scripts, but I can't see a downside to including it; it would also mean shellcheck/syntax highlighting work without any additional faffing.
SC2044: not sure why this is being ignored - it seems simple to fix any loops that we have currently doing this.

Wrong content in last-update file

If something fails during an update, currently the last-update-configset file is written as the last-update file, even though no yum transaction succeeded so this will be whichever was the last transaction before auter ran. Writing this file should depend on the BEFORE/AFTER check.

    echo "${PACKAGE_MANAGER} update -y ${PACKAGEMANAGEROPTIONS} ${RPMS}"
    ${PACKAGE_MANAGER} update -y ${PACKAGEMANAGEROPTIONS} ${RPMS} &>/dev/null
    ${PACKAGE_MANAGER} history info > ${DATADIR}/last-update-${CONFIGSET}
    default_signal_handling

    local HISTORY_AFTER=$(${PACKAGE_MANAGER} history list)

    if [[ "${HISTORY_BEFORE}" == "${HISTORY_AFTER}" ]]; then
      logit "ERROR: Updates failed. Exiting."
      quit 3
    fi

Remove log message when querying the status

auter --status produces the following log:

auter: auter is currently enabled and not running

This should be removed

Running auter --status as non-root user returns an incorrect status

Probably best to add a root user check within print_status() and adjust the output for non-root users. The kill -0 $(cat ${PIDFILE}) returns an error if not run as root, even if the auter process is healthy.

Conditional reboot

From my point of view, a conditional reboot with a configurable package list should be nice. At this point, if AUTOREBOOT=yes, the system reboots if ANY package is updated.

Last night, the system was rebooted because dehydrated RPM was updated.

It would be nice if the system reboots only if a package in some list is updated. For instance, only reboot if glibc*, libstd* or kernel* is updated. The rest is restarted automatically during package update.

Maybe an introduction of a third value for AUTOREBOOT=(yes|no|conditional) and a parameter
AUTOREBOOT_ON_PACKAGES="kernel glibc libstd" which would be a matching list would be nice.

Capture output from the check-update in the apply_updates function

These lines should have the redirection adjusted :
76: "${PACKAGE_MANAGER}" check-update $(rpm -qp ${RPMS}) &>/dev/null
83: local RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} &>/dev/null; echo $?)
should be changed to:
76: "${PACKAGE_MANAGER}" check-update $(rpm -qp ${RPMS}) &>${DATADIR}/last-apply-output-${CONFIGSET}
83: local RC=$(${PACKAGE_MANAGER} check-update ${PACKAGEMANAGEROPTIONS} &>/${DATADIR}/last-apply-output-${CONFIGSET}; echo $?)