mig1984 / bashible

Simple bash DSL framework for writing shell scripts safe and agile.

BASHIBLE

Bashible is a deployment/automation tool written in Bash (DSL). Inspired by Ansible. Simplifies things and prevents usual mistakes.

Features:

• improved readability
• unhandled failures prevention
• skipping already done tasks
• command chaining
• working directory always as expected
• variable checking
• dependencies; calling sub-scripts
• delayed tasks executed on finish
• child termination handler; no processes left running
• modules: template engine, config editing, etc.
• nice output

At the moment, bashible has been used on Arch linux. It may not be compatible with other platforms, because it internally uses GNU/sed, grep, etc.

Suggestions and bugfixes are welcome! :-)

Example script

@ represents a task (block of commands), - represents a command. Both @ and - are just bash functions with arguments. Each block may have multiple AND or OR conditions.

The working directory is automatically set to the script's. Execution will stop immediately on failure, unless you prefix a command by ignore_errors (or register it's exitcode like in this example).

#!/usr/local/bin/bashible

@ Synchronizing files
  - register exitcode as 'synced' of rsync -av /foo /bar

@ Shutting down the machine after successful synchronization
  when synced
  and test -f /etc/do-shutdown
  - shutdown -h now

@ Error happened, sending an e-mail
  when not synced
  - mail [email protected] <<< "synchronzation failed"

Rewritten into pure Bash, the example above may look like this,

#!/bin/bash

cd `dirname $0`
set -eux -o pipefail

echo Synchronizing files
if rsync -av /foo /bar; then
  echo Shutting down the machine after successful synchronization
  if test -f /etc/do-shutdown; then
    shutdown -h now
  fi
else
  echo Error happened, sending an e-mail
  mail [email protected] <<< "synchronzation failed"
fi

Another example

In this example, we are going to set two variables and store the output of ls command in them.
Moreover, the output has to be something, otherwise the execution stops.

#!/usr/local/bin/bashible

@ Loading lists
  - output_to_var HOMES is not empty_output of ls -1 /home
  - output_to_var VHOSTS is not empty_output of ls -1 /etc/nginx/vhosts.d

@ Rsyncing data and saving error messages into a file
  - quiet output_to_file errlog.txt -2 rsync /foo /bar

Both functions output_to_var and output_to_file accept options: -1|--stdout, -2|--stderr (or both). The output_to_file can also --append to it.

By prefixing a command with quiet, no messages will be written on terminal.

(is and of are just sugar words, they do actually nothing, but improve readability)

Another example

A module template is loaded. The module is a sourceable file expected to be in the same directory as bashible is. It adds some more functions.

In this example, the script expects two arguments passed from the commandline ($1, $2), they should not be empty. Also an environment variable HOME has to be something.

The template function is very powerful even it has just 18 lines of code. You can generate dynamic html with it, see examples/template directory.

#!/usr/local/bin/bashible

use template

@ Doing some checks and setting variables
  - output_to_var HOST is not empty_output of echo $1
  - output_to_var PORT is not empty_output of echo $2
  - is not empty_var HOME
  - is empty_dir /home/$HOME

@ Copying default files
  - cp -av /mnt/defaults /home/$HOME

@ Creating .bashrc from a template
  # the template needs two variables to be set, HOST and PORT
  # these are set by arguments of this script ($1 and $2)
  - cd /home/$HOME/
  - output_to_file .bashrc.tmp template /mnt/templates/bashrc.tpl
  - mv .bashrc.tmp .bashrc

The is is a sugar word. It actually does nothing and empty_dir is an alias for is_empty_dir. It's up to you, what you prefer.

If you use cd within a block, it works as expected, but next block will have it's working directory back. On each block start, bashible does chdir to the base directory. You can change it by base_dir.

Install & usage

Install bashible and it's modules (sourceable functions - here I am going to install just one module, edit). Copy everything to the same directory, for instance /usr/local/bin.

wget https://raw.githubusercontent.com/mig1984/bashible/master/bashible
wget https://raw.githubusercontent.com/mig1984/bashible/master/bashible.edit.ble
chmod 755 bashible
chmod 755 bashible.edit.ble
mv bashible /usr/local/bin
mv bashible.edit.ble /usr/local/bin

Then run the script

bashible my-script.ble ARG1 ARG2 ...

or put a she-bang in the beginning of the script and run it as a command

#!/usr/local/bin/bashible

./my-script.ble ARG1 ARG2 ...

Functions

core functions

@ MESSAGE
when
and when
or when
- COMMAND ARGS ...
- VARIABLE = VALUE
- && (conditional loop)
absolute_path PATH
bashible_version
base_dir PATH
delayed COMMAND ARGS ...
evaluate STRING
fail MESSAGE
finish MESSAGE
halt MESSAGE
not COMMAND ARGS ...
ignore_errors COMMAND ARGS ...
is_toplevel
is_empty_dir PATH
is_empty_output COMMAND ARGS ...
is_empty_var VAR
output_to_file DEST OPTS COMMAND ARGS ...
output_to_var NAME OPTS COMMAND ARGS ...
orig_dir
print_error MSG
print_info MSG
print_warn MSG
quiet COMMAND ARGS ...
reset_base_dir
result NAME COMMAND ARGS ...
run PATH ARGS ...
unless_already COMMAND ARGS ...
use FEATURES ...

sugar

For better readability, there are some more ways to do the same.

when not is_empty_dir /home
when is not empty_dir /home

when not is_empty_var HOSTNAME
when is not empty_var HOSTNAME

when not is_empty_output ls /home
when is not empty_output of ls /home

result synced rsync /foo /bar
register exitcode as 'synced' of rsync /foo /bar

(The is and of words actually do nothing. empty_dir is an alias for is_empty_dir. The register (exitcode (as)) is an alias for result.)

file-editing functions - found in bashible.edit module

add_line LINE PATH
append_line LINE PATH
comment_lines_matching REGEXP PATH
prepend_line LINE PATH
remove_lines_matching REGEX PATH
replace_lines_matching REGEXP STRING PATH
replace_matching REGEXP STRING PATH
uncomment_lines_matching REGEXP PATH

template engine - found in bashible.template module

template TEMPLATE_PATH RESULT_PATH

timeout - found in bashible.timeout module

in_timeout SECS COMMAND ARGS ...

network-oriented functions - found in bashible.net module

wait_for_tcp MATCH up|down

TODO

Write more docs and examples.

Modularize. The bashible core in the version 1.0 should contain only necessary functions and should not ever change. For instance, the delayed and unless_already COMMAND ARGS ... functions now need two temporary files. These files are created on every bashible startup. These functions should go into optional modules instead.

Create tests. Bashible uses GNU/grep, GNU/sed and other programs which may not work properly on all platforms.

Make bashible multiplatform.

Create more modules and/or integrate existing Bash libraries.

bashible's Issues

More functions

Hey,

Great initiative,

I'll share with you some functions I used. If you can include them into bashible, it'll be great.
It requires some refactoring as you handle timeout with a separate function.

Note: I do not share printlnError functions as it's mainly the same as yours.

# Usage:
#   wait_file_present [-t TIMEOUT] [-i INTERVAL] FILE
# Parameters:
# -t: timeout in second (default: 60)
# -i: interval in second (default: 1)
# Example:
#   wait_file_present -t 10 -i 2 /tmp/file.txt
function wait_file_present() {
    local FILE TIMEOUT TIMEOUT_ORIGIN INTERVAL
    TIMEOUT=60
    INTERVAL=1
    local OPTIND arg
    while getopts ":t:i:" arg; do
        case $arg in
        t) TIMEOUT="${OPTARG}";;
        i) INTERVAL="${OPTARG}";;
        :) printlnError "Invalid option: $OPTARG requires an argument"; return 1;;
        ?) printlnError "Unkownn option: $OPTARG" ; return 1;;
        esac
    done
    shift $((OPTIND -1))

    FILE="${1?File not defined}"

    TIMEOUT_ORIGIN="${TIMEOUT}"
    until [ -f "${FILE}" ] > /dev/null 2>&1 && SUCCESS=1 || [ "$TIMEOUT" -le 0 ]; do 
        sleep "${INTERVAL}"
        TIMEOUT=$((TIMEOUT-INTERVAL))
    done
    if [ "${SUCCESS}" != "1" ]; then
        echo "Error: File \"${FILE}\" does not exists after ${TIMEOUT_ORIGIN}s"; 
        return 1; 
    fi
}


# Usage:
#   wait_tcp_port_open [-t TIMEOUT] [-i INTERVAL] HOST PORT
# Parameters:
# -t: timeout in second (default: 60)
# -i: interval in second (default: 1)
# Example:
#   wait_tcp_port_open -t 10 -i 2 localhost 8080
function wait_tcp_port_open() {
    local HOST PORT TIMEOUT TIMEOUT_ORIGIN INTERVAL
    TIMEOUT=60
    INTERVAL=1
    local OPTIND arg
    while getopts ":t:i:" arg; do
        case $arg in
        t) TIMEOUT="${OPTARG}";;
        i) INTERVAL="${OPTARG}";;
        :) printlnError "Invalid option: $OPTARG requires an argument"; return 1;;
        ?) printlnError "Unkownn option: $OPTARG" ; return 1;;
        esac
    done
    shift $((OPTIND -1))

    HOST="${1?Host not defined}"
    PORT="${2?Port not defined}"

    TIMEOUT_ORIGIN="${TIMEOUT}"
    SUCCESS=0
    until nc -zvw3 "${HOST}" "${PORT}" > /dev/null 2>&1 && SUCCESS=1 || [ "$TIMEOUT" -le 0 ]; do 
        sleep "${INTERVAL}"
        TIMEOUT=$((TIMEOUT-INTERVAL))
    done
    if [ "${SUCCESS}" != "1" ]; then
        echo "Error: Port \"${HOST}\" \"${PORT}\" is not open after ${TIMEOUT_ORIGIN}s"; 
        echo "Debug: Command: nc -zvw3 \"${HOST}\" \"${PORT}\")" 
        return 1; 
    fi
}


# Usage:
#   curl_status [-c EXPECTED_HTTP_CODE] [-- [CURL_ARGS]...] URL
# Parameters:
# -c: comma separated list of expected HTTP CODE (default: 200)
# -v: verbose (warning: this could print password)
# --: special option to delimit end of wait_curl_status parameters. All options behind are curl options (see example below)
# Example:
#   curl_status https://postman-echo.com/status/200
#   curl_status -c 204 https://postman-echo.com/status/204
#   curl_status -c 204 -- -X GET -H "host: local.com" https://postman-echo.com/status/204
#   curl_status -c 500,000 -- -X GET -H "host: local.com" https://postman-echo.com/status/500
function curl_status() {
    local EXPECTED_HTTP_CODE CURL_STATUS CURL_TIMEOUT VERBOSE=false
    EXPECTED_HTTP_CODE=200
    CURL_TIMEOUT=5

    local OPTIND arg
    while getopts ":c:v" arg; do
        case $arg in
        c) EXPECTED_HTTP_CODE="${OPTARG}";;
        v) VERBOSE=true;;
        :) printlnError "Invalid option: $OPTARG requires an argument"; return 1;;
        ?) printlnError "Unkownn option: $OPTARG" ; return 1;;
        esac
    done
    shift $((OPTIND -1))
    
    EXPECTED_HTTP_CODE_PATTERN=",${EXPECTED_HTTP_CODE},"
    CURL_STATUS=$(curl -sS -o /dev/null -w "%{http_code}" --connect-timeout "${CURL_TIMEOUT}" --max-time "${CURL_TIMEOUT}" "$@" 2>/dev/null)
    if [[ "${EXPECTED_HTTP_CODE_PATTERN}" != *",${CURL_STATUS},"* ]]; then
        echo "Error: Curl returned non expected status code ${CURL_STATUS} (expected status: ${EXPECTED_HTTP_CODE})"; 
        $VERBOSE && echo "Debug: Curl command: curl -s -o /dev/null -w %{http_code} --connect-timeout ${CURL_TIMEOUT} --max-time ${CURL_TIMEOUT} $(_echo_args "$@")" 
        return 1; 
    fi
}

# Usage:
#   wait_curl_status [-c EXPECTED_HTTP_CODE] [-t TIMEOUT] [-i INTERVAL] [-- [CURL_ARGS]...] URL
# Parameters:
# -c: comma separated list of expected HTTP CODE (default: 200)
# -t: timeout in second (default: 60)
# -i: interval in second (default: 1)
# -v: verbose (warning: this could print password)
# --: special option to delimit end of wait_curl_status parameters. All options behind are curl options (see example below)
# Example:
#   wait_curl_status https://postman-echo.com/status/200
#   wait_curl_status -c 204 -t 10 -i 2 https://postman-echo.com/status/204
#   wait_curl_status -c 204 -t 10 -i 2 -- -X GET -H "host: local.com" https://postman-echo.com/status/204
#   wait_curl_status -c 500,000 -t 10 -i 2 -- -X GET -H "host: local.com" https://postman-echo.com/status/500
function wait_curl_status() {
    local EXPECTED_HTTP_CODE TIMEOUT TIMEOUT_ORIGIN INTERVAL CURL_STATUS CURL_TIMEOUT VERBOSE=false
    EXPECTED_HTTP_CODE="200"
    CURL_TIMEOUT=5
    TIMEOUT=60
    INTERVAL=1
    local OPTIND arg
    while getopts ":c:t:i:v" arg; do
        case $arg in
        c) EXPECTED_HTTP_CODE="${OPTARG}";;
        t) TIMEOUT="${OPTARG}";;
        i) INTERVAL="${OPTARG}";;
        v) VERBOSE=true;;
        :) printlnError "Invalid option: $OPTARG requires an argument"; return 1;;
        ?) printlnError "Unkownn option: $OPTARG" ; return 1;;
        esac
    done
    shift $((OPTIND -1))
    EXPECTED_HTTP_CODE_PATTERN=",${EXPECTED_HTTP_CODE},"
    TIMEOUT_ORIGIN="${TIMEOUT}"
    until CURL_STATUS=$(curl -sS -o /dev/null -w "%{http_code}" --connect-timeout "${CURL_TIMEOUT}" --max-time "${CURL_TIMEOUT}" "$@" 2>/dev/null || true) && [[ "${EXPECTED_HTTP_CODE_PATTERN}" == *",${CURL_STATUS},"* ]] || [ "$TIMEOUT" -le 0 ]; do 
        sleep "${INTERVAL}"
        TIMEOUT=$((TIMEOUT-INTERVAL))
    done
    if [[ "${EXPECTED_HTTP_CODE_PATTERN}" != *",${CURL_STATUS},"* ]]; then
        echo "Error: Curl returned non expected status code ${CURL_STATUS} after ${TIMEOUT_ORIGIN}s (expected status: ${EXPECTED_HTTP_CODE})"; 
        $VERBOSE && echo "Debug: Curl command: curl -s -o /dev/null -w %{http_code} --connect-timeout ${CURL_TIMEOUT} --max-time ${CURL_TIMEOUT} $(_echo_args "$@")" 
        return 1; 
    fi
}


# Helper function to print args "$@" preserving the quote
# Usage:
#  _echo_args "first args" "second args"
#  echo "ls $(_echo_args "first args" "second args")"
#  # in a function (which is the main purpose)
#  echo "ls $(_echo_args "$@")"
function _echo_args () {
    local ARG ARGS=""
    for ARG in "${@}" ; do
        # add quote only if $ARG contains a space
        if [[ "$ARG" != "${ARG%[[:space:]]*}" ]]; then
            ARG="\"$ARG\""
        fi
        if [ -z "$ARGS" ]; then
            ARGS="$ARG"
        else
            ARGS="$ARGS $ARG"
        fi
    done
    echo "$ARGS"
}

First glance: bad command names and syntax

Dear @mig1984,

I found bashible on Hacker News, just want to share some thoughts about bashible based on my own Ansible experience.

I checked documents under docs/ directory and some examples, i think the biggest issue of bashible might be unclear command names and syntax, they could/should be greatly improved.

About command names, for example:

• may_fail: better be ignore_errors (used by Ansible)
• add_line, append_line, prepend_line: maybe merge to one command named lineinfile (line in file. used by Ansible), and just use some argument to indicate it should be appended or prepended if not present.
• fill_var: why it's called fill and not something like set?
• var_empty: maybe rename to 'is_empty_var'? easier to understand.
• ...

About syntax, i'd like to compare bashible with cdist. Although cdist itself is wrote in Python-3, but cdist user just writes bash scripting in cdist syntax.

Compare bashible's add_line, append_line and prepend_line for example, compare them with cdist's __line command (it's called cdist type instead of command (bashible) or task (ansible), but let's call it command below and save me some typing). Please allow me to copy cdist official examples below:

# Manage a hosts entry for www.example.com.
__line /etc/hosts \
    --line '127.0.0.2 www.example.com'

# Manage another hosts entry for test.example.com.
__line hosts:test.example.com \
    --file /etc/hosts \
    --line '127.0.0.3 test.example.com'

# Remove the line starting with TIMEZONE from the /etc/rc.conf file.
__line legacy_timezone \
   --file /etc/rc.conf \
   --regex 'TIMEZONE=.*' \
   --state absent

# Insert a line before another one.
__line password-auth-local:classify \
    --file /etc/pam.d/password-auth-local \
    --line '-session required pam_exec.so debug log=/tmp/classify.log /usr/local/libexec/classify' \
    --before '^session[[:space:]]+include[[:space:]]+password-auth-ac$'

# Insert a line after another one.
__line password-auth-local:classify \
    --file /etc/pam.d/password-auth-local \
    --line '-session required pam_exec.so debug log=/tmp/classify.log /usr/local/libexec/classify' \
    --after '^session[[:space:]]+include[[:space:]]+password-auth-ac$'

One command may support multiple arguments to generate different results. This syntax (IMO) is so much better than bashible.

Just my 2 cents. hope it helps a little. :)

Document installation of released version

In order to have reproductible scripts, we have to rely on released version.
Can you document it (and add a changelog & tell that you follow semver)?

bash command result to bashible script

I need run blkid and get partrition id and find line in fstab and replace mount option. How to do that?

bash v4 compatibility

running bashible i get:
./bashible: line 574: shopt: inherit_errexit: invalid shell option name

bash manual says this shopt is only available in posix mode

Originally posted by @ootada in #9 (comment)

GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
CentOS Linux release 7.9.2009 (Core)

Inherit errexit

In this line you mention set -e in subprocesses created by the script:

bashible/bashible

Line 230 in 846919d

# FIXME: there's no way to re-enable "set -e" which does not work in the subshell

I'm not sure if I completely understand what you mean by that, but maybe adding this can work?:

shopt -s inherit_errexit 2>/dev/null || true

This makes subprocesses inherit errexit. But maybe I'm confused and you were not referring to that.

I just discovered this project and I find it amazing. Thanks for the work!

version 1.0.0

I'm interested in starting to roll out some bashible for some general use cases.
It would be nice to get an idea of when you think you'll have a fairly stable version baselined, just so I can understand if there will be breaking changes if I update.
I know you were still adapting it based on a couple of feedback points received.