codehearts / shpy

🕵️‍♀️ POSIX compliant spies and stubs for shell unit testing

shpy

POSIX compliant^[1] spies and stubs for shell unit testing

Features at a glance:

• Create spies for any command or function in the shell environment
• Stub the stdout, stderr, and return value of spies
• See the call count and check arguments passed to spies
• Integrates with the shunit2 testing framework

Table of Contents

• Why Unit Test Shell Scripts?
• Docker Image
• Usage
• Contributing
• API Reference
  • shunit2 Integration
• A Word On Shell Portability

Why Unit Test Shell Scripts?

Like other scripting languages, shell scripts can become complex and difficult to maintain over time. Unit tests help to avoid regressions and verify the correctness of functionality, but where do spies come in?

Spies are useful for limiting the dependencies and scope of a test. Code that utilizes system binaries or shell functions can be tested without running the underlying implementations, allowing tests to focus solely on the system under test. To see this in action, see examples/renamer

The benefits of spies are even greater when testing code that relies on a network. For an example of using spies to stub curl and make unit tests completely offline, see examples/coverfetch

Docker Image

Shpy is available as shpy/shpy on Docker Hub. The latest master node is published as shpy/shpy:latest, while tagged releases are available as shpy/shpy:1.0.0. To use kcov, append -kcov to the tag or use the kcov tag for the latest master node

To use the shpy image, mount your code into /app and specify the command you want to run. When using kcov, you can also mount /coverage and output your coverage reports to that directory

docker --rm -v$PWD:/app:ro shpy/shpy:1.0.0 zsh /app/tests/run_my_tests.sh
#           ^-your project                 ^--------your command---------

The following scripts and binaries are provided by this image

Name	Type	Location
shpy	script	/shpy/shpy
shpy-shunit2	script	/shpy/shpy-shunit2
shunit2	script	/usr/local/bin/shunit2
ash	binary	/bin/sh
bash	binary	/bin/bash
checkbashisms	binary	/usr/bin/checkbashisms
dash	binary	/usr/bin/dash
mksh	binary	/bin/mksh
shellcheck	binary	/usr/local/bin/shellcheck
zsh	binary	/bin/zsh

Usage

Let's try out shpy! If you don't want to install shpy locally you can run the official Docker image like so:

docker run -it --rm shpy/shpy:1.0.0

To use shpy, the SHPY_PATH environment variable must be set as the path to shpy and the shpy script must be sourced. If you're using the Docker image, SHPY_PATH is already set and shpy is located at /shpy/shpy

SHPY_PATH=path/to/shpy
. path/to/shpy

Let's create a spy for the touch command and call it!

createSpy touch
touch my-new-file
ls my-new-file # No such file or directory, touch wasn't actually called

The call to touch was stubbed out with a test dummy in place of the actual implementation. Spies record data about the calls made to them, allowing you to check the call count or call args

getSpyCallCount touch # 1
wasSpyCalledWith touch my-new-file # true
wasSpyCalledWith touch my-old-file # false
getArgsForCall touch 1 # my-new-file

Spies can also simulate successful or unsuccessful calls, like so:

createSpy -o 'call me once, shame on you' -e '' -r 0 \
          -e 'call me twice, shame on me' -o '' -r 1 touch
touch my-new-file # outputs "call me once, shame on you" to stdout, returns true
touch my-new-file # outputs "call me twice, shame on me" to stderr, returns false

When developing tests for complex functions with long chained calls, source all of them and use spy with -u flag. The flag will unset declared function, so complex functions can be tested with a mix of spies and original functions.

source my_script.sh # contains complex_function that calls file_check and directory_check

createSpy -o 'spy test' -u file_check
complex_function # complex_function calls file_check, which is mocked to output "spy test"
                 # Without -u, createSpy would warn that file_check is already declared and wouldn't be mocked

When you're done playing with shpy, it's only polite to clean up after yourself

cleanupSpies
touch my-new-file
ls my-new-file # my-new-file, touch was actually called!

Your shell environment is back to normal, and you've got a new tool at your disposal! 🎓

Contributing

If you'd like to help with shpy's development, or just gain a better understanding of the internals, check out the contributor guidelines

API Reference

To use shpy in your tests, set SHPY_PATH to the location of shpy and source the script:

SHPY_PATH=path/to/shpy
export SHPY_PATH

. path/to/shpy

When using the Docker image, SHPY_PATH is preset as /shpy/shpy for convenience

The SHPY_VERSION environment variable is provided to get the current shpy version

A summary of functions:

Function	Description
createSpy name	Create a new spy, or reset an existing spy
createSpy -r status name	Sets the status code returned when the spy is invoked Can be passed multiple times to set a return value sequence Once the sequence finishes, the last value is always returned
createSpy -o output name	Sets output sent to stdout when the spy is invoked Can be passed multiple times to set an output sequence Once the sequence finishes, the last value is always output When used with -e, standard out is written to first
createSpy -e output name	Sets output sent to stderr when the spy is invoked Can be passed multiple times to set an error output sequence Once the sequence finishes, the last value is always output When used with -o, standard out is written to first
createSpy -u name	A flag to unset declared function, so created spy can run instead otherwise declared function takes precedence Function is not restored after test runs
createStub name	Alias for createSpy
getSpyCallCount name	Outputs the number of invocations of a spy
wasSpyCalledWith name [arg ...]	Returns 0 if the current spy call under examination has the given args
getArgsForCall name call	Prints the arguments from a call to a spy (first call is 1) Single-word arguments are always listed without quotes Multi-word arguments are always listed with double-quotes
examineNextSpyCall name	Examine the next spy invocation when calling wasSpyCalledWith This causes wasSpyCalledWith to verify the second invocation, etc
cleanupSpies	Clean up any metadata on disk or in the environment for a spy

shunit2 Integration

To use shpy asserts in your shunit2 tests, you must also source the shpy-shunit2 script:

. path/to/shpy
. path/to/shpy-shunit2

A summary of asserts:

Function	Description
assertCallCount [message] spy count	Assert the number of times the spy was invoked
assertCalledWith spy [arg ...]	Assert the arguments for the first invocation of the spy Subsequent calls will assert for the second invocation, etc
assertCalledWith_ message spy [arg ...]	Same as assertCalledWith, with a specific assertion message
assertCalledOnceWith spy [arg ...]	Assert the spy was called once and given the specified arguments
assertCalledOnceWith_ message spy [arg ...]	Same as assertCalledOnceWith, with a specific assertion message
assertNeverCalled [message] spy	Assert the spy was never invoked

Use the tearDown hook provided by shunit2 to remove all spies after each test

tearDown() {
  cleanupSpies
}

A Word On Shell Portability

shpy relies on portable but more modern shell features, such as the local keyword. To be clear, shpy does not use any Bashisms

shpy's Issues

Better naming conventions for tests

Things to note:

• Test names are a bit of a mess, they don't mentioned specifically what's under test
  • They should follow a conventional style like itShowsUsageWhenCreatingSpyWithoutArgs
• t/ should be renamed to test/ for clarity, and to make it easier to click on GitHub honestly
• Assert messages should specify what went wrong, not what was expected
  • "Result is true" isn't helpful when the result is actually false

Question about nested functions calls

I was looking how to refactor ol’ scripts without too much rewriting and have functions there tested separately. The easiest way was to source target script (with some logic to prevent it running everything) and then do usual spy/asserts for each function.

However, if functionA calls functionB which calls functionC - then spy for functionB is ignored because I think bash first calls sourced functions and then one in PATH.

Workaround is to remove all nested functions with ‘unset -f functionB’ inside each test.

I wonder if that’s something framework can handle, maybe with a flag?

Infamous trailing spaces in wc util on mac

Hi,

Thanks for an amazing framework, it helped me refactoring tons of shell scripts that overwise were hopeless.

I run into small issues while developing on Mac. The wc util has a nasty behavior on Macs where its return contains leading spaces, as described here https://stackoverflow.com/questions/30927590/wc-on-osx-return-includes-spaces

This makes the output of _shpyGetDirectoryContentsCount function incorrect, which breaks counts when there are multiple spy calls.

I can submit PR, but I'd like to know if this is something worth fixing. I guess people may run into this if they develop on Macs?

Thanks!

Add unit testing example to README

Unit testing is still relatively new to the shell scripting world. The average shell programmer isn't going to know what unit testing is for and when you would use it, so it's even more unlikely that they'd know why you would want to use spies and stubs.

There needs to be a concise yet complete introduction to shell unit testing with spies and fakes that not only explains how to use the functions in shpy, but it would also explain:

• why unit testing is useful
• how you go about unit testing shell scripts
• what problem spies and stubs solve
• how to use spies effectively

I feel like I know what needs to be said, but at this point I still need to think of a programming problem that:

• illustrates the relevant testing concepts
• is a problem that most programmers can relate to and understand
• is concise enough that we don't have to go into needless detail

More 'undefined' vars checks

There are three more instances where undefined variables aren't checked in a way that works with the nounset flag

Example (dummy_test):

testDummy() {
  set -o nounset
  cleanupSpies
}

Output:

# dummy_test: 147: eval: _shpy_spies_dir: parameter not set

The other two instances involve createSpy: if a return value is passed via -r, and if no spy name is passed

Investigate storing spy outputs in memory

Spy output is currently stored on disk, but they could theoretically be stored in memory as _shpy_${spy_name}_stdout_${index} to avoid disk access

Try looking into the following:

• Can a proof of concept be made for this?
• Should this be hidden behind a SHPY_RUN_MODE=memory|disk flag, or made default?
• What impact does this have on runtime of shpy's tests?

Don't forget to update CONTRIBUTING.md to detail the new inner workings!

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update alpine Docker tag to v3.20.2
• Update koalaman/shellcheck Docker tag to v0.10.0
• Update actions/checkout action to v4
• Update codecov/codecov-action action to v4
• Update kcov/kcov Docker tag to v42
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

docker-compose

docker-compose.yml

dockerfile

Dockerfile

• koalaman/shellcheck v0.8.0
• alpine 3.17.3
• kcov/kcov v40
• alpine 3.17.3

github-actions

.github/workflows/test.yml

• actions/checkout v3
• actions/checkout v3
• codecov/codecov-action v3

---

• Check this box to trigger a request for Renovate to run again on this repository

Enforce coding styles

I've been working with Rust lately, and having rustfmt reformat code has been a godsend. I've noticed shpy has differing indentation levels and coding styles from file to file, it'd be nice to run and enforce shfmt for consistency

Use system temp dir for shpy temp dir, if available

As a user, I'd like to mount my scripts as read-only in the container I'm testing in. This isn't seamless because shpy tries to create temp directories in the current working directory

Add multiple output sequences

Similar to the current ability to return a sequence of values with

createSpy -r 4 -r 3 -r 9 some_command

we should be able to pass in a sequence of outputs, which will be printed to stdout on successive calls to the mock, in the order they are passed.

For example:

createSpy -o "foo bar" -o "abc def" some_command
some_command 
# foo bar
some_command
# abc def
some_command
# abc def

Fix coverage reporting

This may be blocked by #14, since a shpy container would make it easy to control and debug our kcov environment. Looks like kcov isn't behaving properly and is spewing output to stdout when it should be in PS4 or something, which causes builds to fail with no reporting data

Tag releases with semver

As a dependency for developers looking to test their scripts, it's wild for shpy to not be versioned. Tag some node as 1.0.0 and we can go from there

Get the actual arguments passed to the mock for a particular call

It would be nice to have a way to get the actual arguments that were passed to the mock for call X

For example:

createSpy some_spy
    
some_spy foo
some_spy
some_spy bar -a foo -baz

getArgsForCall some_spy 1
# foo

getArgsForCall some_spy 2
#

getArgsForCall some_spy 3
# bar -a foo -baz

Shpy doesn't support spies in separate shell processes

Discovered this while writing practical use examples for new users. Shpy works great for shell libraries you source into a script, but is unable to handle executable shell scripts which run in a separate shell process

Modify shpy to support this case. I have a feeling it might be one of the more common use cases

Support return value sequences

I'm really liking shpy so far, especially its support for subshells! I have a function which calls a stub multiple times, and I'm not able to test cases where the stub returns a different value on subsequent calls.

My thought would be supporting a createSpy -r 0 -r 1 -r 2 spyName syntax, and always return the last value after finishing the sequence.

Ignore common warnings with shellcheck

Shellcheck warns about using local, but there's no problem with us using it. That warning should be disabled in our default invocation of shellcheck.

Also take this chance to set up shellcheck with HoundCI. We'll need a .hound.yml and a yaml file for the shellcheck warnings to ignore

Ensure shpy Docker image is pushed/pulled once every 6 months

Docker is introducing an image retention policy for free plans (of which the shpy organization on Docker Hub is) that will remove images which have not been pushed or pulled in 6 months. I agree with the policy of removing unused images, but would like shpy to hang around for developers who discover it to have easy access

I haven't checked if this is really an issue or not (I'm not sure how often our image is used), but worst case we can always auto-publish every few months. I think this makes sense, too, because we'll automatically be kept up to date with the latest versions of supported shells

Publish a container for shpy

Once #15 is done, publish Docker containers for each release and for the latest master node. The container should make it easy for devs to test their scripts with shpy and shunit2 under the more common shells (sh, dash, bash, zsh, maybe fish? ksh? csh?)

Support dash, ksh, and pdksh

shpy already supports ash and mksh, so this should be as simple as pulling the shell binaries into the Docker container and adding new Docker compose services and Travis CI matrices

Support output via STDERR

I'm trying to stub a function which prints its expected output to STDERR, but it doesn't look shpy supports this. Would it be reasonable to add an -e STDERR_OUTPUT option, which would be printed in addition to output from -o?

Fails if nounset bash option set

If the nounset bash option is set, spies will fail due to checking of undefined variables.

Example:

#!/usr/bin/env bash

set -o nounset

testNoUnset() {
  createSpy some_cmd

  some_cmd
}

. ../shpy/shpy
. ../shpy/shpy-shunit2
. ./shunit2

Output:

testNoUnset
../shpy/shpy: line 153: _shpy_inited: unbound variable
ASSERT:unknown failure encountered running a test

Ran 1 test.

FAILED (failures=1)

Detail how shpy works in contributing.md

It would be really nice to have a simplified overview of how shpy works, how it creates the stubs, how it tracks calls, stubs outputs, etc. This should go in the development section of contributing.md

Add call-through option

When a spy is invoked, record the arguments it was invoked with and then execute (call through to) the original function/command.

This will only ever work with a shell that supports something like declare -f or typeset -f, so dash and posh support is a no-go.

Use Docker Compose for testing

Testing uses like, 5 or 6 different Docker images. Docker Compose would manage these for us, provide parallel pulls, and perform parallel testing with output that can be filtered as needed all under docker-compose up