shelldoc: Test Unix shell commands in Markdown documentation
Markdown is widely used for documentation and README.md files that
explain how to use or build some software. Such documentation often
contains shell commands that explain how to build the software or how to
run it. To make sure the documentation is accurate and up-to-date, it
should be automatically tested. shelldoc tests Unix shell commands
in Markdown files and reports the results.
Basic usage
shelldoc parses a Markdown input file, detects the code blocks in
it, executes them and compares their output with the content of the
code block. For example, the following code block contains a command,
indicated by either leading a $ or a > trigger character, and an
expected response:
$ echo Hello
Hello
Lines in code blocks that begin with a $ or a > trigger character
are considered commands. Lines inbetween without those trigger characters
are considered the expected response. shelldoc will execute these
commands and return whether or not the commands succeeded and the
output matches the specificaton:
% shelldoc run README.md
SHELLDOC: doc-testing "go/src/github.com/endocode/shelldoc/README.md" ...
CMD (1): echo Hello ? Hello : PASS (match)
CMD (2): go get -u github.com/endocode/shelldo... ? ... : PASS (match)
CMD (3): export GREETING="Hello World" ? (no response expected) : PASS (execution successful)
CMD (4): echo $GREETING ? Hello World : PASS (match)
SUCCESS: 4 tests (4 successful, 0 failures, 0 execution errors)Note that this example is not executed as a test by shelldoc, since
it does not start with a trigger character. Doing so would
cause an infinite recursion when evaluating the README.md using
shelldoc. Try it :-) The percent symbol is commonly used as a shell
prompt next to $ or a >. It can be used in documentation as a
prompt indicator without triggering a shelldoc test.
Installation
The usual way to install shelldoc is using go get:
$ go get -u github.com/endocode/shelldoc/cmd/shelldoc
...
Executing documentation may have side effects. For example, running
this go get command just installed the latest version of shelldoc
in your system. Containers or VMs can be used to isolate such side
effects.
Details and syntax
All code blocks in the Markdown input are evaluated and executed as tests. A test succeeds if it returns the expected exit code, and the output of the command matches the response specified in the code block.
shelldoc supports both simple and fenced code blocks. An ellipsis,
as used in the description on how to install shelldoc above,
indicates that all output is accepted from this point forward as long
as the command exits with the expected return code (zero, by default).
The -v (--verbose) flags enables additional diagnostic output.
A shell is launched that will execute all shell commands in a single
Markdown file. By default, the user's configured shell is used. A
different shell can be specified using the -s (--shell) flag:
% shelldoc --verbose run --shell=/bin/sh README.md
Note: Using user-specified shell /bin/sh.
...
The shell's lifetime is that of the test run of a single Markdown file. The environment of the shell is available between test interactions:
$ export GREETING="Hello World"
$ echo $GREETING
Hello World
shelldoc uses
the
Blackfriday Markdown processor to
parse Markdown files, and the Cobra
package to parse the command line arguments.
Options
Regular code blocks do not have a way to specify options. The only thing that can be specified about them are the commands and the responses. That means the expected return code must always be zero for the test to succeed.
Sometimes, however, things are more complicated. Some commands are
expected to return a different exit code than zero. Some commands
return exit codes that are unknown up-front. Both options can be
handled by specifying tests in fenced code blocks. Fenced code blocks
may have an info string after the opening characters. This info string
is typically used to specify the language of the listed code. After
the language specifier however, other information may
follow. shelldoc uses this opportunity to allow the user to specify
options about the test. These options are:
```shell {shelldocwhatever}
% echo Hello && false
Hello
```
Try executing this test:
> echo Hello && false
HelloThe shelldocwhatever options tells shelldoc that the exit code of
the following command does not matter. If any expected response is
specified, it will still be evaluated. The test succeeds if the expected
response is produced, no matter the exit code of the command.
An expected exit code is specified using the shelldocexitcode option:
```shell {shelldocexitcode=2}
% (exit 2)
```
This means the test is considered successful if it produces no response and returns 2.
> (exit 2)The shelldocexitcode specifies an exact exit code that is expected. The test fails if the exit code of the command does not match the specified one, or if the response does not match the expected response.
Output formats and integration into CI systems
By default, shelldoc produces human-readable output. Additionally, shelldoc can create a results file in the JunitXML format. This format is natively understood by many continuous integration (CI) systems, like for example Jenkins. The output file is specified using the --xml argument. This feature is demonstrated in shelldoc's own CI and the Jenkinsfile in the repository.
Contributing
shelldoc
is
free and open source software. Everybody
is invited to use, study, modify and redistribute it. To contribute to
shelldoc, feel free to fork it and submit pull requests, or to
submit issues in
the
shelldoc issue tracker. All
contributions are welcome.
To report a bug, the best way is to submit a Markdown file and a
description of how the Markdown file should be interpreted, and how
shelldoc interprets it.
Authors and license
shelldoc was developed
by Mirko Boehm. Commercial support,
if necessary, is provided
by Endocode.
The command line programs of shelldoc are located in the cmd/
subdirectory and licensed under the terms of the GPL, version 3. The
reusable components are located in the pkg/ subdirectory and
licensed under the terms of the LGPL version 3. Unit test and example
code is licensed under the Apache-2.0 license.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent