dannyben / bashly

Bash command line framework and CLI generator

Bashly - Bash CLI Framework and Generator

Create feature-rich bash scripts using simple YAML configuration

bashly.dannyb.co

---

Bashly is a command line application (written in Ruby) that lets you generate feature-rich bash command line tools.

Bashly lets you focus on your specific code, without worrying about command line argument parsing, usage texts, error messages and other functions that are usually handled by a framework in any other programming language.

It is available both as a ruby gem and as a docker image.

Documentation

• Bashly Homepage
• Examples

How it works

1. You provide a YAML configuration file, describing commands, sub-commands, arguments, and flags. Running bashly init creates an initial sample YAML file for you (example).
2. Bashly then automatically generates a bash script (when you run bashly generate) that can parse and validate user input, provide help messages, and run your code for each command.
3. Your code for each command is kept in a separate file, and can be merged again if you change it (example).

Features

Bashly is responsible for:

• Generating a single, standalone bash script.
• Generating a human readable, shellcheck-compliant and shfmt-compliant script.
• Generating usage texts and help screens, showing your tool's arguments, flags and commands (works for sub-commands also).
• Parsing the user's command line and extracting:
  • Optional or required positional arguments.
  • Optional or required option flags (with or without flag arguments).
  • Commands (and sub-commands).
  • Standard flags (like --help and --version).
• Preventing your script from running unless the command line is valid.
• Providing you with a place to input your code for each of the functions your tool performs, and merging it back to the final script.
• Providing you with additional (optional) framework-style, standard library functions:
  • Color output.
  • Config file management (INI format).
  • YAML parsing.
  • Bash completions.
  • and more.
• Auto-generating markdown and man page documentation for your script.

Contributing / Support

If you experience any issue, have a question or a suggestion, or if you wish to contribute, feel free to open an issue or start a discussion.

Visit the How to contribute page for more information.

bashly's Issues

Convert travis shellcheck to a Runfile command that checks all examples

[Feature] Support allowed argument values as an option

First of all, I cannot sing the praises of bashly enough - I have been using it heavily for a BASH CLI in my project for quite some time now and it just works.

One feature that I'd like to see personally is support for allowed options for arguments. For example, a suggestion for the bashly.yml could look like:

- name: login
  short: l
  help: Log on to a cool service

  args:
  - name: role
    help: Role for user
    required: true
	allowed: ["admin", "user"]

The behaviour I am expecting is:
✔️ login admin
✔️ login user
❌ login anythingelse - this could fail with an error message like Invalid role - allowed values are "admin" and "user"

Automatic support for short commands

Right now, subcommands need to define the short attribute in order to have their short version (commit, c).

It would be much nicer if we can automatically support any input and match it to the first matching command - so for a command named commit, it will automatically accept c, co, com etc.

Fails to fallback to the default command if no argument is passed

I might've misinterpreted the default key, but here's my use case:

Assuming a CLI tool called foo, if I run ./foo without any arguments, it should run the default command as specified with the default key, but instead it outputs the usage and exit.

I think the reason is that in parse_requirements function, the default command comes after the fallback command, inverting the order fixes the issue:

bashly/examples/command-default/ftp

Lines 194 to 204 in d149e53

	# :command.command_fallback
	"" )
	ftp_usage
	exit 1
	;;
	* )
	action="upload"
	ftp_upload_parse_requirements "$@"
	shift $#
	;;

Is it possible to provide custom `help` functions in case of wrapper functions?

Is it possible to provide custom help functions in case of wrapper functions?

I would assume to have a way to flag the help as external so I could bypass internal --help processor and pass it to my command or rather a separate generated file which includes implementation for the *_usage function.

Subcommand with short code and no args does not work

With this YAML file:

name: rush
help: Personal package manager
version: 0.1.0

commands:
- name: config
  short: c
  help: Show the configuration file

- name: get
  short: g
  help: Install a package

  args:
  - name: repo
    required: true
    help: Repository name
  - name: package
    required: true
    help: Package name

This does not work:

$ ./rush c

While this, works:

$ ./rush g

Rest flags and arguments for commands

It would be nice to somehow disable the "invalid option" validation error for certain commands which can expect arbitrary options as flags and/or arguments. I am looking to develop a CLI which performs a few specific actions but ultimately delegates to the AWS CLI. Is this somehow possible?

Currently bashly.yml:

name: mycli
help: My amazing CLI utility
version: 0.1.0

commands:
- name: aws
  help: Delegators to AWS CLI
  dependencies:
  - aws
  commands:
  - name: deploy-stack
    help: |-
      Deploys a CloudFormation stack from the provided template file.
    args:
    - name: file_path
      help: Path to a file
      required: true
    - name: stack_name
      help: Name of CloudFormation stack
      required: true
    - name: template_file_path
      help: Path to CloudFormation template
      required: true

Command (which I wish it works):

mycli aws deploy-stack file1 mystack file2 --force-upload --s3-bucket mybucket

However, it exits with error:

invalid option: --force-upload

Would it be possible to relax the validation checks for a few commands?

Add ability to retrieve all config values from the config file

Add code climate

Generate user documentation from bashly.yml

Hi!

Great tool, we use it for our CI scripts, it helpful to follow DRY principles.

I think bashly.yml can be used not only for generate cli command. Also, it useful for generate Markdown documentation in long format where will be described all command and sub-command.

What think about it?

Add full integration tests

Is there an "official" way to call a command from within a command?

I know I can probably do this by invoking the generated function but I am looking for some documentation-based guidance (if possible) on this scenario. I have two commands, let's say, cli dothis and cli dothat.
I want to call cli dothat from within the implementation for cli dothis. Can this be done without knowing the internals of the generated bashly file?

Add documentation for the configuration syntax

Using `catch_all` with `required: true`

Hi, @DannyBen!

I'm just wondering how you would recommenced using the catch_all approach when the additional arguments are required but of unknown length? I see that providing required: true to the catch_all hash changes nothing, which makes sense.

Thanks for the great tool,

• Jakob

Originally posted by @JakobGM in #70 (comment)

Add ability to define examples and environment variable usage sections

So it will look like:

Examples:
  download repo package --force
  download init

Environment Variables:
  REPO_SOURCE
    Source to the repository. If not set, defaults to ~/repos.

`bashly generate --wrap function` outputs a script with excess newlines

When comparing the output of bashly g to the one of bashly g --wrap somefunc, the latter has too many newlines.

It would seem like the sanitization function is not handling this case well.

Support whitelist for args

Similarly to what is done in #66 and discussed in #65 , we should probably implement the same for args.

Exit gracefully if Bash is too old

Bashly is not compatible with older Bash versions, which is totally fine. I'm wondering if it would make sense to fail with a helpful error message if $BASH_VERSION is too old? As it is now, it just causes a syntax error.

I have a lot of users on OSX, which (of course 🙄) ships with a very old version of Bash, and even if I specify in the instructions that a newer version is needed, it still catches people off guard. I'd love to have a way to have the script itself tell the user what they need to do.

I suppose an alternative would be to have a pre-command function that can run at startup for every single command, so people can autonomously implement checks like this, but it would need to run before anything is done that requires a newer Bash version.

What are your thoughts?

Don't break uninterrupted strings longer than 80 characters

If I have a URL that is longer than 80 characters in a help block, it gets split in two. This prevents people from opening it by clicking on it, and makes it annoying to copy it. I think it makes sense to leave long, uninterrupted strings alone and let the terminal deal with it.

Is this as simple as removing this regex substitution? 🤔

bashly/lib/bashly/extensions/string.rb

Line 19 in 6bb6d2f

line.gsub!(/([^\s]{#{length}})([^\s$])/, "\\1 \\2")

catch_all arguments are not printed if not any arg is specified

catch_all arguments are not printed if not any arg is specified.

Expected generated help for catch_all label and help even when args is not filled in.

Flags with arguments consume other flags instead of erroring

Bashly config:

name: download
version: 0.1.0

flags:
- long: --path
  arg: path
  help: path to directory
  default: somedir
- long: --other
  help: test some flag

Expected behavior

$ ./download --path --other
--path requires an argument: --path PATH

Actual behavior

$ ./download --path --other
args:
- ${args[--path]} = --other

Notes

Implementing this will probably mean checking for the supplied flag argument, and if it starts with a hyphen (-), assume it is another flag and disallow it. Therefore, need to consider this downside before implementing.

Asking for missing config values returns the previously found value

We probably either need to add local all over the place in the lib function or clear all the values you tend to use.

Example of how to make it behave unexpectedly:

config_init

echo First we try get foo \"$(config_get foo)\" and it is not there
echo so we set it to BOOM
config_set foo BOOM
echo now it is is \"$(config_get foo)\" which is great
echo but now every missing thing is \"$(config_get baz)\" which is not so great

Add support for defining dependencies

For example, if a script requires curl, we can have this config:

dependencies:
  - curl

This will not be displayed in the usage text, but the script will run a dependencies_filter function before running, and check if curl is installed.

Add support for required environment variables

Can probably be done by allowing an alternative syntax to define environment variables:

environment_variables:
  API_KEY:
    help: Your API Key
    required: true

Another option, is to normalize the environment_variables definition, to be like the others and use a single form for both required or optional:

environment_variables:
  - name: api_key   # will be capitalized
    help: Your API Key
    required: true

Support sub-sub-commands

respect ' ' as single literal flag argument so the argument can contain dashes

Hope you don't mind me bringing up issues I have. Maybe you haven't had much feedback.

I need to be able to pass on native flags to the underlying command (rdiff-backkup).

- long: --options
  short : -o
  arg: options
  help: rdiff options 

so trying
./backup -o '-test -a'

produces
--options requires an argument: --options, -o OPTIONS

I have a work around but it requires escaping every dash

./test -o '\-test \-a'

then in my script
options=$(echo ${args[--options]} | awk '{gsub(/\/," ")}1')

gives
rdiff-backup -test -a

bashly should treat a single quoted argument as a single literal argument and not try to parse within it

Update completions so that they work with local scripts

As discovered in #100, the generated bash completions do not work when the generated script is not installed in the path, and instead called locally (e.g. ./my-app).

The solution is outlined in the same issue, but needs to be implemented in completely.

Todo

• Implement change in the completely gem
• Require updated gem in bashly

Support for optional flag argument (`--flag [arg]`)

It seems that if a flag has a default set in bashly.yml that it wil have that value whether or not the flag was set in the command line.

That was an unexpected behavior.

Is is possbile to get the behvior I want.

flags:
  - long: --someflag
    arg: someflag
    help:  test some flag
    default: true

./,mybashly

I expect ${args[--someflag]} to be empty but it is true

./mybashly --someflg

I expect ${args[--someflag]} to be true and it is

./mybashly --someflg avalue

I expect ${args[--someflag]} to be avalue and it is

in essence I need a flag that can act as a flag without a value when it is not is set or as one with a value if it is set. For example I need to set a default mount point directory if --view is set but if --view /some/mountpoint is passed I need to use that value. If the flag is not set at all I need to ignore this all together.

  - long: --view
    arg: mount
    help: mount snapshot for viewing at mount point ( default is <target>/view)

if [[ ${args[--view]} ]]; then 
    if [[ -e ${args[--view]} ]]; then
        $mount=${args[--view]}
    else
        $mount="${target}/view"
    fi
    cmd="sudo $password $bin -r $target mount $mount"; 
fi

Run shellcheck on generated scripts

and fix templates accordingly

support just sourcing of the generated shell file.

I do this kind of thing at the end of a file I consider a "module" to source with a primary function but may want to run directly

(return 0 2>/dev/null) || run "$@"

that way I can source as a module (say at login) and run it later as function, but if I want to execute right then I can.
This means I namespace any "helper" functions so they will not collide with other sourced functions in my login shell. Maybe they should be prefaced with the command name. What would also be required is to change the run function to the name of the command being generated.

If that is not wanted during development then maybe bashly generate might have a --pro flag to do this in "production".

Remove generated examples from the repositories

Any code change in bashly that alters the generated code even a little bit causes all generated example executables to be changed, making PR review much more messy than it should be with dozens of changed example files.

I believe we can safely remove them and .gitignore them.

Bash completions do not work

I just discovered this tool, looks super cool!

I am trying to reproduce the completions example: https://github.com/DannyBen/bashly/tree/master/examples/completions but it does not seem to work.

Steps I took:

• in an empty directory, create bashly.yml
• paste the configuration from the example into that file
• Run the following commands, as per the example:

$ bashly init
$ bashly add comp function
$ bashly generate

• Run ./cli. I get the output

cli - Sample application

Usage:
  cli [command]
  cli [command] --help | -h
  cli --version | -v

Commands:
  download   Download a file
  upload     Upload a file

Note that, unlike the example, I do not have a completions command

• ./cli completions just prints the same usage information, so I can't eval that to enable completions.

bashly generated the following files:

$ tree
├── cli
└── src
    ├── bashly.yml
    ├── download_command.sh
    ├── initialize.sh
    ├── lib
    │   └── send_completions.sh
    └── upload_command.sh

2 directories, 6 files

My environment:

I am using bashly through docker, as described in the installation instructions.

$ bashly --version
0.6.4
$ bash --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
$ docker image inspect dannyben/bashly
[
    {
        "Id": "sha256:82d3128ff8617e75c2e58638920f046453dd07759846b209c8bd43b0dd5f7845",
        "RepoTags": [
            "dannyben/bashly:latest"
        ],
        "RepoDigests": [
            "dannyben/bashly@sha256:bf1a326659205b5974657e76d57179ee890abf85f0fda1e960630003effb7aea"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2021-08-27T15:24:07.059809147Z",
...

Am I doing something wrong or is this a bug?

bashly won't run by default on Mac, due to zsh standard and bash 3.2

The autogenerated files by bashly check for bash version 4 or higher. On MacOS, the default version is version 3.2, because license reasons. This leads to the message bash version 4 or higher is required.

Instead of bash, MacOS defaults to zsh (by default version 5.8 on latest macOS), which afaik supports all the relevant functionality in bash 4+.

I've just done a bit of testing, and there are two solutions:

Solution #1

brew install bash installs the latest bash in the brew directory, then links it to bash, and does not overwrite default shell (zsh is still default). This all that is required for bashly scripts to run, but updating bash can be viewed as invasive by sysadmin.

Solution #2

Specifically, the two following parts of generated bash scripts need editing on Mac:

• Line 1: #!/usr/bin/env bash -> #!/usr/bin/env zsh
• Further down, this needs to be commented out or changed to support zsh:

  if [[ "${BASH_VERSINFO:-0}" -lt 4 ]]; then
    printf "bash version 4 or higher is required\n"
    exit 1
  fi

I don't know if there's a way to target "default system shell" in the shebang. That would prevent the problem on Mac, but could lead to issues if one is using e.g. fish shell and has bash 4+ available.

Personally, I'm sticking with solution 1, but I thought I'd share my thoughts here, since there were no results for a search for zsh on the github.

Change default help message

Right now, it is:

--help
  Show this help

Which is incorrect. It shows an extended help, and also shows a completely different help if used on a subcommand. This needs to reflect it.

Plus: We should probably display the --help, -h and --version when running the script without params.

heredocs are rendered with prefixed spaces causing the compiled CLI to fail

Heredoc expressions such as the following in command files are rendered with two prefixed spaces for the end character. This causes issues with the heredoc and causes the command to fail. Example:

In command file:

ext_peer_config=$(
  cat <<EOF
cidr: 10.${ip_segment}.0.0/16
id: ${peer_id}
name: ${peer_name}
EOF
    )

In the generated file (after running bashly generate):

  cat >"$peer_context_file" <<EOL
  cidr: 10.${ip_segment}.0.0/16
  id: ${peer_id}
  name: ${peer_name}
  EOL # there are 2 spaces before this character

how to implement boolean flags

....just searching for some help on bash scripting and bam....this repo comes up. way cool. I'm giving it a whirl instead of doing my own getopts thing. BTW thanks for the docker container that made it easy (no need to install ruby, gem etc)

One thing I'm not grokking is implementing boolean flags. I saw no yaml example for such. In getopts I just set an option without : and then can have a variable changed from null to true in getopts.

I see no way to assign true to a variable like with flags that accept an argument.

While I am here what about assigning a default value to a (environment) variable.

Like I have

args:
- name: source
  default: $PWD
  help: source to be backed up

will that work?

Add docs for releasing cli for users

A document or how-to guide would be very helpful to show how the generated cli can be released.
eg. https://oclif.io/docs/releasing

Add hooks for more user code

It would be nice to have the ability to allow user to inject code (similar to how the initialize.sh behaves) in more places, for example:

• The very top of the bash script (solves #114) - this will also be a good way to let people replace the default header if they want to.
• Before the call to any user function (solves #113)
• After the call to any user function

So the execution order might look like this:

1. header (user injectable)
2. initialize (user injectable)
3. other bashly code
4. before (user injectable)
5. user code
6. after (user injectable)

TODO

• Custom header (#120)
• Decide if any additional hooks are needed:
  • Hook for "before user code"
  • Hook for "after user code"

Originally posted in #113 (comment)

Generate Bash autocompletion

Hey there!

Have you considered adding some automatic Bash completion generation to Bashly? The way this is usually accomplished is to have a function that prints out Bash code to set up all the autocompletion nonsense, so that people only have to add something like this to their .bashrc and that's it:

eval "$(script_name generate_completion)"

Basically, Bashly would have to add a command that, when run, outputs something like what you see in this SO answer: https://stackoverflow.com/a/17881946

Thanks for making Bashly!

Make all hard coded strings configurable

Consider regenerating completions on `bashly generate`

Right now, the only way to regenerate the bash completions function is by rerunning bashly add comp function.

The reasons it is like this are:

1. Different users will choose different places to integrate these completions
2. There is no place in the config file to specify things like "and also regenerate completions"

Although it seems like a high level of effort for the time being to change this, from user perspective, it is most definitely needed.

It would be nice if bashly generate is smart enough to know how completions were added, and redo the same thing.

Add some script for the user to specify initialization actions

For example, for setting environment variables or do other bootstrapping.

Perhaps in src/init.sh or src/lib/init.sh?

Consider adding stdlib for trimming a string to $COLUMNS wide

Will help with DannyBen/rush-cli#38.

Expand explanation of tool

Found a helpful comment on HN thread to help describe the tool. Consider adding to Readme:

argparse equivalent for Bash.

You provide a YAML file listing commands, subcommands, arguments, and flags, and it automatically generates a Bash script that can parse and validate them, provide help messages, and run your code for each command.

It also lets you keep the actual code for each command and subcommand in separate files, which are merged together into one distributable Bash script at generation time.

It's basically a templating system to auto-generate argument parsing so you don't have to solve that again or deal with things like optparse.

Add ability to include custom shell scripts

Perhaps by watching for src/includes/*.sh and simply prenending these to the final script?

Add ability to generate extra includes

Once we support custom includes through #6, we can easily have bashly init generate some useful includes, like:

• config.sh - with functions for reading/writing INI files
• colors.sh - with functions for easily printing color output
• more suggestions?

Multiline help messages are not showing properly

• Fix command header help
• Fix flag help
• Fix arg help
• Fix root command header when using --help
• Fix environment variables
• Fix examples
• Add example(s)
• Add tests

`say` is not defined in Runfile

I'm trying to run shellcheck, but getting an error:

$ bundle exec run shellcheck
Traceback (most recent call last):
        9: from /home/wolf/src/vend/bashly/vendor/bundle/ruby/2.7.0/bin/run:23:in `<main>'
        8: from /home/wolf/src/vend/bashly/vendor/bundle/ruby/2.7.0/bin/run:23:in `load'
        7: from /home/wolf/src/vend/bashly/vendor/bundle/ruby/2.7.0/gems/runfile-0.12.0/bin/run:10:in `<top (required)>'
        6: from /home/wolf/src/vend/bashly/vendor/bundle/ruby/2.7.0/gems/runfile-0.12.0/lib/runfile/runner.rb:47:in `execute'
        5: from /home/wolf/src/vend/bashly/vendor/bundle/ruby/2.7.0/gems/runfile-0.12.0/lib/runfile/runner.rb:102:in `run'
        4: from /home/wolf/src/vend/bashly/vendor/bundle/ruby/2.7.0/gems/runfile-0.12.0/lib/runfile/runner.rb:133:in `docopt_exec'
        3: from /home/wolf/src/vend/bashly/vendor/bundle/ruby/2.7.0/gems/runfile-0.12.0/lib/runfile/action.rb:18:in `execute'
        2: from Runfile:20:in `block in <top (required)>'
        1: from Runfile:20:in `each'
Runfile:23:in `block (2 levels) in <top (required)>': undefined method `say' for main:Object (NoMethodError)

Not sure where say is supposed to come from, I assume a library somewhere should be defining it?

Add support for `-abc`, `-a=arg` and `--flag=arg` notation

Support for these flag notations can probably be achieved easily if we add a normalization function before calling parse_requirements:

bashly/lib/bashly/views/command/run.erb

Line 5 in 7592d5e

parse_requirements "$@"

Changing it to something like this:

normalize_input   # this function will set a new array named $input
parse_requirements "${input[@]}"

Possible implementation

#!/usr/bin/env bash
normalize_input() {
  local arg flags

  while [[ $# -gt 0 ]]; do
    arg="$1"
    if [[ $arg =~ ^(--[^=]+)=(.+)$ ]]; then
      input+=("${BASH_REMATCH[1]}")
      input+=("${BASH_REMATCH[2]}")
    elif [[ $arg =~ ^(-[^=])=(.+)$ ]]; then
      input+=("${BASH_REMATCH[1]}")
      input+=("${BASH_REMATCH[2]}")
    elif [[ $arg =~ ^-([^-].+)$ ]]; then
      flags="${BASH_REMATCH[1]}"
      for (( i=0 ; i < ${#flags} ; i++ )); do
        input+=("-${flags:i:1}")
      done
    else
      input+=("$arg")
    fi    
    
    shift
  done
}

declare -a input
normalize_input "$@"
for value in "${input[@]}"; do
  echo ": $value"
done

Output

$ ./test.sh -a -b arg -cde -f=arg --flag=arg -a=b=c --flag=anything=goes
: -a
: -b
: arg
: -c
: -d
: -e
: -f
: arg
: --flag
: arg
: -a
: b=c
: --flag
: anything=goes

Possibly unused code

This code:

bashly/lib/bashly/views/command/run.erb

Lines 17 to 24 in fb0f513

	<%= condition %> [[ ${args[--version]} ]]; then
	version_command
	elif [[ ${args[--help]} ]]; then
	long_usage=yes
	<%= name %>_usage
	elif [[ $action == "root" ]]; then
	root_command
	fi

Seems to be unused (for --version and --help), since it seems that for the root command these flags are intercepted here:

bashly/lib/bashly/views/command/fixed_flags_filter.erb

Lines 1 to 22 in fb0f513

	# :command.fixed_flag_filter
	case "$1" in
	<%- if short_flag_exist? "-v" -%>
	--version )
	<%- else -%>
	--version | -v )
	<%- end -%>
	version_command
	exit
	;;
	<%- if short_flag_exist? "-h" -%>
	--help )
	<%- else -%>
	--help | -h )
	<%- end -%>
	long_usage=yes
	<%= function_name %>_usage
	exit 1
	;;
	esac

Changing the offending code to this, does not break any test:

  <%= condition %> [[ $action == "root" ]]; then
    root_command
  fi

Consider making the change.