erichs / composure

Don't fear the Unix chainsaw

 e88~~\  e88~-_  888-~88e-~88e 888-~88e   e88~-_   d88~\ 888  888 888-~\  e88~~8e
d888    d888   i 888  888  888 888  888b d888   i C888   888  888 888    d888  88b
8888    8888   | 888  888  888 888  8888 8888   |  Y88b  888  888 888    8888__888
Y888    Y888   ' 888  888  888 888  888P Y888   '   888D 888  888 888    Y888    ,
 "88__/  "88_-~  888  888  888 888-_88"   "88_-~  \_88P  "88_-888 888     "88___/
                               888

# Composure: don't fear the Unix chainsaw

These light-hearted functions make programming the shell easier and more intuitive:

• Transition organically from command, to function, to script
• Use an unobtrusive help system with arbitrary shell metadata
• Automatically version and store your shell functions with Git

static analysis and automated tests:

Demo!

Composing a simple network monitoring script (4 minutes)

Compatibility

Composure is POSIX-compliant, and is known to work on ksh93, zsh, and bash, on osx and linux.

Please feel free to open an issue if you have any difficulties on your system.

Installing

Clone this repo, into a directory of your choice and source it from your shell's profile or rc file.

On Bash:

    cd /your/favorite/directory
    git clone https://github.com/erichs/composure.git
    cd composure
    echo "source $(pwd)/composure.sh" >> ~/.bashrc   # or, ~/.bash_profile on osx

Note for users with versions prior to 1.4.0: Previous installation methods will work, but cloning the repo is now recommended, as it makes the glossary and reference commands 100x faster.

Craft - Draft - Revise - Write

journey
    title Typical Usage
    section Craft
      Create Useful Shell Command: 1: Shell
    section Draft
      Give it a name with 'draft': 2: Editor, Git
      Use your new shiny function: 3: Shell
    section Revise
      Edit/maintain your function with 'revise': 4: Editor, Git
    section Write
      Persist to shell scripts with 'write': 5: Shell
      Profit $$$: 6: You

Loading

Crafting the command line

REPL environments are great for trying out programming ideas and crafting snippets of working code, aren't they? Composure helps you make better use of the REPL environment constantly at your fingertips: the shell.

Many Unix users I know like to iteratively build up complex commands by trying something out, hitting the up arrow and perhaps adding a filter with a pipe:

  $ cat servers.txt
  bashful: up
  doc: down

  up-arrow

  $ cat servers.txt | grep down
  doc: down

  up-arrow

  $ cat servers.txt | grep down | mail -s "down server(s)" [email protected]

Composure helps by letting you quickly draft simple shell functions, breaking down your long pipe filters and complex commands into readable and reusable chunks.

Draft first, ask questions later

Once you've crafted your gem of a command, don't throw it away! Use 'draft ()' and give it a good name. This stores your last command as a function you can reuse later. Think of it like a rough draft.

  $ cat servers.txt
  bashful: up
  doc: down

  up-arrow

  $ cat servers.txt | grep down
  doc: down

  $ draft finddown

  $ finddown | mail -s "down server(s)" [email protected]

Revise, revise, revise!

Now that you've got a minimal shell function, you may want to make it better through refactoring and revision. Use the 'revise ()' command to revise your shell function in your favorite editor.

• generalize functions with input parameters
• add or remove functionality
• add supporting metadata for documentation

  $ revise finddown
  finddown ()
  {
      about finds servers marked 'down' in text file
      group admin
      cat $1 | grep down
  }

  $ finddown servers.txt
  doc: down

Get it in Writing

When it is time to put your function or functions to use in a shell script, just call write:

  $ write finddown > finddown.sh

Edit the main() function, chmod +x, and you're ready to go!

Arbitrary shell metadata!

Composure uses a simple system of dynamic keywords that allow you to add metadata to your functions. Just call 'cite ()' to initialize your new keyword(s), and use them freely in your functions:

  foo()
  {
      cite about
      about perform mad script-foo
      echo 'foo'
  }

Retrieve your metadata later by calling 'metafor ()':

  typeset -f foo | metafor about  # displays:
  perform mad script-foo

By default, composure knows the keywords: about, param, group, author, and example.

These default keywords are used by the help system:

Intuitive help system

The 'glossary ()' function will automatically summarize all functions with 'about' metadata. If called with a 'group' name as a parameter, it will summarize functions belonging to that group.

To display apidoc-style help for a function, use 'reference ()'.

  $ glossary   # displays:
  cite                creates a new meta keyword for use in your functions
  draft               wraps last command into a new function
  finddown            finds servers marked 'down' in text file
  foo                 perform mad script-foo
  glossary            displays help summary for all functions, or summary for a group of functions
  metafor             prints function metadata associated with keyword
  reference           displays apidoc help for a specific function
  revise              loads function into editor for revision
  write               writes one or more composed function definitions to stdout

  meanwhile

  $ glossary admin   # displays:
  finddown            finds servers marked 'down' in text file

  and

  $ reference draft  # displays:
  draft               wraps last command into a new function
  parameters:
                      1: name to give function
  examples:
                      $ ls
                      $ draft list
                      $ list

Git integration

If you already use git, installing composure will initialize a ~/.local/composure repository, and store and version your functions there. Just use 'draft ()' and 'revise ()', they automatically version for you.

Composure supports the XDG Base Directory specification, and will respect your local XDG_DATA_HOME environment variable.

Why do this?

• the latest version of any function you've composed may always be sourced from your composure repo
• never throw away code--keep your one-off functions in your composure 'junk drawer', and grep through it later for long-forgotten gems
• every version of every function you write is always available to you via basic git commands

Persistent access

Draft or revise a function, and the latest version is automatically sourced into your current shell environment. By default, composure automatically sources all of your composed functions when you source the composure.sh script. If you are concerned about shell startup time, have many hundreds of versioned shell functions, or otherwise want to control which functions are loaded from your composure repository, you may disable the default behavior by adding the following line to your shell's startup script:

export LOAD_COMPOSED_FUNCTIONS=0

Credits

Composure grew out of ideas taken from from Gary Bernhardt's hilarious talk The Unix Chainsaw (31 minutes), which refers to the Elements of Programming described in MIT's SICP text:

• primitive expressions
• means of combination
• means of abstraction

Known Issues

'glossary ()' and 'reference ()' do not support nested functions with metadata.

revise works well if your editor is terminal-based, like Emacs or Vim. If you use a windowed editor like Atom, VSCode, or Sublime, you will need to check to see if your editor supports a flag argument that allows it to wait for the files to be closed before returning. If this is supported, you can create a small script to launch your editor in this mode, and specify that script path in your EDITOR var. See #10.

composure's Issues

Fix pipe/&& precedence for some bash versions

The fix introduced by c96e44a does not work for me. I use bash "4.3.11(1)-release (x86_64-pc-linux-gnu)" on Ubuntu 14.04. The test function

fc_check () {
  echo history:
  history | tail -3

  echo fc check
  fc -ln -1
  fc -ln -1 | head -1
  (fc -ln -1)
  (fc -ln -1 | head -1)
  echo $(fc -ln -1)
  echo $(fc -ln -1 | head -1)
}

produces the following output

history:
  552  echo c2
  553  echo c3
  554  fc_check 
fc check
     echo c3
     echo c3
     echo c3
     echo c3
echo c3
fc_check

The last two lines are quite surprising. My fix for this strange behaviour looks like this:

--- composure.sh_upstream   2014-11-20 00:27:16.726785858 +0100
+++ composure.sh    2014-11-20 00:28:57.223999557 +0100
@@ -284,7 +284,7 @@
   if [ -z "$num" ]; then
     typeset lines=1
     # some versions of 'fix command, fc' need corrective lenses...
-    (fc -ln -1 | grep -q draft) && lines=2
+    lines=$(fc -ln -1 | grep -q draft && echo 2 || echo 1)
     # parse last command from fc output
     cmd=$(fc -ln -$lines | head -1 | sed 's/^[[:blank:]]*//')
   else

Empty file created?

~ ❯❯❯ awk ' { if ( length > L ) { L=length} }END{ print L}' ~/.zprezto/init.zsh
101
~ ❯❯❯ draft longest_line
draft:46: file exists: /tmp/draft.4fOX
I see you don't have a /home/pablo/.local/composure repo...

would you like to create one? y/n: y
creating git repository for your functions...
Initialized empty Git repository in /home/pablo/.local/composure/.git/
[master (root-commit) 282a875] initial commit
 1 file changed, 1 insertion(+)
 create mode 100644 README.txt
[master bd9b16d] Draft longest_line: initial draft
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 longest_line.inc
zero-length file, revision aborted!

I'm posting a log when I run the draft command after running set -x in case it's useful.

Should say `revise` saves and moves on before any edits if your editor returns on open

Ran into the same problem as #10

If I wasn't so persistent about this I would've thought this program was no longer working. Luckily it works as long as you change editors or change that part of the code

Edit: seems the "wait" argument is an option for vscode as well. Might want to mention the insight that the editor might simply have that

stripping semicolons breaks case statements

Write function outputs cite function as well as parameters

Hello,

Is the below line correct? I suspect it should read typeset -f "$@"

composure/composure.sh

Line 526 in 55fdb9f

typeset -f cite "$@"

glossary does not generate output as shown in Readme.md

Hi,

glossary does not show the built in commands. Are those excluded meanwhile? So please update your example.

metafor's "sed-fu" seems overly complicated. Triggers bug in busybox with musl. Can it be simplified?

The buggy behavior, which only happens in a busybox sed which uses musl's libc (take a look in alpine:latest, with apk add sed), is that the leading quotation marks are not removed (this is counter to the POSIX regex spec):

→ keyword=hello
→ LINE="$keyword 'foobar'" # a trivial use with quotes
→ echo $LINE | busybox sed -n "/$keyword / s/['\";]*\$//;s/^[ 	]*\(: _\)*$keyword ['\"]*\([^([].*\)*\$/\2/p" # musl bug
'foobar
→ echo $LINE | sed -n "/$keyword / s/['\";]*\$//;s/^[ 	]*\(: _\)*$keyword ['\"]*\([^([].*\)*\$/\2/p"
foobar

I'd like to suggest an edit to this sed-fu, but I'm not certain which parts of this functionality are intentional, and which are accidental. The guiding principle seems to be "this should work a little like echo, but we definitely do not want to eval"

The behavior of the current sed-fu appears to be:

• Search for the keyword (we will only print lines that match it)
• The keyword must be the first word on the line, but it can be : _keyword because of Allow functions with metadata to run before load
• The keyword and the whitespace preceding it are removed
• All leading and trailing quotes on the argument are removed (even if they are of different lengths [0], or are not symmetrical [1], though we'd expect the outer pair to match in normal usage)
• A trailing semicolon is removed, whether or not [2] it is within the quotes
• ([^([].*\)* looks like it's trying to do something like exclude open braces, but all that does is start the match earlier [3]
• Attempting to quote two different arguments is interpreted as allowed internal quotes [4]
• The character immediately following the keyword must be exactly a space. Tabs are disallowed. (typeset probably normalizes this so it wouldn't matter)

→ LINE="$keyword 'foobar;\"''\""; !echo # Allowed: uneven[0], mismatched[1] quotes and a semicolon within the quotes [2]
foobar
→ LINE="$keyword '(foobar;'"; !echo # Odd: A leading paren in the quotes [3], semicolon within the quotes [2]
'(foobar
  → LINE="$keyword 'foo' 'bar'"; !echo # Unsupported: two different quoted arguments [4]
foo' 'bar

I'm treading on Chesterton's fence, which prohibits me from removing these unless I know what they're for. But several of these don't look like they do anything for us. If they don't, I'd propose a shorter sed script:

sed -En "s/['\"]?;?\$//g; s/^\s*(:\s+_)?${keyword}\s+['\"]?//p"

This sed invocation does not break in musl, but it also makes several opinionated changes. I don't want to pass over them without calling them out:

• Only one quote is removed from the start and end - quotes are still allowed to mismatch in kind and number, but that number is only zero or one.
• The trailing semicolon is not removed if it's within the quotation marks
• Whatever ([^([].*\)* is trying to do; it doesn't
• Whitespace is more flexible - no constraints on spaces vs. tabs or how many of them there are (beyond syntax requirements). This can harm readability where tabs are never expected,

→ keyword=hello
→ LINE="$keyword 'foobar'" # a trivial use with quotes
→ echo $LINE | busybox sed -n "/$keyword / s/['\";]*\$//;s/^[ 	]*\(: _\)*$keyword ['\"]*\([^([].*\)*\$/\2/p" # musl bug
'foobar
→ echo $LINE | busybox sed -En "s/['\"]?;?\$//g; s/^\s*(:\s+_)?${keyword}\s+['\"]?//p" # worked around
foobar

Do these changes seem like improvements, or do they miss the mark?

write function is broken

When I try to write a single function to a file I don't get sensible output.

$ echo "Hello World!"
Hello World!

$ draft hello
[master a23d548] Draft hello: Initial draft
 1 file changed, 1 insertion(+), 1 deletion(-)
Git Comment: 
On branch master
nothing to commit, working directory clean

$ write hello
#!/usr/bin/env bash
for f in about author example group param version
do
  eval "$f() { :; }"
done
unset f
main() {
  echo "edit me to do something useful!"
  exit 0
}

main $*

$ typeset -f hello
hello () 
{ 
    author 'IsoLinearCHiP';
    about '';
    param '';
    example '';
    group '';
    echo "Hello World!"
}

I think the problem is caused by b875c76 . The change in line 497. As far as I can tell, that line is trying to typeset -f "cite hello", which obviously does not work as there is no function cite hello with a space in it.

The previous version was not doing any harm, though I think the quotes are superfluous anyway, since spaces are not allowed in function names to my knowledge.

Changing that line back to the previous version fixes the error for me.

typeset -f fails on GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)

Updating all the typeset -f references to declare -f resolved everything.

(ansible) DESKTOP-R0H5FVK:composure earchibald$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.

drafting over existing commands

I think it's a problem that you can draft a command that already exists, and it just clobbers the old one without asking. At least it can be undone in git.

Do you think instead it ought to bail out with an error and suggest revise, ask if you want to overwrite or revise it, or just call revise instead?

draft function name clash

'draft' function is defined in the lib/composure.bash

Unfortunately the name of this function is clashing with the draft binary Draft actively developed for the K8S domain.
Simply renaming 'draft' function here to say 'draft_fun' would do.

ref: originally raised as #1453 for Bash-It

_letterpress should wrap better on skinny terminals

For example, I have a terminal width of 83.

_longest_function_name_length | awk '{print $1 + 5}'

Returns 40. If any about info is greater than 43 characters, it just wraps and is difficult to read.

Could composure leverage the column utility to do this?

shellcheck test

Could someone explain to me how the current tests are working?

Locally shellcheck throws a lot of errors, and when I copy/paste composure into shellcheck.net there are a lot of complaints too.

I see that the tests are using shellcheck.php, but are there some disables that I'm unaware of?

Glossary (_typeset_functions, _shell) fail in script

(Re)-using composure 'glossary' inside a script fails when _shell does not detect bash, but the scriptname instead. Causing _typeset_functions to switch to z-shell mode resulting in a long list of typeset errors:

typeset: cannot use `-f' to make functions
[...]
typeset: `[1]="_typeset_functions_about"': not a valid identifier
[etc]

A typical offending ps -p $$ line as used by _shell (detects shell 'help' iso. 'bash' here)

81515 ttys004    0:00.08 bash /Users/berend/.local/composure/Composure/main.sh c help

Solution (BSD Darwin and GNU) add ps option to select command-name (w/o args):

ps -p $$ -o comm

functions not loaded; glossary does not list functions in repo

Edit: See fix attached.

Given

• installed composure.sh using basher
• symlinked ~/.local/composure to checkout (a submodule in ~/.conf)
• can draft/revise functions OK

When I synchronize the setup with another box, same setup, I do not see the functions. Even though the checkout has the files. I can revise them and then they load, but the initial shell env does not.

To test I've used:

$ export LOAD_COMPOSED_FUNCTIONS=1
$ . ~/.basher/cellar/bin/composure.sh
$ glossary
cite          creates one or more meta keywords for use in your functions
draft         wraps command from history into a new function, default is last command
glossary      displays help summary for all functions, or summary for a group of
              functions
metafor       prints function metadata associated with keyword
reference     displays apidoc help for a specific function
revise        loads function into editor for revision
write         writes one or more composed function definitions to stdout

What could cause composure.sh to fail to load the functions initially?

use GNU printf from coreutils if available

after an upgrade to OSX El Capitan, the printf shell built-in no longer works as expected. Use gprintf, from the coreutils package, if it is available.

Slow glossary command

Running glossary command seems very slow on my system (zsh 5.0.7 x86_64-apple-darwin14.0.0). I only have one user defined function. Is this expected?

cite                                                                  creates one or more meta keywords for use in your functions
draft                                                                 wraps command from history into a new function, default is last
                                                                      command
glossary                                                              displays help summary for all functions, or summary for a group of
                                                                      functions
metafor                                                               prints function metadata associated with keyword
ping.google                                                           Test connection via Google DNS (8.8.8.8)
reference                                                             displays apidoc help for a specific function
revise                                                                loads function into editor for revision
write                                                                 writes one or more composed function definitions to stdout
15.5386480000s elapsed

revise commits not being made with external editor

I'm using sublime text with it's hand subl in my bin, and composure opens the file just fine, but when it's saved in my editor, and I head back to the terminal, I enter the commit message, but never actually commits the changes. A subsequent revise script opens the original. Is something else required besides simply setting up my $EDITOR ?