Make it possible to write 'xargs' about docopt

[keleshev]

I think I changed my mind and started appreciating disable_interspersed_args, so If someone does a pull-request into a new branch (is it possible in git?) or into develop, I will include it in the next version.

However, I don't like the name, we need to come up with something else. Maybe posixly_correct similar to getopt? Or maybe something else?

[keleshev]

I'm not sure I understand you:

1. you yourself declared -0 to be an option synonymous with --null
2. if you want to use -- you should put it in your usage message (e.g. [--])

[EvanED]

What I want is a mode for the parser to stop considering anything after the first positional argument to be anything but another positional argument. So in xargs -0 foo, the -0 is an option to xargs and foo is COMMAND and there's no ARG. However, if I move the -0 after foo, xargs foo -0, now the -0 should be considered an ARG instead of an option to xargs (because it appears after the first positional argument, foo).

For a concrete (if strange) example with existing utilities, suppose I want to run several time occurrences passing in the -p flag to time ("precise posix format"). I can do this with xargs time -p:

$ printf "first\nsecond\nthird" | xargs -n1 time -p echo
first
real 0.00
user 0.00
sys 0.00
second
real 0.00
user 0.00
sys 0.00
third
real 0.00
user 0.00
sys 0.00

and this is very different from running xargs -p time (I've edited the following a little to make it easier read):

$ printf "first\nsecond\nthird" | xargs -n1 -p time echo   
time echo first ?...y
first
0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2256maxresident)k
0inputs+0outputs (0major+171minor)pagefaults 0swaps

time echo second ?...y
second
0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2256maxresident)k
0inputs+0outputs (0major+171minor)pagefaults 0swaps

time echo third ?...y
third
0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2256maxresident)k
0inputs+0outputs (0major+171minor)pagefaults 0swaps

however, by experiment it seems that you can't get the first behavior with docopt: if xargs used docopt, it would always slurp the -p as an option to xargs instead of passing it down to time like in the first case.

This also seems to be true if you use a usage string of xargs.py [-p] COMMAND [ARG...]

[EvanED]

Lots of standard Linux utilities have an interface like this, and all of them (which take options at least, which excludes one or two) suffer from this problem: chroot, env, nice, nohup, stdbuf, timeout, sudo, and ssh, as well as "third party" programs like valgrind.

As perhaps a clearer example: compare env --help ls vs env ls --help. The former displays help for env, and the latter displays help for ls. Any argument parser (including docopt) that allows options to appear at any point will always behave in the first way.

[keleshev]

Well, options are not positional arguments; their position should not matter.

There are all kinds of funny CLI out there, also in Unix, take tar: you can do tar -xf file.tar or you can drop the dash tar xf file.tar. Is this a feature or a misfeature?! Should docopt support this? Probably not.

docopt is not made for all-kinds-of CLI interfaces; it is made only for sane CLI interfaces.

As I understand, you propose to make options' position matter, right? If yes, then this will not happen. If that is not what you propose, please clarify what DSL/API changes you are proposing.

[keleshev]

If I was to implement xargs I would pass a subset of argv to docopt. I did something similar when I implemented git CLI in docopt's new branch any-options:

https://github.com/docopt/docopt/blob/e55ba698ce5536fbd65eade5f5ab25b962289d10/examples/git/git.py

    # Make argv for subparsers that does not include global options:
    i = sys.argv.index(args['<command>'])
    sub_argv = sys.argv[i:]

[EvanED]

You say docopt is only for sane CLI interfaces, but what I want is a sane interface -- I'd argue more sane (and beautiful) than requiring the user to say env -- ls --help or whatnot. And I still think it's worthwhile to add explicit support. (For reference, the optparse module has a disable_interspersed_args flag you can set, and argparse has an argparse.REMAINDER feature that you can use.)

Your suggestion to pass a subset of argv is a useful one though, though I think an actual implementation would turn out to be a bit convoluted; it seems like you'd need to call docopt once to get args['<command>'] so you could extract its index, then call it again with a slice of the array up to that index, then ignore the second args['<command>'] and instead build it manually from argv starting at that index. Here's what I came up with -- this requires the any-options branch:

"""Usage:
  xargs.py [options] [--] [COMMAND [ARG...]]

Options:
  --help                  Show help
  --null,-0               Input items terminated by 0
"""

from docopt import docopt
import sys

if __name__ == '__main__':
    # Do a first parse to find where COMMAND is
    args = docopt(__doc__, help=False, any_options=True)
    first_positional = sys.argv.index(args["COMMAND"])

    # Do a second parse to get our program's options
    non_positionals = sys.argv[:first_positional+1]
    args = docopt(__doc__, version="1.0", argv=non_positionals)

    # Then stick in the positionals
    args["COMMAND"] = sys.argv[first_positional]
    args["ARG"] = sys.argv[first_positional+1:]

    print(args)

(Updated to change the variable positionals to non_positionals; the original name, of course, was backwards.)

[EvanED]

BTW, as a concrete proposal, I'd suggest one of two things: the first is a keyword argument to docopt() that stops looking for options at the first positional argument (a la disable_interspersed_args), and the second would be a syntax for sort of a "greedy argument repetition", so I could say xargs [options] [COMMAND [ARGS+++]] or something (just to make up a syntax).

Actually the first seems like it'd be really easy: http://pastebin.com/v52uRped (I could turn that into a pull request if you'd like). The diff is relative to the any-options branch because that's what I had checked out.

[keleshev]

I'm slightly against this feature. If I see foo --bar -- spam --eggs I can immediately understand what is happening, which option relate to which command, it is just more readable. So I would never advise to use "interspersed" arguments in new programs. We already have xargs and tar, there is no need to re-implement them. And no need to repeat their mistakes.

So unless this feature is heavily requested, it will not make it in docopt.

[EvanED]

I'll make one more attempt at arguing for this feature, then I'll leave you alone. :-) Personally, I feel that the unique aspect about command line syntax and the shell in general is that, unlike most programming languages, it is and should be optimized for writing, rather than reading, because I'd say that for virtually everyone, almost all the time using the shell language is interactively working at the shell. And because of this, losing explicitness in a trade for conciseness is a good trade. For simple cases like the programs I mentioned above, I don't think I've ever been confused by what option goes with which command.

For instance, if I have a program that crashes, I want to hit ctrl-P to recall the previous command, ctrl-A to go to the start of the line, type valgrind and hit enter -- I don't want to worry about whether my command takes options that conflict with valgrind's, and I don't want to spend the extra effort to type --.

Adding to this whole thing is the fact that the allowance for options to come anywhere is not a POSIX or traditional Unix thing -- I'm pretty sure that's a GNU convention. And it's one that hasn't been adopted everywhere:
Linux:

$ touch abc --help
Usage: touch [OPTION]... FILE...
...
$ ls
$

SunOS

$ touch abc --help
$ ls
--help  abc
$

(Actually --help isn't the best test in the Sun case because even touch --help isn't legal. But the same thing happens with touch abc -a, and touch -a abc is legal and does different things.)

If we're having a show of hands, I would also lean towards implementing something like disable_interspersed_args. There are many kinds of programs that make it their business to execute other programs with given arguments (interpreters, job automators...) and it seems odious to demand that they all prepend a -- to their invocations.

[EvanED]

Heck, even Python has this problem: python --help foo.py is very different than python foo.py --help. Have you ever gotten confused about whether the arguments to your Python script are going to your script or to Python? This sort of syntax is far from non-standard. (raskug's mention of interpreters made me think of Python, but of course the same thing is probably true of, well, nearly all other language interpreters out there.)

And for context, Boost's program_options library for C++ has the same problem, and I've been trying (not very much) to get a patch accepted to it as well for a while. From what I can tell, it seems that someone expresses some degree of interest in this feature every couple months.

[fuadsaud]

I agree with disable_interspersed_args

Also agree that the handling for xargs.py -- foo is a bit strange.

[encukou]

Perhaps a syntax like xargs [options] [--] <command> [args ...]? I'd appreciate if the information was present in the usage string (even though, sadly, it's not there in most existing programs).

[keleshev]

Actually I totally going to implement this. Thanks a lot @EvanED for suggestion and for arguing for it! I now like this idea so much I'm going to implement interspersed_args instead of any_options (#4).

The reason is that interspersed_args works so much better for program dispatch comparing to any_options.

Git CLI using any_options:

...
    args = docopt(__doc__,
                  version='git version 1.7.4.4',
                  help=False,
                  any_options=True)

    # Handle -h|--help manually.
    # Otherwise `subcommand -h` would trigger global help.
    if args['<command>'] is None:
        print(__doc__.strip())
        exit()

    # Make argv for subparsers that does not include global options:
    i = sys.argv.index(args['<command>'])
    sub_argv = sys.argv[i:]

    if args['<command>'] == 'add':
        # In case subcommand is implemented as python module:
        import git_add
        print(docopt(git_add.__doc__, argv=sub_argv))
...

Git CLI using interspersed_args:

...
    args = docopt(__doc__,
                  version='git version 1.7.4.4',
                  interspersed_args=False)

    if args['<command>'] == 'add':
        # In case subcommand is implemented as python module:
        import git_add
        print(docopt(git_add.__doc__, argv=args['<args>']))
...

It took me time to realize that.

The only question -- how to name it?

interspersed_args=True
disable_interspersed_args=False
posixly_correct=False
options_first=False

Any other ideas? Which one do you like most?

[keleshev]

what do you think, @docopt guys?

interspersed_args sounds best to me, though I can't really rationalize why. disable_interspersed_args has the benefit of having a precedent, but it's just unnecessarily verbose imo.

[EvanED]

I'm glad I convinced you in the end. :-)

If you are interested in my naming opinions, disable_interspersed_args, in addition to being verbose, is a negative, which I usually don't like; I would thus also vote interspersed_args. Something like require_options_first=False would be my second choice. (You could also consider interspersed_options or interspersed_opts.)

(posixly_correct, even though that corresponds to getopt's interpretation, I think sounds like it could have other implications (are there other GNUisms that it would exclude? the answer may be "no", but I think that's non-obvious.) options_first doesn't seem as clear to me.)

[keleshev]

Calling it posixly_correct can be misleading, since docopt has other deviations from posix (like --gnu-style options and commands).

interspersed_options sounds good. Maybe intersperse_options is even better?

Looked up thesaurus now, maybe intermix(ed)_options or mix(ed)_options? mix_options is short and sweet, but looses some of the semantics.

Semantically it should be probably intermix_options_and_arguments, or allow_options_follow_arguments or allow_options_after_arguments.

[keleshev]

options_precede_arguments?
options_before_arguments?
options_follow_arguments?
arguments_follow_options?
arguments_after_options?

I'm somewhat inclined towards options_first, but I really want to make this one right.

[keleshev]

0.6.0 is now released with options_first argument as discussed above. Thanks a lot guys and especially @EvanED for suggestion and discussion. I could have made a big mistake with any_options. Now, I think, docopt is much better with options_first.