comonicon / comonicon.jl Goto Github PK

we should make precompile statements configurable, since it could be something non-test based.

I think we can consider using a runtime that spawns a process in backend like

https://github.com/dmolina/DaemonMode.jl

then we don't need to compile anything. Another reference is https://github.com/tkf/JuliaCLI.jl I think this is also beneficial for ComoniconGUI

currently only the required sections are displayed and parsed in the help info. We should definitely print the other parts as well, e.g if the doc string contains some sections other than Arguments, Options or Flags (like an Example section, or !!!note section etc.)

XRef: fonsp/PlutoUtils.jl#23

the regex should match ^-f$ first then try to match ^--flag=(.*) or -f(.*) etc. Or the later will give nothing.

since we have too many keywords for install, I think it's time to switch to a Comonicon.toml file to make things simpler.

Whenever I run the -V or --version command I never get a version number. All it prints out is the command name. Also, it prints the name of the casted function instead of the name of the executable as specified by the user when installing.

I have a script where I want to add something like this to the program description:

Parses filenames. Files must match pattern ^([^_]+_S\d+)_L001_R(1|2)_001.fastq.gz$

However, if I copy-paste the regex string directly into the description, it interprets the regex as markdown. Fair enough. However, if instead I interpolate the regex string in, like this:

"""
Foos the filepath.
* Files must match pattern $REGEX
"""
@main function foo(x)
    [ ... ]

Then it displays:

Foos the filepath
    •    Files must match pattern
        r"^([^_]+S\d+)L001R(1|2)001.fastq.gz:($(Expr(:incomplete,
        "incomplete: invalid string syntax")))

Version:

(@v1.5) pkg> add Comonicon#master
   Updating git-repo `https://github.com/Roger-luo/Comonicon.jl.git`
   Updating registry at `~/.julia/registries/General`
[ 2020-09-04 17:42:22.566 Warn  (/Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Types.jl:951) [Pkg.Types] ] -- could not download https://pkg.julialang.org/registries
  Resolving package versions...
Updating `~/.julia/environments/v1.5/Project.toml`
  [863f3e99] + Comonicon v0.6.2 `https://github.com/Roger-luo/Comonicon.jl.git#master`
Updating `~/.julia/environments/v1.5/Manifest.toml`
  [863f3e99] + Comonicon v0.6.2 `https://github.com/Roger-luo/Comonicon.jl.git#master`
  [5dd3f0b1] + MatchCore v0.1.0
  [8bf52ea8] + CRC32c

shell>

julia> versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM)
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

code:

import Comonicon.@main

"""
ArgParse example implemented in Comonicon.

# Arguments

- `x`: an argument

# Options

- `--opt1 <arg>`: an option
- `-o, --opt2 <arg>`: another option

# Flags

- `-f, --flag`: a flag
"""
@main function main(x; opt1=1, opt2::Int=2, flag=false)
    println("Parsed args:")
    println("flag=>", flag)
    println("arg=>", x)
    println("opt1=>", opt1)
    println("opt2=>", opt2)
end

Error:

 ./m.jl
ERROR: LoadError: could not open file /Users/xxx/Downloads/bin15/..cmd/cmd.jl
Stacktrace:
 [1] include(::String) at ./client.jl:457
 [2] top-level scope at /Users/xxx/.julia/packages/Comonicon/bSiwU/src/parse/cast.jl:275
 [3] include(::String) at ./client.jl:457
 [4] top-level scope at none:1
in expression starting at /Users/xxx/Downloads/bin15/m.jl:11

node command should store the module object, or we won't be able to locate where are the sub-commands actually defined.

I'm thinking about removing the version keyword in @main entirely since I believe one should not create a CLI that prints a different version compared to its host package. Current Comonicon will try to detect your project version automatically, I think I should just error when you don't write a version in the Project.toml file if the CLI is created inside a project module rather than a script.

currently we only support --option <arg>, should also support this.

@cast function f(x=true; y=(x? +1 : -1))
    ...
end

gives

ERROR: LoadError: LoadError: TypeError: non-boolean (String) used in boolean context

Over at https://github.com/ericphanson/SearchablePDFs.jl I've been trying out Comonicon to make a command line tool to generate searchable PDFs from rasterized PDFs. E.g.

searchable input.pdf output.pdf

However, if the pdf name has -'s in it, like Author - Year.pdf (generated by Zotero, in my case), then the CLI interprets it as passing options, even if I quote it, like

searchable "Author - Year.pdf" output.pdf

It seems to work OK if the pdf name is like Author-Year.pdf, i.e. without spaces.

it's great to have a generated autocompletion script, but sometimes we can provide better autocompletion by customizing the generated ones. Thus, it'd be necessary to have a custom auto-completion script get installed.

currently @command_main won't store the EntryCommand it generates, this will block #13 and kinda make generating other entries harder to be consistent with the current entry. We should store it as CASTED_COMMANDS["main"] = EntryCommand(...)

I think in principal we can compile distributable sysimgs and let users download it via BB, this will make command line apps much more easier to install and fast.

In some cases, we want to put CLI arguments into a file.
Perhaps it is a good idea for Comonicon to support it so that we program maker won't need to support it himself/herself.

For example:

app --conifg_file config.yaml #(or toml/json etc)

in which config.yaml is

a: 1
b: true

is taken the same as

app --a 1 -b

This feature should be optional and probably configurable in comonicon.toml .

currently, the workflow uses a GitHub action to create system images when the repo gets tagged. This does not work perfectly and is not very secure since no verification actually happens. We should try to make use of artifact system in some way.

But I haven't find out a smooth and simple to way to do it.

we don't have to print the arg type if it's a ::String, since it will be String by default (without any type declaration) anyways

we can provide an option in install to enable sysimg compilation of given project and use the generated sysimg in the CLI shebang.

to respect the shell convention, I think we should automatically convert an option name contains _ to - in shell options.

we now have a huge latency regression for CLI scripts, this is because in the v0.11.x rewrite, I start wondering whether we should have compile cache for scripts.

The problem is although, having compile cache for scripts give a huge boost for the test case, but it is actually not a solution to real world scripts. Real world scripts themselves
have a huge latency, cache the compile result of Comonicon won't solve this issue, thus I'm thinking that instead of doing compile scripts we need to think about doing deamon
mode execution.

however, this would need us to figure out how to do the workspace so that we can isolate different CLIs, and should they share the same process? I guess probably not, but then how do they get triggered?

@main function main(;niterations::Int=3000, seed::Int=1234, radius::Float64=1.5)
end

this gives

Error: command main expect at most 0 arguments, got 1, use -h or --help to check more detailed help info
  main [options]

generate autocompletion script for CLIs.

it will make things much more composable to allow defining new plugins for another CLI package, e.g if we have a CLI package Foo

module Foo

@cast function command1(arg1, arg2)
end

@command_main
end

since @cast will register all the command in Foo.CASTED_COMMANDS. we can insert plugins via

module Bar

using Foo

@cast function command2(arg1, arg2)
end

Foo.CASTED_COMMANDS["command2"] = CASTED_COMMANDS["command2"]

end

then we can just reuse the entry command (in some way, need to find out a proper way to recompile Foo) in Foo, and then Foo will get a set of new commands.

If I have a command that takes specific flags with certain arguments, a normal behavior in terminal applications is to display the help dialog on an incorrect input. Say my command, test, only takes -i <arg> or -q, if I type in test -i or test -m, it would be nice for it to fail nicer than a stacktrace of the juilia code and print something like Invalid arguments, output of test -h, and return an exit code of 1 to the terminal.

we should allow one to adjust display width (at least at the CLI config side) since sometimes the commands usage doc can be too long.

the other way is to print the usage doc beyond the doc string, e.g

foo [options...] <arg1> <arg2::Float64>
       some docstrings.....

the current implementation of generating Julia AST from our CLI IR is a bit messy. we should rework this and make the code more readable at some point (which also improves extensibility)

we need to add autoload -Uz compinit && compinit to .zshrc in some way, or the auto completion won't be enabled.

once I was thinking to only allow one syntax on this - which I still think so. But then we need to choose a syntax -s=<value> or -s<value>? or support both? I find I'm actually more used to -s=<value> syntax myself.

I'd like to see what other people think, please comment if you have a preference too.

I think a nice feature would be the ability to add a default set of options for a given program, maybe in the form of an @default macro?

Consider the use case where you want to display the output of -h if someone runs the command with no parameters or arguments, you might be able to write something like @default -h. Not sure what the best design decision here would be.

After reading the front page I still don't know how to make a cli.

Seems to have lots of details that's distracting like ➜ Comonicon git:(master) What's that for? It's master branch of your package. But it doesn't help me understand how to make a cli?

A simple step by step guide on how to make a CLI would be nice.

we currently won't build the system image by default, if the download fails, so that we can throw a nice error msg during building to the REPL and let users to build it manually via a generated function XX.comonicon_build(). I'd like to discuss whether we should keep this behavior or do something else, especially given that we are not able to build system images for MacOS (Roger-luo/IonCLI.jl#24)

I'll just leave this here as a memo, since currently the compiled app still needs around 300MB space for a single CLI, which is not ideal. But since according to JuliaCon talk, this will be improved in the future, I'll wait for this comes out and start thinking about it.

@Roger-luo

When upgrading to 0.11 from 0.10.2, I had a precompilation error which appears to be due to the fact that cast_args method doesn't seem to be able to handle a zero-length args input gracefully. I have a couple commands that I've casted that have zero arguments and so it didn't like these at all.

Snapshot of stacktrace:

ERROR: LoadError: BoundsError: attempt to access 0-element Vector{ComoniconTypes.Argument} at index [0]
Stacktrace:
  [1] getindex
    @ .\array.jl:801 [inlined]
  [2] last
    @ .\abstractarray.jl:437 [inlined]
  [3] cast_args
    @ ~\.julia\packages\Comonicon\qwttE\src\cast.jl:297 [inlined]
  [4] cast(f::Function, name::String, args::Vector{Comonicon.JLArgument}, options::Vector{Comonicon.JLOption}, flags::Vector{Comonicon.JLFlag}, line::LineNumberNode)
    @ Comonicon ~\.julia\packages\Comonicon\qwttE\src\cast.jl:278

I've got a PR that fixes this on my end that I will submit for your approval.

Hello, I was just starting to look at Julia tools to help build command line interfaces. I was wondering if it is possible to create nested commands in an app. For example, if I wanted to create something like the docker CLI, then there are nested commands like docker run and docker ps, among others. I was just wondering if comonicon can handle something like this yet, or if this is an enhancement request. Thanks so much.

Add a switch so that Comonicon can generate targeting CLI or GUI.

I want to support an unknown number of parameters, but below doesn't seem to work:

@cast function f(xs::Int...)
    ...
end

Neither does this

@cast function f(xs::Vector{Int})
    ...
end

What's the correct way to achieve what I want?

Could you explain the main differences so that I can more easily choose between the available approaches in future projects?

https://github.com/docopt/DocOpt.jl

Perhaps adding a section to the README would be helpful.

The following message is the output ofion release -h with IonCLI v0.5.1. Note that wantto, specificversion, bydefault, thepackage, and others at the beginning of each line; there should be one space here.

  release [options] [flags] <version> [<path>]

release a package.

Args

  <path>                            optional argument. path to the project you
                                    wantto release, default is the current working
                                    directory.

  <version>                         version number you want to release. Can be a
                                    specificversion, "current" or either of (major, minor,
                                    patch)

Options

  -b, --branch <branch name>        branch you want to register, use master branch
                                    bydefault.

  -r, --registry <registry name>    registry you want to register the package. If
                                    thepackage has not been registered, ion will try
                                    toregister the package in the General registry.
                                    Orthe user needs to specify the registry to
                                    registerusing this option.

currently I'm generating the precompile statement from @command_main since this package will only create one single function needs to be compiled. However, it seems that if I generate

precompile(Tuple{typeof($(__module__).command_main), Array{String, 1}})

directly, the runtime will be a bit slower than using the precompile statement generated by SnoopCompile

function _precompile_()
    ccall(:jl_generating_output, Cint, ()) == 1 || return nothing
    precompile(Tuple{typeof(IonCLI.command_main), Array{String, 1}})
end

_precompile_()

I'm not sure why, this needs some further investigation.

A lot of times when I make a command line tool using Julia, I like to have the ability to provide arbitrary number of arguments such as

rm [options] [flags] args... deletes ever argument in args
mkdir [options] [flags] args... makes all the directories listed in args

Currently when I try to do this, I get an invalid syntax for command line entry: arg::String...

Could this be extended to support variable argument lengths?

It seems #34 is causing this error on Linux somehow, any idea? @mehalter

It breaks the build of PlutoCLI https://github.com/Roger-luo/PlutoCLI.jl

  Building PlutoCLI → `~/.julia/packages/PlutoCLI/yYmHi/deps/build.log`
┌ Error: Error building `PlutoCLI`:
│ stty: 'standard input': Inappropriate ioctl for device
│ [ Info: generating /home/roger/.julia/bin/pluto.jl
│ [ Info: generating /home/roger/.julia/bin/pluto
│ ERROR: LoadError: failed process: Process(`stty raw`, ProcessExited(1)) [1]
│
│ Stacktrace:
│  [1] pipeline_error at ./process.jl:525 [inlined]
│  [2] run(::Cmd; wait::Bool) at ./process.jl:440
│  [3] run at ./process.jl:438 [inlined]
│  [4] prompt(::IOStream, ::String, ::Bool) at /home/roger/.julia/packages/Comonicon/1Kc5X/src/tools/tools.jl:14
│  [5] prompt(::String, ::Bool) at /home/roger/.julia/packages/Comonicon/1Kc5X/src/tools/tools.jl:5
│  [6] write_path(::String, ::Bool, ::Base.EnvDict) at /home/roger/.julia/packages/Comonicon/1Kc5X/src/tools/build.jl:585
│  [7] write_path at /home/roger/.julia/packages/Comonicon/1Kc5X/src/tools/build.jl:571 [inlined]
│  [8] install_env_path(::Bool) at /home/roger/.julia/packages/Comonicon/1Kc5X/src/tools/build.jl:561
│  [9] install_script(::Module, ::Dict{String,Any}) at /home/roger/.julia/packages/Comonicon/1Kc5X/src/tools/build.jl:238
│  [10] install at /home/roger/.julia/packages/Comonicon/1Kc5

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

In many cases, especially on Windows, a GUI might be helpful to promote the usage of a package.

The GUI could be just a form that maps all arguments to controls and a START button.

I hope it supports different GUI backends like imGUI, Blink etc.

Probably it is better to implement this in another package rather than modifying Comonicon so that
Comonicon focuses on CLI only.

Now that PATH and FPATH are not being done interactively when not in an interactive session, it seems like you cannot have a build script run using julia --project deps/build.jl to set up the PATH or FPATH even when it is specified explicitly in the arguments. I think it would be best to move the interactive check inside of the install_env_path and install_completion functions where if quiet or !isinteractive(), then it takes the default or specified value, or if it is !quiet && isinteractive() and nothing is specified manually, then ask.

I think if this approach is taken, it would make sense that by default it does nothing (default export_path, completions = false, false) and if they are specified (either are set to true) then you don't need to ask even if its interactive. I think this would help make it more explicit to the user when things will be changed in the system especially when using --quiet flag.

Now that I think about it, if you decide to default to false, there might not be a need for a quiet flag other than to do less verbose output/no output which is how it is normally used.

Let me know what you think of these ideas and if you want, I can help get the ball rolling with these changes!

Also thanks for being so responsive and quick to replying/changing stuff! This project is looking great!

I think to be consistent with what we are printing in the help info, we should allow one to use Args as the docstring section name for arguments.

this requires:

1. add asset field in Comonicon.toml
2. add this when packing the tarball