denoland / deno_task_shell

Cross-platform shell for deno task.

deno_task_shell

// parse
let list = deno_task_shell::parser::parse(&text)?;

// execute
let env_vars = HashMap::from(&[
  ("SOME_VAR".to_string(), "value".to_string()),
]);
let cwd = std::env::current_dir()?;

let exit_code = deno_task_shell::execute(
  list,
  env_vars,
  &cwd,
  Default::default(), // custom commands
).await;

deno_task_shell's Issues

Share non-process stdin across multiple commands

stdin when not from the process' stdin needs to be shared across multiple commands. Ex. echo "abcdefghijklmnop" | (deno eval "const b = new Uint8Array(10);Deno.stdin.readSync(b);console.log(b)" && deno eval "const b = new Uint8Array(10);Deno.stdin.readSync(b);console.log(b)")

I'm not exactly sure how to do this, but will look into it later as it's an edge case that's not blocking.

Edit: Looks like https://crates.io/crates/os_pipe is already used in deno.

Support redirects

Support glob expansion

• globstar
• nullglob? Maybe yes? error on no matches

`pwd` only shows the physical path

Here I have /home/luca symlinked to /home/luca/test. pwd in deno task, not matter what argument I give it, will always show me the physical path (aka pwd -P). Currently there is no way to get the logical path with the pwd in deno task.

$ deno task pwd
Warning deno task is unstable and may drastically change in the future
Task pwd pwd --help
/home/luca
$ pwd
/home/luca/test
$ pwd -P
/home/luca

Re-enable stdin test after Deno 1.21.1 release

denoland/deno#14704

Request for Thread-Safe Methods in deno_task_shell.

Hi all,
I am currently utilising deno_task_shell in a multi-threaded Rust application and have encountered situations where the need for thread safety within shell task execution is critical. Specifically, methods like execute_with_pipes are currently not guaranteed to be thread-safe, which can lead to issues in concurrent execution scenarios.

The ability to safely execute shell commands across multiple threads (we're using tokio at present) would greatly enhance the utility and robustness of deno_task_shell, especially for applications dealing with high concurrency or parallel processing needs.

Examples/Use Cases:
Concurrent Command Execution: Running multiple shell commands in parallel where each command's input/output doesn't interfere with others.
Shared State Management: Safely managing shared state or resources between different shell commands or tasks being executed concurrently.
Suggested Enhancements:
Implementation of thread-safe primitives like Arc, Mutex, RwLock, or usage of atomic types for shared state management within the crate's internal structure.
Introduction of thread-safe variants of existing methods or entirely new methods designed for concurrent execution.

I believe that enhancing deno_task_shell with thread-safe capabilities would be greatly beneficial to the Rust community. I am looking forward to any discussion on this matter and am willing to assist in the implementation or testing phases if needed.

Thank you for considering this feature request.

Regards,

Ability to execute tasks by injecting Deno args

Task definition by injecting ${Deno.args[0]}:

{
  "tasks": {
    "test:watch": "deno test --watch -A --unstable ${Deno.args[0]}"
  }
}

and run task:

deno task test:watch MyDirectory

Support `|&` pipelines

This pipes stdout & stderr to stdin instead of only stdout to stdin.

Running task with argument will enclose it in quotes

deno task node:test test-http-upgrade-agent.js
Warning deno task is unstable and may drastically change in the future
Task node:test deno test --unstable --allow-all node/_tools/test.ts "test-http-upgrade-agent.js"
error: No such file or directory (os error 2)

Deno processes spawned using `deno task` do not exit when the parent process exits

Consider this example:

deno.jsonc:

{
  "tasks": {
    "script": "deno run script.ts",
  }
}

script.ts:

console.log(Deno.pid);
while (true);

reproduction.ts:

const BROKEN = ["deno", "task", "script"];
const WORKING = ["deno", "run", "script.ts"];

const parent = Deno.run({ cmd: BROKEN });
console.log(`Spawned ${parent.pid}`);
await sleep();

parent.close();

function sleep(): Promise<void> { return new Promise(resolve => { setTimeout(() => { resolve(); }, 1000); }); }

Here's the behavior using the BROKEN cmd in reproduction.ts:

> deno run --allow-all reproduction.ts 
Check file:///reproduction.ts
Spawned 15852
Warning deno task is unstable and may drastically change in the future
Task script deno run script.ts
2036

> taskkill /F /IM deno.exe             
SUCCESS: The process "deno.exe" with PID 15040 has been terminated.
SUCCESS: The process "deno.exe" with PID 2036 has been terminated.

Here's the same execution but this time using the WORKING cmd:

> deno run --allow-all reproduction.ts 
Check file:///reproduction.ts
Spawned 17712
17712

> taskkill /F /IM deno.exe             
ERROR: The process "deno.exe" not found.

deno 1.20.5 (release, x86_64-pc-windows-msvc)
v8 10.0.139.6
typescript 4.6.2

Feature request: `head` & `tail`

It would be useful if deno_task_shell builds in head and tail commands with option -n. They usually can be used for picking up the first or last line of a command output, e.g.:

git tag --sort=-version:refname | head -n 1

Can't load environment variable with space from .env file

Based on #30, I added this task in deno.json

{
  "tasks": {
    "echo": "export $(cat .env | xargs) && deno eval \"console.log(Deno.env.get('DATABASE_URL'))\""
  }
}

And my .env file is below:

DATABASE_URL="Provider=MSOLEDBSQL;Data Source=localhost:1433;Initial Catalog=AdventureWorks;User ID=sa;Password=password

When running deno task echo, the following output is produced:

% deno task echo
Task echo export $(cat .env | xargs) && deno eval "console.log(Deno.env.get('DATABASE_URL'))"
Provider=MSOLEDBSQL;Data

Note that the environment variable value is cut off after the first white space.

head -n, --lines don't support the leading '-'

Version

$ deno --version
deno 1.40.1 (release, x86_64-unknown-linux-gnu)
v8 12.1.285.6
typescript 5.3.3

Step to reproduce

deno.jsonc

{
  "tasks": {
    "test-head": "head -n -1 test.txt"
  }
}

test.txt

01
02
03
04
05

Except behavior

$ head -n -1 test.txt
01
02
03
04

Actual behavior

$ deno task test-head                   
Task test-head head -n -1 test.txt
head: invalid digit found in string

[feature request] Support xargs -I <replace_str>

The xargs command is super useful but is too constrained to use for invocations that depend on positional parameters. This feature request asks for the traditional -I/-i/--replace flags to be added.

Eg:

$ echo -e 'a\nb\nc'|xargs -I arg echo 1 arg 2
1 a 2
1 b 2
1 c 2

Unfortunately, -I always constrains the maximum number of lines processed at a time to 1 and approximates the following loop:

$ for i in a b c; do echo 1 $i 2; done
1 a 2
1 b 2
1 c 2

Although both looping and xargs -I provide similar functionality, I think they serve different uses. Simple loops are wanted for simple lists and xargs is wanted for pipes.

Use case

In the absence of this support, task users are at the mercy of the subcommands they call. For example, here's how you would have to do copy with the current version of xargs:

$ echo -e 'a\nb\nc'|xargs cp -t dist

However, many tools don't support non-positional arguments like the target parameter so this isn't an option. Even Deno's own cp command doesn't support -t so this also fails:

{
  "tasks": {
    "hello": "echo -e 'a\nb\nc'|xargs cp -t dist"
  }
}

$ deno task hello
Task hello echo -e 'a
b
c'|xargs cp -t dist
cp: unsupported flag: -t

References

• Briefly mentioned on the original xargs issue
• #78

Support cat and xargs?

I think we should consider supporting these commands cross platform. It would allow for someone to load env variables from a .env file easily:

export $(cat .env | xargs)

Ref: #24

https://github.com/denoland/deno_std/blob/896039ddb90ec2c62d078293fff31debf35e1d89/deno.json#L39

Env file cross-platform

How can I load environment variables from an .env file in a way that works cross-platform?

I have too many variables to add to the command line. I could load them manually from inside my app using dotenv, but I wouldn't like to add the --allow-read permission just for that, and also I'd like to keep my script as clean as possible, with as few deps as possible.

How can I reference the arguments?

Deno says that tasks accept task args:

> deno task --help
deno-task 
Run a task defined in the configuration file

  deno task build

USAGE:
    deno task [OPTIONS] [ARGS]

ARGS:
    <task>
            Task to be executed

    <task_args>...
            Additional arguments passed to the task

So I have a deno.json file like this:

{
  "tasks": {
    "example": "echo \"Hello $1!\""
  }
}

And am invoking it like this and seeing weird output:

> deno task example Justin
Warning deno task is unstable and may drastically change in the future
Task example echo "Hello $1!" "Justin"
Hello ! Justin

Why is the arg printed out in the wrong spot and how can I reference it if not by $1 ?

deno: command not found when running from git-for-windows shell

Contents of deno.json

{
    "tasks": {
        "version": "deno --version"
    }
}

I run the task by running deno task version and get

Warning deno task is unstable and may drastically change in the future
Task version deno --version
deno: command not found

Expected output

Warning deno task is unstable and may drastically change in the future
Task version deno --version
deno 1.20.3 (release, x86_64-pc-windows-msvc)
v8 10.0.139.6
typescript 4.6.2

To workaround this issue I can set the task to include deno.exe instead of deno, but then it stops working in powershell and non-windows platforms. Please let me know if you need more information to reproduce this issue 🙂

Watch in task commands

It would be helpful to have a watch command in deno task, e.g.:

{
  "task": {
      "build": "watch deno build-html.js"
  }
}

Not sure if I got the syntax right, but the idea would be the command would run whenever build-html.js changes.

Ability to execute files with shebang on Windows and older versions of Ubuntu?

For example, say you had:

{
  "tasks": {
    "my_task": "./my_script.js"
  }
}

And the script had:

#!/usr/bin/env -S deno run --unstable --allow-write --allow-read --allow-run
// ...etc...

Perhaps if the script has a shebang starting with #!/usr/bin/env -S then deno task should just figure this out. This apparently already works on Mac so making this work on all platforms would reduce some issues.

From @jespertheend in denoland/deno#14036

I like this method because it keeps the required permissions close to the actual file, and running the script is extremely simple because all you have to do is run ./tools/format.js.

But the downside is... it doesn't work very well on Windows. Maybe I'm missing a simple solution for this, but as far as I can tell /usr/bin/env is not really a thing on Windows, and shebang support isn't great either. So Windows users are stuck with having to run scripts using deno run while trying to figure out what the required permissions are (or running with -A).
This wasn't a huge problem for me though. I'm mostly developing on macOS, so I was using shebangs for now, though I did want to have better cross platform support eventually.

To implement this, I think we should attempt to parse the shebang. If we can, execute it the same way on all platforms. If we can't, we should try to run it as a command, but print a warning saying it won't work cross platform and to recommend fixing the shebang.

[feature request] Support looping over simple lists

Historically, looping has been pretty handy. Simple support is wanted to avoid lengthier JavaScripts:

$ for i in a b c; do echo 1 $i 2; done
1 a 2
1 b 2
1 c 2

The intended use case is simple lists and differs from xargs which is useful for mapping stdin to a command.

References

• Split from similar xargs -I issue

Support ?= short hand for setting variable if not already set

https://www.gnu.org/software/make/manual/html_node/Setting.html#Setting

This would make it easier to re-use deno tasks in other tasks that have different environment variables.

{
  "tasks": {
    "run": "export APP_ENV ?= production && deno run --allow-read=. --allow-write=. scripts/run.js",
    "run-dev": "export APP_ENV = development && deno task run"
  },
}

Currently that short hand won't work and will give you the following error.

$ deno task run
Task run export APP_ENV ?= production && deno run --allow-read=. --allow-write=. scripts/run.js
error: Error parsing script 'run'.

Caused by:
    Globs are currently not supported, but will be soon.
      ?= production && deno run --allow-read=. --allow-write=. scripts/run.js
      ~

Since the shorthand doesn't work currently, you have to either repeat or move the common part to a separate task. The downside to the second shorthand example below is that people might use run without APP_ENV set.

{
  "tasks": {
    "run": "export APP_ENV = production && deno run --allow-read=. --allow-write=. scripts/run.js",
    "run-dev": "export APP_ENV = development && deno  run --allow-read=. --allow-write=. scripts/run.js"
  },
}

{
  "tasks": {
    "run": "deno run --allow-read=. --allow-write=. scripts/run.js",
    "run-prod": "export APP_ENV = production && deno task run"
    "run-dev": "export APP_ENV = development && deno task run"
  },
}

I was told by @bartlomieju to ping @dsherret on this feature request.

Feature request: if / then / else support

For writing simple scripts it could be nice to have some bash-like if / then / else support.

Support suffix for sleep command

https://man7.org/linux/man-pages/man1/sleep.1.html

Failure to parse exit command args should return 2 exit code

Just found this comment in Dax (written by someone else, not me):

https://github.com/dsherret/dax/blob/c885ba58ba52035a6cf99629dbcac90209a94604/src/commands/exit.ts#L13

This line should be changed to 2 it seems:

deno_task_shell/src/shell/commands/exit.rs

Line 26 in 26ef383

ExecuteResult::Exit(1, Vec::new())

Deno task `cp` wildcard does not work correctly

Hey,

I want to recursively copy all .html files from ./src/client/ to ./dist/. For this I wanted to use the following task command:

cp -r ./src/client/**/*.html ./dist/

This command should normally (in my opinion) copy all files ending with .html recursively to ./dist/. However, only the first file ending with .html is copied to ./dist/.

Is this how it is meant to be?

Short summary

• Command used:

cp -r ./src/client/**/*.html ./dist/

• Expected behavior:
  All files ending with .html and located in .src/client/ are copied to ./dist/ with the same folder structure.

Deno Glob Expansion Reference

# match .ts files in the current and descendant directories
echo **/*.ts

• Actual behavior:
  Only the file which ends with .html and is in the first directory level will be copied to ./dist/.

Support `wait` command to wait for all async commands.

Support the wait keyword like bash&sh does to allow waiting for async commands to complete.
That way, you can run commands in parallel and wait for all to finish before the task quits.

Or does the task just not exit if not all of the async commands are completed? The the manual should maybe be updated?

Consider relying on uutils/coreutils

At the moment, deno_task_shell reimplements commands from coreutils.
Therefore, I can imagine, that it's relatively easy to hit a case where these commands behave unexpectedly.
An alternative would be to rely on uutils/coreutils instead.
They aim to be a drop-in replacement for the GNU utils. Nushell is already starting to port to uutils, you can read their story here: https://www.nushell.sh/blog/2023-09-05-why-uu.html.

Here is the PR where uutils/coreutil's cp command was added to nushell: nushell/nushell#10097

[QUESTION] Why not use Rust coreutils?

Sorry, if this comes across as a "talk is cheap" or this has been discussed before (didn't see it in the closed issue list).

But why re-create all the built-in commands, when they already exist as cross-platform Rust programs in https://docs.rs/crate/coreutils/0.0.13?

Would you be able to support every requests that has been here with cargo build --features "cp mv rm mkdir pwd sleep echo cat" --no-default-features (minus xargs and exit) and parse the deno task commands with Module oursh::program::posix?

Sorry, if you are having fun implemented these commands yet again, then you should absolutely continue the fun!

I am just curious to know if you are aware of existing Rust programs that does a similar thing and why you choose not to use them?

Text after quote is not parsed correctly

mkdir 'a/b'/c
# or even
mkdir 'a/b'c

Task test mkdir 'a/b'/c
error: Error parsing script 'test'.

Caused by:
    Unexpected character.
      /c
      ~

`deno task` - Support brace expansion

I have this in my deno.json

{
  "tasks": {
    "check": "deno check services/*/{prod,dev}.ts"
  }

When I run deno task check, I got:

Task check deno check services/*/{prod,dev}.ts
error: Error parsing script 'check'.

Caused by:
    Unexpected character.
      {prod,dev}.ts

My deno version:

deno 1.36.3 (release, aarch64-apple-darwin)
v8 11.6.189.12
typescript 5.1.6

Support `cp -r`

As title.

Expose `execute_with_pipes`?

Im working on integrating deno_task_shell into pixi. Its perfect for our use case so Im very happy this crate exists.

However, we also want to be able to create tests (similar to the TestBuilder in this crate) where we can capture the output of a shell command that is run. I couldn't figure out a way to do this because most of the code is pub(crate).

Would it be an option to expose execute_with_pipes and the required types so that we can consume the output of shell commands? I'm happy to make a PR.

Support redirection with pipeline

{
  "tasks": {
    "test": "echo 1 | sed -e 's/1/2/' > output.txt"
  }
}

This doesn't redirect the stdout of sed command to output.txt, which is unexpected from normal shell behavior:

$ deno task test
Task test echo 1 | sed -e 's/1/2/' > output.txt
2
$ cat output.txt
cat: output.txt: No such file or directory

$ echo 1 | sed -e 's/1/2/' > output.txt
$ cat output.txt
2

deno_task_shell doesn’t handle escaping of parentheses `(` and `)`

I was trying to use the dax module (which internally uses deno_task_shell) to call some ImageMagick commands but encountered an issue when the command contains escaped parentheses.

FYI: dax is a cross platform shell tools for Deno inspired by zx. dax uses deno_task_shell internally to parse commands.

Some usage of ImageMagick requires using parentheses (( and )) as command line arguments so they’re usually escaped, like in this example:

magick eye.gif \( storm.gif -negate \) +append cmd_bracket.gif

They have nothing to do with subshells, but dax (and thus deno_task_shell) fails to parse commands containing raw and escaped parentheses. The following Deno tasks also demonstrates this issue:

{
  "tasks": {
    "paren": "echo ( foo bar )",
    "paren2": "echo \\( foo bar \\)"
  }
}

$ deno task paren
Task paren echo ( foo bar )
error: Error parsing script 'paren'.

Caused by:
    Unexpected character.
      ( foo bar )
      ~

$ deno task paren2
Task paren2 echo \( foo bar \)
error: Error parsing script 'paren2'.

Caused by:
    Unexpected character.
      ( foo bar \)
      ~

The workaround I found is to enclose the parentheses with single- or double-quotes, like this:

{
  "tasks": {
    "paren": "echo '(' foo bar ')'"
  }
}

$ deno task paren
Task paren echo '(' foo bar ')'
( foo bar )

Question: Is it by design or a bug?

grep

I'd like the task shell to be able to use grep. In the olden days, using Node, shelljs provided grep and sed, very useful.
https://github.com/shelljs/shelljs