Command parsing about zx

[Minigugus]

At first, maybe just a pipe function is enough (#14 (comment)):

let numberOfLinesContainingFooInAllFilesInDir = +(await pipe(
  $`find ${dir} -type f -print0`,
  $`xargs -0 grep foo`,
  $`wc -l`
))

would become something equivalent to:

let numberOfLinesContainingFooInAllFilesInDir = +(await readAll(
  pipe(
    spawn('find', [dir, '-type', 'f', '-print0']),
    spawn('xargs', ['-0', 'grep', 'foo']),
    spawn('wc', ['-l']),
  )
))

The pipe($`a`, $`b`) is pretty clear and allows for native Node stream to be piped too, without the need to parse the command line, and therefore doesn't break how | works currently.

[antonmedv]

zx is a command runner, not a cross-platform command library.

This is exactly true. zx doesn’t provide cross-platform support.

@Minigugus yes, I also like just providing pipe func idea. Let's take a look into it.

[antonmedv]

After thinking this is out of scope for zx. If follow this way we can end up with something odd like @yarnpkg/shell (not POSIXcompatible, different odd language with something implemented, and something not).
zx is just a wrapper around other shells (bash, etc)

[catpea]

Here is what parsing a command (not a chain, sorry) with peg.js would entail, this does not support pipes and redirects.

There used to exist a gist that handled that as well but google.com returns nothing now, even so the biggest hurdle with peg is handling quote escapes and this snippet covers that.

The information bellow only shows the command being parsed, and the result, I didn't include the peg grammar on this page as it is somewhat lengthy, see link below.

Command Line Parsing Grammar (no pipes)
https://gist.github.com/notheotherben/819ad3a3ada4a05e6fcbd9fcb27a992f

Example of a non-simple command we are going to parse:

ENV_X=true ENV_Y="yes please" ./test/my_exec arg1 -f1 "arg with spaces" 'another arg' --flag2 yet\ another\ arg --flag=10

And the resulting JavaScript Object

{
   "env": [
      "ENV_X=true",
      "ENV_Y=yes please"
   ],
   "exec": "./test/my_exec",
   "args": [
      "arg1",
      "-f1",
      "arg with spaces",
      "another arg",
      "--flag2",
      "yet another arg",
      "--flag=10"
   ]
}

You can try it out online (paste the https://gist.github.com/notheotherben/819ad3a3ada4a05e6fcbd9fcb27a992f command.peg code into "1 Write your PEG.js grammar" box) at https://pegjs.org/online

Please note that peg is not the only JavaScript parser generator, this is just a simple example, to show how simple this task is.

Maybe you could create a standalone library for parsing command line that people can use for other experiments.

(Just to show how wildly useful parsing a command-line into a JavaScript object is. For example, one could create a web terminal that accepts standard UNIX like command line, parses it, and then just emits an EventEmitter with a standard options object. You could build a terminal that taps into a EventEmitter based SPI in 5 lines of code or less.)

Thank you for being brave and opening this issue.

[RichiCoder1]

It was mentioned on reddit, but an alternative might be using https://github.com/yarnpkg/berry/tree/master/packages/yarnpkg-shell which would also have the side benefit of windows compat, thought at the cost of potentially having edge cases if not well documented.

[RichiCoder1]

Ah! And someone pointed out the other option of using ShellJS #11

[catpea]

In my opinion ShellJS and yarnpkg-shell add too much to zx, which should revolve around command parsing.

ShellJS and yarnpkg-shellrepresent a potentail library of commands, that could be installed in addition to zx, and this library would only do one thing. GIve windows grep and cat and others, so it would be a cross platform compatibility feature, that is nice but not what zx does.

In a bit more detail,

ShellJS and yarnpkg-shell can't be used as is, their code probably needs to be pulled out and a new library called zx-cross-platform-compatibility-layer can be plugged into zx if a person is unfortunate enough to be stuck without cat and ls, but still wants to run unix like commands on whatever they are at.

ShellJS and yarnpkg-shell will run, wherever node is installed, so they are cross-platform.

But at the same time,
I believe that zx should focus on parsing command line, and just spawn('command', [...argv])

That someone does not have cat and ls, and can't run unix commands on windows, is not really a problem that zx should be trying to solve.

Something tells me that the real number of commands is a pretty big number, Bash has 40 and BusyBox something like 310.

I think zx should not attempt to offer cross platform commands, but rather focus on executing whatever commands exist on a system - well.

zx has cd, fetch, question, chalk and that becomes confusing, making it look like zx deals in providing commands. zx is a command runner, not a cross-platform command library. It could offer one as a zx-cross-platform-compatibility-layer but that would be a lot of work.

pwd() and chdir() are special in node, they affect the path that you are in, so having those two (or just cd) is kind of neat.

fetch, question, chalk is sugar these items exist as a kind of a built in, becasue of they are witin the top 5 things that a script writer would need.

It would be more clear if xz was presented this way first:

#!/usr/bin/env node

import {$} from 'zx'
await $`date`

The $ represents a smart way of executing commands. And if you want chalk then just import {$, chalk} from 'zx' it is just a single word it does not need to come as a global (but it would be packaged within zx so that you don't need to install anything and make a package.json and all that.)

cd or pwd() and chdir() should just come from import {pwd, chdir} from process as well.

And what $ really means to all of us, is that we can just say:

await $`cat package.json | grep name`

instead of:

const { exec } = require('child_process');
exec('cat package.json | grep name', (error, stdout, stderr) => {
  if (error) {
    console.error(`exec error: ${error}`);
    return;
  }
  console.log(`stdout: ${stdout}`);
  console.error(`stderr: ${stderr}`);
});

It is a cute little idea, with a cute dollar sign, representing the shell prompt.

Moreover, if we do #35, we won't be using exec, which is memory intensive as it buffers everything up,

But Spawn, which will allow for spawned_grep.stdin.write(data);

So cat package.json would stream data by means of node streams into grep name.

So once we have Command parsing, a whole new world opens to us, now we can check if grep is available and if not then use the zx-cross-platform-compatibility-layer based on https://github.com/yarnpkg/berry/tree/master/packages/yarnpkg-shell to plug it in, provided that zx was installed with the optional compatibility layer in the first place.

I am not a developer here, I do not know who Anton is,
so all of this is my personal opinion.

$`` is a very good idea, but keep the built in commands far away, because there is probably 300 of those and that part will never be finished.

Focus on Command parsing and spawned_grep.stdin.write(data); where data comes from spawned_cat.stdout.on("data") that is to say:

$ will happily convert this:

await $`cat package.json | grep name`

into the pretty confusing (certainly as commands grow larger) this:

//snip

spawned_cat.stdout.on("data", data => {
    spawned_grep.stdin.write(data)
});

//snip

Again this is just my private opinion, I do not work here, I only make the mess!