dsherret / dax Goto Github PK

• components(): Iterator<string>
• startsWith(text: string | PathRef | URL): boolean;
• endsWith(text: string): boolean;
• equals(path: string | URL | PathRef): boolean;
• linkTo / linkToSync methods (copy symlinkTo)

The methods names should be left to right least specific to most specific and prefer absolute symlinks:

• createAbsoluteSymlinkTo -> createSymlinkTo
• createRelativeSymlinkAt -> createSymlinkRelativeTo -- I'm not sure why this one is opposite with "At"`. I'll probably remember when I go to look at this issue, but I think it was just misnamed.

dax should implement a new feature that enables partial window scrolling in the terminal, similar to the behavior seen in the Docker build process.

The current behavior of dax when running scripts or applications that produce a large amount of output in the terminal makes it difficult to navigate through the output as it scrolls off the visible portion of the terminal window. The introduction of this feature would greatly enhance the developer experience by allowing users to easily scroll through and review the output without losing context.

This would require capturing output of a subcommand, limiting it to N lines, and performing a partial scroll of the terminal window when those N lines overflow.

An example, limited to 5 lines, might look like the below. Previous lines (output before the deno_napi line) would be hidden from view.

X running `cargo build`
   Compiling deno_napi v0.37.0 (/Users/matt/Documents/github/deno/deno/ext/napi)
   Compiling deno_url v0.107.0 (/Users/matt/Documents/github/deno/deno/ext/url)
   Compiling deno_ffi v0.94.0 (/Users/matt/Documents/github/deno/deno/ext/ffi)
   Compiling deno_net v0.99.0 (/Users/matt/Documents/github/deno/deno/ext/net)
   Compiling deno_fs v0.17.0 (/Users/matt/Documents/github/deno/deno/ext/fs)

Ran into a problem on a production pup instance running a web server using command deno task prod. Once started, the server cannot be killed programatically. Pup uses dax internally.

The problem is solved by starting the server using command deno run instead of deno task.

When aborting deno task prod using CTRL+C at the command line, everything works as it should.

Complete repro at https://github.com/Hexagon/dax-issue-146

Repro steps

Running deno run -A main.ts directly using dax works fine

# First start - all good, 
> deno run -A works.ts
Listening...
... aborted with exit code 124

# Second start - all good
> deno run -A works.ts
Listening...
... aborted with exit code 124

Running deno task prod using dax seem to leave the actual deno process hanging

# First start - all good, running deno task prod using dax
> deno run -A failure.ts
Listening...
... aborted with exit code 124

# Second start - the server from the first start is still listening on the port, even though everything see fine?
> deno run -A failure.ts
AddrInUse: Address already in use
... aborted with exit code 124

Code for repro

deno.json

{
    "tasks": {
        "prod": "deno run -A main.ts"
    }
}

main.ts

const server = Deno.listen({ port: 8083 });
console.log(`HTTP webserver running.  Access it at:  http://localhost:8080/`);

/* ... the rest of the example server imlementation */

works.ts

import $ from "https://deno.land/x/dax/mod.ts";

const child = $`deno run -A main.ts`.spawn();

// abort the child after 3s
await $.sleep("1s");
child.abort();

await child; // Error: Aborted with exit code: 124

failure.ts

import $ from "https://deno.land/x/dax/mod.ts";

const child = $`deno task prod`.spawn();

// abort the child after 3s
await $.sleep("1s");
child.abort();

await child; // Error: Aborted with exit code: 124

Maybe something like:

const data = await $.request("https://plugins.dprint.dev/info.json")
  .showProgress()
  .json()

Somewhat requires #47 in order to display the download sizes nicely.

It would be nice if instead of writing stuff like this:

try {
  return JSON.parse(Deno.readTextFileSync(this.#filePath)) as { ip: string };
} catch (err) {
  if (err instanceof Deno.errors.NotFound) {
    return undefined;
  } else {
    throw err;
  }
}

...if I could instead do:

const maybeData = $.path(this.#filePath).maybeJsonSync<{ name: string }>();
const data = $.path(this.#filePath).jsonSync<{ name: string }>();

I will work on this probably this week.

For example:

const example = "example";
await $`echo ${example}`.printCommand();

Outputs:

> echo example
example

Not sure about the name though. This is useful for debugging purposes especially. Probably the > should be white and the command text light grey.

I tested already the progress api and it works as expected

I tried to checkout the repo, but from some reason this test fails https://github.com/dsherret/dax/blob/main/mod.test.ts#L881

progress => ./mod.test.ts:881:6
error: AssertionError: Values are not equal:


    [Diff] Actual / Expected


    [
+     "Downloading Test",
    ]

  throw new AssertionError(message);
        ^
    at assertEquals (https://deno.land/[email protected]/testing/asserts.ts:190:9)
    at file:///home/mrcool/dev/deno/others/dax/mod.test.ts:886:3

I followed with a debugger and it seems for some reason the instantiation seems to fail here https://github.com/dsherret/dax/blob/main/src/console/utils.ts#L166 , so its weird I get the assertion error then, how did it reach there , maybe assertEquals has some internal try catch

I made sure I regenerated the wasm file with deno task wasmbuild, but same test error

Just an example that uses Promise.all for people who don't know about that in JavaScript.

Someone may be using this and not want to grant env var permissions (even though --allow-run is effectively --allow-all). It should still attempt to resolve commands in that scenario.

First, thank you for your work on Deno! I really enjoy the quality of what you and the team build 😄

I would normally have made something like this a Discussion topic instead of an issue, but I wonder if you've had thoughts on collaborating on https://github.com/c4spar/deno-dzx to add your ideas to that project? It is the most established Deno version of zx that I've seen (and contributed to) and I think it might be valuable to cross-pollinate ideas in that more established project!

Just wanted to put it on your radar in case you haven't already seen it - cheers!

When using the default stdout/stderr streams for a $ command, text written to the different streams in a particular order can appear out of order when printed.

Example

if I run deno run -A main.ts on the following file:

// main.ts
import $ from "https://deno.land/x/[email protected]/mod.ts";

await $`deno eval 'console.log("1: out"); console.error("2: err"); console.log("3: out"); console.log("4: out"); console.error("5: err");'`

the output printed to the terminal is:

1: out
3: out
2: err
5: err
4: out

By contrast, if I inherit the streams from the parent process as:

// main.ts
import $ from "https://deno.land/x/[email protected]/mod.ts";

await $`deno eval 'console.log("1: out"); console.error("2: err"); console.log("3: out"); console.log("4: out"); console.error("5: err");'`
  .stdout("inherit").stderr("inherit");

then the output to the terminal is:

1: out
2: err
3: out
4: out
5: err

Is this "out of order" output because the streams are buffered when not inherited?

It would be nice to have a selection and multi-selection api. I worked on it previously and have a branch locally, but it looks like getting arrow key presses is completely broken in Deno on Windows (denoland/deno#5945), so that will need to be fixed first.

We should remove the fs and path exports and replace it with functionality in PathRef.

In the following example:

import { $ } from "https://deno.land/x/[email protected]/mod.ts";

await $.request("https://example.com").noThrow().timeout(10).fetch();

unless example.com responds alarmingly fast, this code will throw an AbortError when the request is aborted thanks to the timeout(10).

This is the expected behavior of the abort controller, but in my opinion, is unexpected when the method you chain onto the RequestBuilder is called "noThrow".

In my opinion, either of the following would help things be more intuitive when noThrow is set and a timeout is triggered:

(a) rename noThrow to something else (ignoreHTTPStatus?) - this is a breaking change, but maybe with the module at v <1.0.0, that's ok?

(b) chain a catch onto the RequestBuilder.fetch method which simply returns undefined

This is just a suggestion, so as always feel free to close as won't fix if you feel that the API already makes sense for the majority of users, but wanted to surface the idea just in case!

Cheers, and thanks for considering :)

Generally when writing scripts, you don't need complex argument parsing and want something that can easily be figured out by looking at the code. You also don't need help text. For that reason, I think an approach like the following might be more appropriate than something more featureful like deno_std's argument parser.

$.args.on("prepare", () => {
  console.log("Preparing...");
});

$.args.on("build", () => {
  console.log("Building...");
});

> deno run -A script.ts prepare
Preparing...
> deno run -A script.ts build
Building
> deno run -A script.ts other
Unknown command: other

Possibilities:
  - prepare
  - build

More complex example:

const args = $.args(); // shorthand for `const args = $.args(Deno.args);`

await args.on("build", async (args) => {
  await args.on("data", (args) => {
    console.log(1);
    if (args.hasFlag("force") || args.hasFlag("f") || hasChanged()) {
     await buildData();
     console.log(2);
    }
  });
  
  // can be sync or async
  args.on("info", (args) => {
    console.log(3);
  }); 
});

args.on("version", (args) => {
  console.log(4);
});

// no match here causes a non-zero exit code that shows all the found options

> deno run -A script.ts build data
1
2
> deno run -A script.ts build data
1
> deno run -A script.ts build data --force
1
2
> deno run -A script.ts build data -f
1
2
> deno run -A script.ts build info
3
> deno run -A script.ts build other
Unknown command: build other

Possibilities:
  - build data
  - build info
> deno run -A script.ts version
4
> deno run -A script.ts invalid
Unknown command: invalid

Possibilities:
  - build
  - version

• People could call a function on args to prevent the "unknown command" error (maybe args.markHandled() or something).
• $.args.hasFlag("force") for --force and $.args.hasFlag("f") for -f
• $.args.get("time"): Arg | undefined -- always requires an equals sign
• $.args.getOrThrow("time"): Arg | undefined
  • $.args.getOrThrow("time").number()
• $.args.throwUncapturedFlags() - Throw for any flags that haven't been captured (maybe)

Don't know if its a dax or a deno issue, or if its the way things works
But It would be great if this can be fixed

code:

import { $ } from "https://deno.land/x/[email protected]/mod.ts";
await $`donotexistbinary`;

result

Uncaught Error: Command not found: donotexistbinary
    at resolveCommand (https://deno.land/x/[email protected]/src/shell.ts:773:11)
    at eventLoopTick (ext:core/01_core.js:182:11)
    at async executeCommandArgs (https://deno.land/x/[email protected]/src/shell.ts:5
71:27)
    at async executeSimpleCommand (https://deno.land/x/[email protected]/src/shell.ts
:560:10)
    at async executeSequentialList (https://deno.land/x/[email protected]/src/shell.t
s:386:20)
    at async spawn (https://deno.land/x/[email protected]/src/shell.ts:375:18)
    at async CommandChild.pipedStdoutBuffer (https://deno.land/x/[email protected]/sr
c/command.ts:572:20)

The stack doesn't reach to user code

I think that a case can be made in either direction for whether or not $.request should have a default timeout (as of now, it does not have one). Since the principle use case of dax is as a scripting tool, presumably it will frequently have uses in:

• CI / CD pipelines
• Background processing tasks

and other places where it can be reasonably expected that a human is not sitting there watching a script proceed (and able to notice that a request is hung due to any number of potential issues).

Due to the fact that the current implementation makes no distinction between the various parts of making a request (and I don't know if it even reasonably can while remaining a simple fetch under the hood) such as the time it takes to connect, to lookup DNS, to actually download the data, etc, I think it makes sense to have this default timeout be "generous" within the context of calling an API or downloading a webpage. I would suggest something like 60_000 (one minute) as the default.

I looked around at default http clients from different ecosystems (ruby, php, go, etc) and found that it's pretty common not to have a default timeout, so the precedent is definitely there to keep it as is, but in my opinion it makes sense to have a default for the role that dax fills.

Thanks for considering!

The ${completed}/${total} should have a way to be formatted as MiB, GiB, etc. Ideally the API would be very simple and opinionated (so no custom formatting)... maybe something like:

const pb = $.progress(`Downloading ${url}`)
  .kind("bytes") // this
  .length(byteCount);

import $ from "https://deno.land/x/[email protected]/mod.ts";
//
// run a command
await $`echo 5`; // outputs: 5

Requires

✅ Granted network access to "deno.land".
✅ Granted env access to all.
✅ Granted read access to <CWD>.

Are these really needed ? and if so maybe we can document them

Also do you happen to have a good way to figure out what's asking for permission, I think I'll try to deno run --inspect-brk later

Hello,

Thanks for a great tool.
I wanted to write neatly by chaining pipe() to command call.

I wrote the following code. This met my requirements.
What do you think?

import { CommandBuilder } from "https://deno.land/x/[email protected]/mod.ts";
CommandBuilder.prototype.pipe = function (next: CommandBuilder) {
  const p = this.stdout("piped").spawn();
  return next.stdin(p.stdout());
};

const ret = await $`echo "foo\nbar"`
  .pipe($`grep foo`)
  .text();
console.log(ret);

which it seems already is, you can check here we were discussion ideas on how to extend it
impactaky/dax_extras#2 (comment)

import $ from "https://deno.land/x/[email protected]/mod.ts";
import { addExtras } from "./vendor/raw.githubusercontent.com/impactaky/dax_extras/1.0.0/mod.ts";

addExtras($); // this will mutate it in place

the function idea seems to work, but we want to known the dax version so we can make a semver check on it,

do you think we can add a version prop to $ , so something like $.version = "0.32.0" / or current version

Thank you for your work on this project!

It would be excellent if there was a way to get the interleaved stdout and stderr as it occurred in realtime in the shell sub-process. This functionality is present in zx for example in the ProcessOutput#toString() method. It's also present in the execa Node module via the all property of the result object.

I use zx a lot for operational/orchestration shell scripting, but would love to switch over to dax since I love Deno and the principles of your project align better with my preferences, and having this capability would help make that transition smoother.

Cheers!

It's not good right now:

> const p = $.path(".").resolve()
undefined
> p
PathRef {}

Should probably be PathRef("..path goes here...") or something like that.

It might be neat to have an API that can be used to tell if some external changes have occurred since the last time a command was run. This would be useful for not doing an action if unnecessary to do so.

const tracker = $.changeTracker(import.meta, "data files"); // cache keyed on this current file and a string

tracker.addMtime("data/file1.json"); // hashes based on the file path's last modified time
tracker.addPath("data/file2.json"); // hashes based on the file path's content
tracker.addMtime("some_dir"); // hashes based on the descendants mtime
tracker.addPath("some_dir"); // hashes based on the descendants contents
tracker.addValue(123456); // hashes a specific value, which could have a source anywhere else

// multiple paths or values
tracker.addPaths(paths);
tracker.addMTimes(otherPaths);
tracker.addValues(values);

// will always run if the output path doesn't exist
tracker.addOutputPath("output/data.json");

// run if changed
if (tracker.hasChangedSync()) {
  await doStuff();
  tracker.commitSync();
}

// or as a single call
await tracker.runIfChanged(async () => {
  await doStuff();
});

// builder pattern
await $.changeTracker(import.meta, "data files")
  .addPaths(paths)
  .addOutputPath("output/data.json")
  .runIfChanged(async () => {
    await createOutputDataJsonFile();
  });

The hash could be saved in local storage.

It appears that resolveCommand fails to correctly resolve the location of commands that are available on the system path. This causes scripts to fail with the error thrown here.

The following simple example will crash:

import $ from "https://deno.land/x/[email protected]/mod.ts"
const winget = await $`winget --version`.text();
// console.log(await $.which("winget"));
// console.log(await $.commandExists("winget"));

The issue lies in deno_which.
Here is a failing test case for deno_which:

import {
  assertEquals,
  assertRejects,
} from "https://deno.land/[email protected]/testing/asserts.ts";
import { Environment, which, whichSync } from "./mod.ts";

const expectedWingetLocation = await getLocation("winget");

Deno.test("should get path", async () => {
  await runTest(async (which) => {
    const result = await which("winget");
    checkMatches(result, expectedWingetLocation);
  });
});

Deno.test("should get path when using exe on windows", {
  ignore: Deno.build.os !== "windows",
}, async () => {
  await runTest(async (which) => {
    const result = await which("winget");
    checkMatches(result, expectedWingetLocation);
  });
});

async function runTest(
  action: (
    whichFunction: (
      cmd: string,
      environment?: Environment,
    ) => Promise<string | undefined>,
  ) => Promise<void>,
) {
  await action(which);
  await action((cmd, environment) => {
    try {
      return Promise.resolve(whichSync(cmd, environment));
    } catch (err) {
      return Promise.reject(err);
    }
  });
}

function checkMatches(a: string | undefined, b: string | undefined) {
  if (Deno.build.os === "windows") {
    if (a != null) {
      a = a.toLowerCase();
    }
    if (b != null) {
      b = b.toLowerCase();
    }
  }
  assertEquals(a, b);
}

async function getLocation(command: string) {
  const cmd = Deno.build.os === "windows"
    ? ["cmd", "/c", "where", command]
    : ["which", command];
  const p = await Deno.run({
    cmd,
    stdout: "piped",
  });
  try {
    return new TextDecoder().decode(await p.output()).split(/\r?\n/)[0];
  } finally {
    p.close();
  }
}

Seems we're missing this really easy command to implement.

• pwd - Prints the name of
  the current/working directory.

For example:

import { build$, CommandBuilder, RequestBuilder } from "https://deno.land/x/dax/mod.ts";

const commandBuilder = new CommandBuilder()
  .cwd("./subDir")
  .env("HTTPS_PROXY", "some_value");
const requestBuilder = new RequestBuilder()
  .header("SOME_NAME", "some value");

const $ = build$({
  commandBuilder,
  requestBuilder,
  // this part is new
  extras: {
    sayHi() {
      console.log("Hi!");
    },
  },
});

$.sayHi(); // outputs "Hi!"

const new$ = $.build$({
  // this part is new
  extras: {
    sayBye() {
      console.log("Bye!");
    },
  },
});

new$.sayHi(); // outputs "Hi!"
new$.sayBye(); // outputs "Bye!"

This would just not consult the current environment (so no permission prompt) when creating the command context’s env vars.

Maybe it could be another override to request

the advantage it would skip loading the file into memory while keeping the cool progress bar

Currently it will override any existing file, maybe having an option for that is better

I have found the .logIndent() API to be odd/confusing. Having to wrap everything in some sort of async handler makes it very difficult to write straight forward imperative code. I personally like the console.group() model, where there is a set state which gets indented and outdented in the order of execution.

There are certain configuration settings that are useful to be able to mutate on an existing $ rather than constructing a new one. This mutable configuration should never be anything that could potentially break other code, but for logging purposes being able to modify if a command is printed to the console seems useful.

So instead of:

import { build$, CommandBuilder } from "...";

const $ = build$({
  commandBuilder: new CommandBuilder().printCommand(),
});

Users could do:

import { $ } from "...";

$.setPrintCommand(true);

It should be easy to get a writable stream of a file from a PathRef in a non-error prone way.

It should be possible to do something like:

const child = $`echo 1 && sleep 10 && echo 2`.stdout("piped").spawn();
await $`deno eval 'await Deno.stdin.readable.pipeTo(Deno.stdout.writable);'`.stdin(child.stdout());

I think probably the return value of spawn() should instead be a special kind of Promise that has methods like .stdout() and .stderr() on it.

I think it's ok that this is a little verbose since this isn't a common scenario.

IIRC, I think right now it downloads on every run? Not good. (maybe this just requires updating wasmbuild)

Is there any way to make the following code execute successfully

Example 1

import $ from "https://deno.land/x/dax/mod.ts";

const a = "12345"
const b = "23456"

await $`diff <(echo ${a}) <(echo ${b})`

Output

error: Uncaught "Unexpected character.\n  <(echo 12345) <(echo 23456)\n  ~"

Example 2

import $ from "https://deno.land/x/dax/mod.ts";

const a = "12345"
const b = "23456"

const tempFileA = await Deno.makeTempFile();
await Deno.writeTextFile(tempFileA, a);

const tempFileB = await Deno.makeTempFile();
await Deno.writeTextFile(tempFileB, b);

await $`diff ${tempFileA} ${tempFileB}`

Output

1c1
< 12345
\ No newline at end of file
---
> 23456
\ No newline at end of file
error: Uncaught Error: Exited with code: 1
          throw new Error(`Exited with code: ${code}`);
                ^
    at CommandChild.pipedStdoutBuffer (https://deno.land/x/[email protected]/src/command.ts:587:17)
    at eventLoopTick (ext:core/01_core.js:181:11)

Env

[email protected]
deno 1.33.2

I think the permission prompting could be a little better and explanatory.

Edit: investigated all permission prompts...

1. Once deno supports Wasm modules then we can get rid of needing to save and read the cache directory.
2. Calling Deno.cwd() is necessary when the shell is created and cannot be lazily evaluated since it could change while the shell is executing, which would lead to very unexpected behaviour. This can be bypassed by providing an explicit cwd option.
3. Getting env access to all when the shell is initialized is necessary in order to get the environment state when the shell is spawned (can't be lazily evaluated, which is the same issue as the last point). This could be mitigated by supporting clearEnv() in the future.

In my current project, we have a few instances where we are using pipes in order to feed data to certain commands, e.g. echo something | kubectl apply -f - for applying kubernetes objects. As PipeSequence is currently unsupported, I have tried converting them over to use the .stdin() method on the $ helper. However I have noticed when doing so that my processes seem to hang -- I can see the subprocess fire up, but it never exits.

In further debugging the issue I was able to determine that the stdin stream was being written out OK, however, it seems that the command consuming the stream (kubectl in my case) was waiting for some sort of flush operation or EOF. e.g.:

const someYaml = 'pretend this is real';
console.log({ someYaml });
const result = await $`kubectl apply -f -`.stdin(someYaml).text();
console.log(result);

results in:

{ someYaml: "pretend this is real" }

and kubectl will just sit there. However, if I quickly hack at executeCommandArgs and move the stdin.close() for the subprocess from the finally block into the actual writeStdin function, so that the stream is closed once the content has been written, kubectl completes successfully:

diff --git a/src/shell.ts b/src/shell.ts
index 6e4794b..1af3a02 100644
--- a/src/shell.ts
+++ b/src/shell.ts
@@ -560,7 +560,6 @@ async function executeCommandArgs(commandArgs: string[], context: Context) {
       completeController.abort();
       context.signal.removeEventListener("abort", abortListener);
       p.close();
-      p.stdin?.close();
       p.stdout?.close();
       p.stderr?.close();
     }
@@ -571,6 +570,7 @@ async function executeCommandArgs(commandArgs: string[], context: Context) {
       return;
     }
     await pipeReaderToWriter(stdin, p.stdin!, signal);
+    p.stdin?.close();
   }

   async function readStdOutOrErr(reader: Deno.Reader | null, writer: ShellPipeWriter) {

results in:

{ someYaml: "pretend this is real" }
error: error validating "STDIN": error validating data: invalid object to validate; if you
choose to ignore these errors, turn validation off with --validate=false
{ result: "" }

Which is more in line with what I would expect to see -- kubectl wonking via stderr in this case, or successfully completing if I were feeding it real junk.

I can submit a PR for the above change easily enough (tests will pass with the change), but I wanted to double check first to make sure I wasn't missing something obvious with how to use this. Its been a long week...

Thanks!

I didn't run into a problem yet, but just reading the code, it seems like wasm caching just checks if the file exists locally

So if the wasm file gets new apis, it wont be re-downloaded again because it exists already

Maybe the wasm caching need to be aware of dax version

It would be nice to be able to spawn an executable or script with shebang at a PathRef like so:

const ciScript = $.path("./.github/workflows/ci.generate.ts");
await ciScript.run();

I have noticed in long running processes, I would like to give a "progress" message... both a determinate and indeterminate variety:

• Progress bar, where the percent complete is known
• Activity/progress/spinner, where the completion endpoint is known, but would like to provide a spinner to indicate activity.

Right now there’s no easy way to provide an array of arguments.

This should work:

const args = [“arg1”, “arg 2”];
await $`command ${args}`;

Hello, thanks for this project

I mostly use this in repl so I end up writing a lot of await $c1 await $c2, I feel like those awaits are really unneeded and verbose. It would be nice if ax supported a sync api using Deno.spawnSync

Hello and thanks for the super useful project!

I was fiddling around with the typings for the CommandBuilder and registerCommand/registerCommands in order to support auto completion for custom commands that have been added and was wondering if you would be interested in adding this feature in. It does add a bit of complexity to the CommandBuilder types but I think it can be pretty useful in the long run.

Here is a quick demo of how it works:

And here is the commit on my fork (still needs type tests and I'm sure I need to add the LiteralUnion in a few more spots): curtislarson@cd24df6

Let me know your thoughts!

It's getting more complicated. It could use some tests that works with a local server.

Edit: Started to add this in #55

It would be really handy if we could extend the shell command parser out with support for additional, user-defined commands. For my current use case, for example, it'd be great to be able to provide Deno-based drop-ins for things like gzip, tar and so on, but you probably don't want every possible command under the sun within dax itself...?

I know our vet has complained about my cats getting a little too fat, anyway...

Similar to $.request: https://deno.land/x/[email protected]/src/request.ts?source#L135