oconnor663 / duct.rs

a Rust library for running child processes

duct.rs

Duct is a library for running child processes. Duct makes it easy to build pipelines and redirect IO like a shell. At the same time, Duct helps you write correct, portable code: whitespace is never significant, errors from child processes get reported by default, and a variety of gotchas, bugs, and platform inconsistencies are handled for you the Right Way™.

• Documentation
• Crate
• GitHub repo
• the same library, in Python

Examples

Run a command without capturing any output. Here "hi" is printed directly to the terminal:

use duct::cmd;
cmd!("echo", "hi").run()?;

Capture the standard output of a command. Here "hi" is returned as a String:

let stdout = cmd!("echo", "hi").read()?;
assert_eq!(stdout, "hi");

Capture the standard output of a pipeline:

let stdout = cmd!("echo", "hi").pipe(cmd!("sed", "s/i/o/")).read()?;
assert_eq!(stdout, "ho");

Merge standard error into standard output and read both incrementally:

use duct::cmd;
use std::io::prelude::*;
use std::io::BufReader;

let big_cmd = cmd!("bash", "-c", "echo out && echo err 1>&2");
let reader = big_cmd.stderr_to_stdout().reader()?;
let mut lines = BufReader::new(reader).lines();
assert_eq!(lines.next().unwrap()?, "out");
assert_eq!(lines.next().unwrap()?, "err");

Children that exit with a non-zero status return an error by default:

let result = cmd!("false").run();
assert!(result.is_err());
let result = cmd!("false").unchecked().run();
assert!(result.is_ok());

duct.rs's Issues

Test the dir and relative exe scenario

stop using Self in public APIs

It makes the docs ugly :p

add a Debug impl for Handle

Right now deriving Debug is blocked on a few inner types (JoinHandle and AtomicLazyCell). Though we also have the option of implementing Debug explicitly.

take a look at tokio-process

https://github.com/alexcrichton/tokio-process

Could we add pipe support to tokio?

remember to derive Debug

automatically wait when we kill?

The number of situations where you wouldn't want to immediately wait on the child are super limited:

• You have a huge number of running children, and you want to signal them all before waiting on any of them for performance reasons. (I'm not sure if this really matters with SIGKILL, but it could with SIGTERM, since children could handle that signal and take an arbitrarily long time.)
• You have children who might be blocked in the kernel mode and so unresponsive even to SIGKILL. (For example, reading an unresponsive FUSE filesystem.) You want to try to kill them without necessarily locking the parent if they don't exit immediately, because you have some other strategy to Really Kill Them if the first one doesn't work.

Are those super obscure situations really worth the relatively common mistake of forgetting to wait and leaking PIDs?

make *_file more generic

It should accept some kind of trait bound that lets you pass a PipeWriter from os_pipe, or a ChildStdin from the standard library. If the FromFile trait in os_pipe doesn't make it easy to do that, we should revise that trait. Maybe it should be called FromHandle?

Maybe stdin_file/stdout_file/stderr_file should really be called stdin_handle/stdout_handle/stderr_handle? There's some ambiguity around whether those methods are taking a handle (they are) or giving a handle (they aren't), but I think that ambiguity is there with the *_file names too.

capture_both() ?

Get the pid of a started command

Is there any way to retrieve the pid of a running command?

Stop overusing asref

In most cases we should be using Into.

an easy way to get the output reader of a pipeline, without supplying our own pipes?

Missing license file?

This is specially important for those who might want to use duct in commercial products. Can you please license duct? Thanks.

proper rust documentation

work around current_dir differences

See rust-lang/rust#37868

This is probably worth testing noting in the implementation notes. I think it will affect Python too.

look into dynamically detecting `waitid` support

user code in the middle of pipes?

Call a user-supplied function with input and output pipes all set up, running on its own thread, as part of a pipeline of real child processes?

The main reason not to do this is that most cases work just fine with read and input. The only time you'd need this fancy approach is if you have too much data to fit in RAM. Even then you could simulate it yourself with os_pipe + duct.

add some linting to CI

@TalhaKhatri fixed a bunch of Clippy warnings in #52. It would be nice to get something into Travis to make sure we don't regress these fixes over time.

One thing to think about: I get the impression that Clippy adds new lints all the time. That could mean that our code "breaks" without actually changing, just because Clippy started checking something new. Maybe the standard way to deal with this is to pin a specific version of Clippy? I haven't looked into it yet.

Windows support!

To make this work, I'll need to copy the pipe code from libstd: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/windows/pipe.rs

Here's a good example of an AppVeyor setup: https://github.com/tomaka/glutin/blob/master/appveyor.yml

Keep track of the commands (or files) that caused an error, for more informative messages

Inspired by http://julialang.org/blog/2013/04/put-this-in-your-pipe

maybe stop trying to accept &File?

We have to jump through a ton of hoops to be able to accept &File (in addition to just File) and pass its lifetime around. The resulting depends on the lifetime covariance rules and can be confusing as hell. Since files have the try_clone method which will probably always work (and which we rely on internally in any case), it might make sense to drop all our lifetime parameters for now. We can reconsider adding this stuff back when IntoCow either gets undeprecated or replaced with something better?

fix Path handling

We'll need to do the prepend-dot thing mentioned in the spec.

dealing with owned data

Options:

• Define some trait and implement it manually for every type we support in a give interface (Path, PathBuf, str, etc.). This sucks, because we can't take advantage of random types that might implement AsRef<Path> etc.
• Define both stdin() and stdin_owned(), stdout() and stdout_owned(), etc. Then have each of those be regular generic methods taking T: AsRef<...> and T: Into<...>.
• Define only the reference versions above, but implement ToOwned on the whole expression, which returns an equivalent expression with a 'static lifetime. This would simplify the interfaces, but it would require an extra copy in the owned case.

specify a wait timeout?

Change user for subprocess

Would it be possible to add the ability to change uid/gid before spawning a command?

Specifically I was thinking of adding a function with a similar semantics to unchecked, i.e. it will change the entire sub-expression to run as the specified user, so you can both do it easily for the entire expression and for specific parts.

I am not sure whether it would be possible to support windows, but for our use-case it would be fine if this was implemented as duct::unix::ExpressionExt.

If you are interested in the feature, I could try creating a PR.

some way to kill a running process?

Probably start will need to keep more data around. We might need another type of explicit tree to mirror the expression tree, which knows how to wait and kill its children, similar to Rust's stdlib.

must_use for returned structs

make the Handle variants public?

If we ever wanted to let the user inspect a running expression, and maybe signal just one side of it or something, this would be the way to do it.

env and full_env

Hmm, full_env kind of assumes we're being passed a map, which isn't ideal. Maybe go with the env_remove/env_clear model from Command?

Add expected Return code

E.g. I have a process that should return with exit 1 I would like to expect this and still want to get an Ok(String)

build test binaries properly

Same issue as oconnor663/os_pipe.rs#5.

consider getting rid of `sh` entirely?

https://www.reddit.com/r/rust/comments/618jvw/announcing_duct_a_child_process_library_thats/dfdn2m3/

theoretical support for asynchronous waiting

If callers are willing to handle SIGCHLD themselves, they should be able to ask duct whether a given expression has finished yet.

use the stdlib's ExitStatus

duct.rs/src/lib.rs

Line 208 in aac4dd3

Ok(try!(command.status()).code().unwrap()) // TODO: Handle signals.

Now that there are public constructors, we can use them: rust-lang/rust#32713

cwd

Automatically split the fist argument to `cmd!` macro?

Hi!

A common usability issue with constructing a command line is that each argument must be a separate string literal:

cmd!("git", "symbolic-ref", "--short", "HEAD")

I think a nice qol improvement would be to automatically split the first argument of cmd! on whitespace, so the same could be written more compactly as

cmd!("git symbolic-ref --short HEAD")

Note that this allows one to supply additional arguments just fine:

cmd!("git symbolic-ref --short", commit_hash)

Does this look like a useful feature to have?

Command with regex

Hi 🙂

First, thanks for your work, it's really useful!

I have a problem executing commands containing a regular expression...
I'd like to execute a command which looks like that:

ps cax | grep "myprocess$"

I tried different alternatives of:

let stdout = cmd!("ps", "cax")
    .pipe(cmd!("grep", "-e", "'myprocess$'"))
    .read()
    .unwrap();
println!("stdout: {:?}", stdout);

But couldn't make it work... I always get an error like the one below:

Custom { kind: Other, error: StringError("command [\"grep\", \"-e\", \"\\\'myprocess$\\\'\"] exited with code1") }

Did I miss something, or it's just not possible at the moment with duct?

Thank you!

document the interaction between `stdout` and `dir`

cmd!("echo", "foo").dir("bar").stdout("blub.txt").run().unwrap() creates a file blub.txt in the current directory instead of inside the bar directory.

Is this expected? I thought dir behaved like a std::env::set_current_directory before and after the call.

Either way, I think it should be at least documented.

should read's UTF8 conversion be lossy?

a Handle API

Current thought:

• wait(&self) -> Result<ExitStatus>
• kill(&self) -> Result<()>
• output(self) -> Result<Output>

And probably have start return just Handle for consistency instead of Result<Handle>.

Originally misfiled at oconnor663/shared_child.rs#5.

what's up with std::io::prelude on 1.13.0?

This commit 269a8e5 broke the build on 1.13.0, but not on 1.15.0. That's why the AppVeyor badge is currently red. What's up with that?

reenable stable builds after 1.12

We're waiting on ExitStatusExt::from_raw to stabilize.

use path join with cwd instead of canonicalize?

Though its not actually clear to me that the getcwd system call does anything different from canonicalize. Should check that out.

I broke coveralls uploading

Related to: huonw/travis-cargo#58

Setup and kill process group instead of just immediate child

See how watchexec does it: https://github.com/mattgreen/watchexec/blob/master/src/process.rs

The stdlib Process sends a kill syscall to the process: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/unix/process/process_unix.rs#L246

However, that doesn't kill all processes, especially with servers, see watchexec/cargo-watch#25.

I really don't want to implement process handling myself. That's what Duct does. So it would be really neat if Duct could handle it :)

consider environment case insensitivity on Windows

See oconnor663/duct.py#48 on the Python side.

On Python we have to uppercase env var names, because we have to match what the Python interpreter is giving us via os.environ.copy(). What's the situation in Rust? Do we need to deal with this?

use error_chain

should there be a way to quote paths in shell mode, for Windows?

Also, is it bad that relative path behavior with dir is inconsistent between cmd! and sh?

parent() returns Some("") for single-component relative paths

use std::path::Path;

fn main() {
    println!("{:?}", Path::new("/").parent());  // None
    println!("{:?}", Path::new(".").parent());  // Some("")
    println!("{:?}", Path::new("foo").parent());  // Some("")
}

The latter two cases feel weird to me. Some("") by itself is kind of a contradiction, on the one hand saying "yes there is a parent" and on the other hand returning an invalid path that really means "no there isn't actually a parent." We've also tried to avoid creating empty path components in other cases, like the double-slash case Path::new("a//b").parent().

For consistency with /, it probably makes sense to have the parent of . be None. For foo I could imagine going either of two ways, either Some(".") or None. If folks agree that one of those options would be nice in theory, then I guess the second question is whether a behavior change here would break backwards compatibility too much to consider doing it. Would it make sense for me to put together a PR and the ask for a Crater run or something like that?

unset without full_env?

You might want to express something like "the variable $FOO must not be set for this expression, but any other variables given as outer env calls are fine." That's not currently possible, because full_env will stomp on any outer env calls.

The usual solution will be "just fold those outer env calls into the map you're giving to full_env." However, there might be some rare cases where you don't control the code calling full_env? Unlikely, but if there was an API that satisfied all possible use cases that would be cool :)

support some way to control the executable name?

Maybe the right solution here is to make it possible to create a command expression from a Command you've instantiated yourself?