smol-rs / async-process

Async interface for working with processes

async-process

Async interface for working with processes.

This crate is an async version of std::process.

Implementation

A background thread named "async-process" is lazily created on first use, which waits for spawned child processes to exit and then calls the wait() syscall to clean up the "zombie" processes. This is unlike the process API in the standard library, where dropping a running Child leaks its resources.

This crate uses async-io for async I/O on Unix-like systems and blocking for async I/O on Windows.

Examples

Spawn a process and collect its output:

use async_process::Command;

let out = Command::new("echo").arg("hello").arg("world").output().await?;
assert_eq!(out.stdout, b"hello world\n");

Read the output line-by-line as it gets produced:

use async_process::{Command, Stdio};
use futures_lite::{io::BufReader, prelude::*};

let mut child = Command::new("find")
    .arg(".")
    .stdout(Stdio::piped())
    .spawn()?;

let mut lines = BufReader::new(child.stdout.take().unwrap()).lines();

while let Some(line) = lines.next().await {
    println!("{}", line?);
}

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

async-process's Issues

Issue calling `Child.output()` on ubuntu-latest with `v1.8.0`

Description

We encountered an unexpected problem when using async-process on the ubuntu-latest environment in our CI workflow after it was updated to version v1.8.0 on September 25th.

The issue manifests as erratic behaviour in our code, primarily blocking, when we make calls to Child.output(). We would appreciate any assistance in troubleshooting and resolving it. If you need any further information or logs, please let us know.

Associated Issue: near/near-workspaces-rs#312

reduce the `Reaper::zombies` from AtomicU64 to AtomicUsize or less

async-process/src/reaper/wait.rs

Lines 24 to 25 in 581c0a0

	/// Number of zombie processes.
	zombies: AtomicU64,

AtomicU64 is only available for target with cfg(target_has_atomic = "64") which does not exist on some less powerful 32-bit architecture (mips*, thumbv7*, armv5te, hexagon, powerpc, sparc, csky, riscv32*). This line makes this crate cannot compile.

I doubt if it is practical to support more than 4 billion zombies on 32-bit systems. Besides, the other atomic fields are AtomicUsize already.

async-process/src/lib.rs

Lines 105 to 114 in 581c0a0

	/// The number of tasks polling the SIGCHLD event.
	///
	/// If this is zero, the `async-process` thread must be spawned.
	drivers: AtomicUsize,
	/// Number of live `Child` instances currently running.
	///
	/// This is used to prevent the reaper thread from being spawned right as the program closes,
	/// when the reaper thread isn't needed. This represents the number of active processes.
	child_count: AtomicUsize,

Implement AsRawHandle on Child

Just did it for Tokio: tokio-rs/tokio#3987

While on Unix, having the PID (via Child::id()) is sufficient, on Windows only the handle returned by the process creation has full privileges over the process. As such, it's helpful to implement AsRawHandle for Child.

My immediate usecase is to build up a smol/async-process variant in https://github.com/watchexec/command-group, which supports an async refactor of Cargo Watch. I'm probably going to go with Tokio for that, but for completeness and for other projects which already use async-std I'd love to add an implementation. To do process groups on Windows I use JobObjects, and assign the process handle to the job.

I'm happy to make a PR for it if you approve :)

feature add: raw_arg

This is needed for some arguments to work correctly on some commands:

https://doc.rust-lang.org/nightly/std/os/windows/process/trait.CommandExt.html#tymethod.raw_arg

related to: rust-lang/rust#29494

use of unstable library feature 'process_set_argv0'

Any chance to support those of us building Rust applications for Linux distributions that haven't yet updated Rustc beyond 1.43.0 (Ubuntu)?

[Question] Timeout handling

Is there any way to interrupt a command once it reaches a timeout?
Something like what process_control offers?
Could I even just use something like future interrupt to achieve this?

Use a more efficient strategy for waiting on processes

Right now, we use SIGPROC to wait for process events on Unix and thread pooling to wait for process events on Windows. This is a very inefficient strategy that doesn't scale for many child processes, like using poll() for I/O notification.

Most operating systems have better ways of handling events like this.

• Linux has pidfd, which can be registered directly into async-io.
• BSD has EVFILT_PROC, which is now exposed in async-io.
• Windows has waitable handles, which need to be exposed in async-io. They can also be made more efficient on the polling side, see smol-rs/async-io#25

`ChildStdin`, et cetera should implement `AsRawFd/AsRawHandle`

I was going to try to implement I/O safety types on the handles as per sunfishcode/io-lifetimes#38, but then I realized that ChildStdin, ChildStdout and ChildStderr don't implement AsRawFd at all.

I believe that this is due to the fact that Unblock is used to wrap the inner std type, which makes it difficult to access the methods of the inner type. I see three possible ways to resolve this:

• On Windows, wrap ChildStd* in an Arc that is stored in the structure as well as the Unblock, so that methods on the inner type can be accessed more easily.
• Make it easier to access the inner file descriptor for Unblock types. We could potentially implement AsRawHandle on Unblock<T: AsRawHandle> that panics if the inner type is not accessible, or maybe some kind of get_ref(&self) -> Option<&T> general purpose method.
• Implement AsRawFd on unix, since it is easy to get the inner type via async_io::Async, but don't implement the corresponding types on Windows.

`Child::status().await` without closing stdin?

I'd like to be able to await on a running processes to finish without async-process closing its stdin. I saw that there is a try_await which doesn't close stdin, but that function isn't async.

Seal CommandExt

Both unix::CommandExt and windows::CommandExt are not intended to be implemented by users, and they should be sealed.

Note that this is a breaking change. (However, given the standard library added a new method in the minor version to os-specific extension traits (not sealed) (1, 2), we may not consider this is a breaking change that requires a major version bump.)

Related: smol-rs/async-fs#10

More ligthweight waiting for signals

I've noticed in #4 that the waiting for signals on unix is more heavy-weight than it probably deserves. The signal-hook::iterator::Signals is the most comfortable interface for application usage, as it allows conveniently blocking on an iterator and can handle multiple distinct signals at once, but it is also quite a bit of code that needs to be compiled (it can be turned off by a feature) and needs a separate thread for the blocking. It doesn't seem this crate would need any kind of multiple signals support and blocking is probably even an anti-feature.

I would send a pull request with doing something more lightweight, but I want to discuss design first.

• The easiest way would be to replace the Signals with pipe. This would allow removing the feature flag and cutting down on the amount of code being compiled.
• The next step would be to use some FD-thing (pipe/socket) that is async ready and starting a task instead of thread. But that would probably mean the code between unix and windows would diverge a bit further.
• Or, alternatively, is the wakeup mechanism inside smol async-signal-safe, or does it use mutexes or something like that? If it is safe, it would be possible to just notify from within the signal handler itself. That would need a bit of unsafe (promise that it's really async-signal-safe), but would allow going directly to signal-hook-registry and not creating further file descriptors.

What do you think would be the best way forward?

QUERY: Compiling on Rust 1.64.0

I apologise if this issue is being raised too far down the dependency tree...
I am currently unable to compile this crate under Rust 1.64.0 due to the following:

error[E0599]: no method named `as_fd` found for struct `Async` in the current scope
   --> /home/nheitz/.cargo/registry/src/github.com-1ecc6299db9ec823/async-process-1.5.0/src/lib.rs:514:16
    |
514 |         self.0.as_fd()
    |                ^^^^^ method not found in `Async<std::process::ChildStdin>`

Are there any workarounds for this, or should I back off to a previous version of Rust? This worked as of Rust 1.62.1.

new release?

Could we make a new release for async-* series crate?

`From<std::process::Command>` ignores pre-configured values for `stdin`/`stdout`/`stderr`

Hello! Thanks for the great library, I'm finding it very useful in my current project however I think I've recently run into a bug when spawning an async command converted from a std::process::Command. This should be a simplified reproducible test case highlighting the problem:

async fn main() {
    let mut command = std::process::Command::new("echo");
    command.arg("hello world")
        .stdout(std::process::Stdio::null());
    async_process::Command::from(command)
        .status()
        .await
        .unwrap();
}

When running the above, you should see that hello world is printed to the console, even though we've explicitly said that Stdout should be redirected to Stdio::null().

As far as I can see, this is due to:

1. the implementation of spawn() defaults the value of stdin to Stdio::inherit():

   async-process/src/lib.rs

   Lines 971 to 983 in e84c3fd

   	pub fn spawn(&mut self) -> io::Result<Child> {
   	if !self.stdin {
   	self.inner.stdin(Stdio::inherit());
   	}
   	if !self.stdout {
   	self.inner.stdout(Stdio::inherit());
   	}
   	if !self.stderr {
   	self.inner.stderr(Stdio::inherit());
   	}
   	Child::new(self)
   	}

2. the implementation of from() marks stdin as not having been explicitly set (ie. stdin: false), even though it might have been explicitly set on the inner command:

   async-process/src/lib.rs

   Lines 1040 to 1051 in e84c3fd

   	impl From<std::process::Command> for Command {
   	fn from(inner: std::process::Command) -> Self {
   	Self {
   	inner,
   	stdin: false,
   	stdout: false,
   	stderr: false,
   	reap_on_drop: true,
   	kill_on_drop: false,
   	}
   	}
   	}

The work-around is to set .stdin()/.stdout()/.stderr() after converting from a sync command to an async command, but obviously this is a bit of a gotcha and isn't immediately obvious that it's necessary.

Ideally the solution would be to set stdout: true in the implementation of From<std::process::Command> when it's already been explicitly set for inner, I'm not sure if there is a way to know whether that's the case? Alternatively, maybe this behaviour just needs to be explicitly documented?
Or perhaps: why do we even need to handle the defaulting in this library? Isn't that going to be handled by std::process::Command anyways? The documentation does state that "By default, stdin, stdout and stderr are inherited from the parent.", so perhaps there isn't any reason to do so in this library?

I'm happy to implement any changes / fixes (including just docs improvements), but would just need guidance as per the above on what you think the right solution might be. Thanks!

Cross compilation to Android fails with unresolved symbols

Configuration

async-process v2.2.2
Android NDK v26.3.11579264

Description

I'm encountering issues when cross-compiling a Rust library for Android. The compilation fails with unresolved symbols related to the async-process library. Specifically, the errors indicate that PidfdFlags and pidfd_open cannot be found in the process module:

error[E0433]: failed to resolve: could not find `PidfdFlags` in `process`
   --> <...>/async-process-2.2.2/src/reaper/wait.rs:151:30
    |
151 |                     process::PidfdFlags::empty()
    |                              ^^^^^^^^^^ could not find `PidfdFlags` in `process`

error[E0425]: cannot find function `pidfd_open` in module `process`
   --> <...>/async-process-2.2.2/src/reaper/wait.rs:149:38
    |
149 |                 let pidfd = process::pidfd_open(
    |                                      ^^^^^^^^^^ not found in `process`

I've examined the project's code and found that the problematic features are explicitly targeting Android:

async-process/src/reaper/wait.rs

Line 135 in a9e4b09

target_os = "android"

I'm wondering if there's any particular configuration that could resolve this issue.

How to reproduce

1. Install the Android NDK.
2. Install the Rust std for the Android target, e.g. aarch64-linux-android:

$ rustup add target aarch64-linux-android

3. Build the library for the Android target:

$ cargo build --target=aarch64-linux-android

How does this compare to Tokio's Command?

https://docs.rs/tokio/latest/tokio/process/struct.Command.html

Is there a reason to use this one over Tokio's command? I landed here due to someone else having issues when receiving SIGCHLD in their process, wondering if I should be worried about the same, or if there are any advantages to switching.

Wrong example for Command::output

The rustdoc for the output() method on Command gives an example for the spawn() method.