smol-rs / async-fs

Async filesystem primitives

async-fs

Async filesystem primitives.

This crate is an async version of std::fs.

Implementation

This crate uses blocking to offload blocking I/O onto a thread pool.

Examples

Create a new file and write some bytes to it:

use async_fs::File;
use futures_lite::io::AsyncWriteExt;

let mut file = File::create("a.txt").await?;
file.write_all(b"Hello, world!").await?;
file.flush().await?;

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

async-fs's Issues

File::from() with pipes not working?

I have faced an issue when doing async_fs::File::from(File::from_raw_fd(fd)), where fd is a pipe.

The issue is that in non-blocking mode I'm getting errors about broken pipe in child worker process that is trying to write to the pipe like uv_write(): broken pipe: https://users.rust-lang.org/t/piping-file-descriptors-beyond-std-io-out-err-to-child-process/48089

There doesn't seem to be such issue when using blocking mode with child threads.

Is this related to async-io's note that is?:

NOTE: Do not use this type with File, Stdin, Stdout, or Stderr because all of the supported operating systems have issues with them when put in non-blocking mode.

I've prepared reduced example that can be run with:

git clone https://github.com/nazar-pc/mediasoup.git
cd mediasoup
git checkout rust-reduced
npm i
cd rust
RUST_LOG=mediasoup=debug cargo run

The issue happens when I'm doing reads on async_fs::File. Same reads with blocking code works fine, but requires a separate thread.

Use blocking::Task instead of a boxed pinned future

async-fs/src/lib.rs

Line 285 in de5105b

Busy(future::Boxed<(std::fs::ReadDir, Option<io::Result<std::fs::DirEntry>>)>),

Should be a very easy saved heap allocation

Consider making File `Sync`

Hello,

I am trying to migrate my code from async_std to async_fs and I am encountering issues with File not being Sync in async_fs while it is in async_std.

This is a concern for usage with Tide, for instance, which only supports Send + Sync for setting a response body:

async {
    let file = async_fs::File::open("foobar").await.unwrap();
    let file = futures_lite::io::BufReader::new(file);
    let mut res = tide::Response::new(tide::StatusCode::Ok);
    res.set_body(tide::Body::from_reader(file, None));
}

Is there any chance to have File being Sync, please ? Otherwise what workaround would you recommend, please ?
Wrapping the whole into an Arc is a pain to implement in async world.

Thanks in advance !
(Do not hesitate to tell if you are tired will all the beginners issue I opened those days on your various projects...)
Cheers.

Conditionally use more efficient strategies for polling file I/O

For Windows there is file completion (for some operations, we'd have to fall back to blocking for others), for Linux there is io_uring and for BSD there is aio. It would be nice to use one of these strategies instead of the inefficient blocking code if they are available.

Web support

Would there be interest in adding support for Web through the File System API?

It would be pretty straightforward, as it's all natively async anyway and there would be no need to spawn any blocking threads. Probably the most interesting part is documentation and converting errors to io::Error.

All major browsers already support this, except Safari which doesn't support writing files in async, but only in sync.

Happy to make the PR of course!

Is File::from_raw_fd() support possible?

I'm writing a library that communicates with worker process using file descriptor.
So in order to send and receive messages I'm creating files with File::from_raw_fd().

My understanding of this topic is very limited at the moment, so hopefully what I wrote makes sense 🙂

Seal DirBuilderExt, DirEntryExt, and OpenOptionsExt

unix::DirBuilderExt, unix::DirEntryExt, unix::OpenOptionsExt, and windows::OpenOptionsExt are not intended to be implemented by users, and they should be sealed.

Note that this is a breaking change. (However, given the standard library added a new method in the minor version to os-specific extension traits (not sealed) (1, 2), we may not consider this is a breaking change that requires a major version bump.)

Related: smol-rs/async-process#6, f4cc6e5#commitcomment-42948266

Support OS-specific FileExt traits.

Unix FileExt and Windows FileExt. They are useful when using .read_at(buf, offset) instead of .seek(pos); .read(buf). Because it calls pread(2) instead of lseek(2) + read(2) which is more efficient.

not working on nightly

error[E0046]: not all trait items implemented, missing: `as_flags`
    --> /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/async-fs-1.3.0/src/lib.rs:1431:1
     |
1431 | impl std::os::unix::fs::OpenOptionsExt for OpenOptions {
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `as_flags` in implementation
     |
     = help: implement the missing item: `fn as_flags(&self) -> std::result::Result<i32, std::io::Error> { todo!() }`

Document that dropping a async_fs::File with buffered writes may not immediately close file on drop

I just found an unexpected use case that would be worthwhile documenting here, and in blocking. As best as I can tell, if a user does:

let mut file = async_fs::File::open("some-file").await?;
file.write_all(b"some bytes").await?;
drop(file)

The underlying std::fs::File is not necessarily dropped when we drop the async_fs::File. This can happen because of how blocking implements AsyncWrite. Whenever it starts out in the Idle state, it creates an async pipe, and submits the (read_pipe, real_file) to blocking's background thread pool. Any subsequent writes are written into the write_pipe that's eventually consumed by the background task. If there's a large number of outstanding writes, then it might take a decent amount of time before the underlying std::fs::File is actually dropped.

The fix for this is to explicitly a function that ultimately calls Unblock::poll_stop, like async_fs::File::close, or async_fs::File::poll_flush.

We ended up running into this in Fuchsia when downloading a large number of files. While we thought we had sufficient controls around the number of concurrently written files by controlling when we dropped async_fs::File, but some users were hitting the "too many opened files" error because by default OSX's ulimit -n is only 256.

Would it be possible to update the docs for async_fs::File to mention that users should run flush/sync_data/sync_all/close/etc if they write data, otherwise the underlying file might not get closed until some indeterminate point in the future?