danburkert / fs2-rs

Extended utilities for working with files and filesystems in Rust

fs2

Extended utilities for working with files and filesystems in Rust. fs2 requires Rust stable 1.8 or greater.

Features

• file descriptor duplication.
• file locks.
• file (pre)allocation.
• file allocation information.
• filesystem space usage information.

Platforms

fs2 should work on any platform supported by libc.

fs2 is continuously tested on:

• x86_64-unknown-linux-gnu (Linux)
• i686-unknown-linux-gnu
• x86_64-apple-darwin (OSX)
• i686-apple-darwin
• x86_64-pc-windows-msvc (Windows)
• i686-pc-windows-msvc
• x86_64-pc-windows-gnu
• i686-pc-windows-gnu

Benchmarks

Simple benchmarks are provided for the methods provided. Many of these benchmarks use files in a temporary directory. On many modern Linux distros the default temporary directory, /tmp, is mounted on a tempfs filesystem, which will have different performance characteristics than a disk-backed filesystem. The temporary directory is configurable at runtime through the environment (see env::temp_dir).

License

fs2 is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2015 Dan Burkert.

fs2-rs's Issues

Missing posix_fallocate in uClibc Linux environment

Hi! Recently, I stumbled upon this problem:

error[E0425]: cannot find function `posix_fallocate` in crate `libc`
   --> /home/operutka/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/unix.rs:103:30
    |
103 |     let ret = unsafe { libc::posix_fallocate(file.as_raw_fd(), 0, len as libc::off_t) };
    |                              ^^^^^^^^^^^^^^^ help: a constant with a similar name exists: `SYS_fallocate`
    | 
   ::: /home/operutka/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.71/src/unix/uclibc/arm/mod.rs:924:1

When trying to build my project for a uClibc based Linux environment. Apparently, uClibc does not have the posix_fallocate function. Is there any easy way around this?

Build fails because of mismatched types for posix_fallocate.

I just did a multirust update and cargo clean && cargo build.

Rustc version -

ico@arch ~ $ rustc --version
rustc 1.8.0-nightly (57c357d89 2016-02-16)

with the error message

Compiling fs2 v0.2.2
/home/ico/.multirust/toolchains/nightly/cargo/registry/src/github.com-88ac128001ac3a9a/fs2-0.2.2/src/unix.rs:63:67: 63:79 error: mismatched types:
 expected `i64`,
    found `u64` [E0308]
/home/ico/.multirust/toolchains/nightly/cargo/registry/src/github.com-88ac128001ac3a9a/fs2-0.2.2/src/unix.rs:63     let ret = unsafe { libc::posix_fallocate(file.as_raw_fd(), 0, len as off_t) };
                                                                                                                                                                                  ^~~~~~~~~~~~
/home/ico/.multirust/toolchains/nightly/cargo/registry/src/github.com-88ac128001ac3a9a/fs2-0.2.2/src/unix.rs:63:67: 63:79 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error

I checked out libc's source but the type looks to be in order.

Apple testing

Hello, your README.md says that continuous testing is done on :

• x86_64-apple-darwin (OSX)
• i686-apple-darwin

But it seems that you do not have automated build&testing for those two platforms. (It seems that a few more are missing looking at your latest results.

Am I just not looking in the right place ?

The reason I'm asking is because my CI/CD failed tests on OSX and it seems to be caused by fs2's file locking not working properly on OSX

FileExt::duplicate is the same as File::try_clone

Just a heads up 😃

Windows corner case: "access denied" when opened in append mode

Consider the following test program:

extern crate fs2;

use fs2::FileExt;
use std::fs;

fn main() {
    let f = fs::OpenOptions::new()
            .create(true)
            .append(true)
            .open("myfile.txt").expect("error creating");
    f.lock_exclusive().expect("error locking");
}

On Linux, it works fine. On Windows 10, it fails:

    Finished dev [unoptimized + debuginfo] target(s) in 0.99s                                                           
     Running `target\debug\examples\debug.exe`                                                                          
thread 'main' panicked at 'error locking: Os { code: 5, kind: PermissionDenied, message: "Access is denied." }', libcore
\result.rs:945:5                                                                                                        
note: Run with `RUST_BACKTRACE=1` for a backtrace.                                                                      
error: process didn't exit successfully: `target\debug\examples\debug.exe` (exit code: 101)   

If I use .write(true) instead of .append(true), it works fine. If I add .read(true), it also work. This seems to be because LockFileEx requires either the GENERIC_READ or GENERIC_WRITE permission on the underlying handle, while in std::fs on Windows activating .append(true) means that the underlying handle doesn't have the full set of GENERIC_WRITE bits.

This basically seems to be how the Windows API works, so I don't think fs2 can make the surprising behavior go away, but I think it could be helpful to have this described somewhere in the fs2 API docs. I am happy to write the words — do you have any suggestions as to where the best place to put some relevant language would be?

Ref: rust-lang #54118.

Building fs2 on UNIX

Hi,

how do I compile this Crate without pulling windows dependencies?

When I do cargo build --release I get:

error: no matching package named `kernel32-sys` found (required by `fs2`)

Well duh - I'm building it on unix - so I don't have kernel32-sys. I checked some Rust and Cargo docs and seems that it would require a [features] section in Cargo.toml for the target to be even selectable - is that true?

Right now it contains these target definitions:

[target.'cfg(unix)'.dependencies]
libc = "0.2.2"

[target.'cfg(windows)'.dependencies]
winapi = "0.2"
kernel32-sys = "0.2"

Switch to using raw fd/handle?

Is it possible to switch the API to use raw fd/handle instead so that it could be more trivially compatible with async_std::fs::File?

Compiling fs2 in AIX fails

Compiling fs2 in AIX fails with below error.,
rustc --crate-name fs2 /.cargo/registry/src/index.crates.io-d11c229612889eed/fs2-0.4.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=85c2f526c0211be9 -C extra-filename=-85c2f526c0211be9 --out-dir /tmp/pip-install-2doeq8db/tokenizers_a6dcd574fc864cb793f0985f2a16d7f5/target/release/deps -C strip=debuginfo -L dependency=/tmp/pip-install-2doeq8db/tokenizers_a6dcd574fc864cb793f0985f2a16d7f5/target/release/deps --extern libc=/tmp/pip-install-2doeq8db/tokenizers_a6dcd574fc864cb793f0985f2a16d7f5/target/release/deps/liblibc-3a5102201a3aa852.rmeta --cap-lints allow -C link-arg=-bbigtoc -L/opt/freeware/lib -D_ALL_SOURCE -lpython3.9

error[E0425]: cannot find function `allocate` in module `sys`
   --> /.cargo/registry/src/index.crates.io-d11c229612889eed/fs2-0.4.3/src/lib.rs:104:14
    |
104 |         sys::allocate(self, len)
    |              ^^^^^^^^ not found in `sys`
    |
note: found an item that was configured out
   --> /.cargo/registry/src/index.crates.io-d11c229612889eed/fs2-0.4.3/src/unix.rs:102:8
    |
102 | pub fn allocate(file: &File, len: u64) -> Result<()> {
    |        ^^^^^^^^
note: found an item that was configured out
   --> /.cargo/registry/src/index.crates.io-d11c229612889eed/fs2-0.4.3/src/unix.rs:108:8
    |
108 | pub fn allocate(file: &File, len: u64) -> Result<()> {
    |        ^^^^^^^^
note: found an item that was configured out
   --> /.cargo/registry/src/index.crates.io-d11c229612889eed/fs2-0.4.3/src/unix.rs:143:8
    |
143 | pub fn allocate(file: &File, len: u64) -> Result<()> {
    |        ^^^^^^^^

Locking on windows is not advisory

I wish I had more details offhand, but I'm just recalling some unexpected behavior I encountered recently.

I took an exclusive lock on a file on windows, and then subsequent I/O operations (I don't remember if they were reads or writes) started erroring.

Again, I'm sorry I don't have details offhand. It's late here.

posix_fallocate fails on ZFS+FreeBSD

This error never shows up on Linux because glibc tries to emulate fallocate. However, it seems this call could be emulated with truncate on FreeBSD.

fcntl instead of flock

I was looking into using this module but need for my locks on unix the properties of fcntl instead of flock. So I was wondering if you would accept a PR that adds an option to switch to fcntl.

Document error.

Trait FileExt method try_lock_exclusive has error.

error: could not compile `fs2`.

Hi,
Just trying to install another Rust crate called cargo-wasi (which uses fs2) and am receiving the following error.
Any thoughts?

Thank you!

   Compiling fs2 v0.4.3
error[E0433]: failed to resolve: use of undeclared type or module `sys`
  --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:98:9
   |
98 |         sys::duplicate(self)
   |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:101:9
    |
101 |         sys::allocated_size(self)
    |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:104:9
    |
104 |         sys::allocate(self, len)
    |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:107:9
    |
107 |         sys::lock_shared(self)
    |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:110:9
    |
110 |         sys::lock_exclusive(self)
    |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:113:9
    |
113 |         sys::try_lock_shared(self)
    |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:116:9
    |
116 |         sys::try_lock_exclusive(self)
    |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:119:9
    |
119 |         sys::unlock(self)
    |         ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:126:5
    |
126 |     sys::lock_error()
    |     ^^^ use of undeclared type or module `sys`

error[E0433]: failed to resolve: use of undeclared type or module `sys`
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:169:5
    |
169 |     sys::statvfs(path.as_ref())
    |     ^^^ use of undeclared type or module `sys`

error: aborting due to 10 previous errors

For more information about this error, try `rustc --explain E0433`.
error: could not compile `fs2`.

Type mismatch on latest nightly, compiles on stable.

https://github.com/danburkert/fs2-rs/blob/master/src/unix.rs#L62

~/.multirust/toolchains/nightly/cargo/registry/src/github.com-88ac128001ac3a9a/fs2-0.2.2/src/unix.rs:63:67: 63:79 error: mismatched types:
 expected `i64`,
    found `u64` [E0308]

~/.multirust/toolchains/nightly/cargo/registry/src/github.com-88ac128001ac3a9a/fs2-0.2.2/src/unix.rs:63     let ret = unsafe { libc::posix_fallocate(file.as_raw_fd(), 0, len as off_t) };
                                                                                                                                                                                       ^~~~~~~~~~~~

~/.multirust/toolchains/nightly/cargo/registry/src/github.com-88ac128001ac3a9a/fs2-0.2.2/src/unix.rs:63:67: 63:79 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error
Could not compile `fs2`.

The fix seems easy enough, but it's in unsafe code and I'm not experienced enough to know of the possible repercussions of satisfying the compiler blindly in this situation. Thank you.

0.4.0 doesn't compile for 32-bit targets

$ cargo clone --vers '0.4.0' fs2 && cd $_
$ cargo build --target i686-unknown-linux-gnu
    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling libc v0.2.18
   Compiling fs2 v0.4.0 (file:///home/japaric/tmp/fs2)
error[E0308]: mismatched types
   --> src/unix.rs:128:41
    |
128 |                 allocation_granularity: stat.f_frsize,
    |                                         ^^^^^^^^^^^^^ expected u64, found u32

error: aborting due to previous error

error: Could not compile `fs2`.

To learn more, run the command again with --verbose.

Incorrect documentation on try_lock_exclusive

I think the docs are wrong here, should be 'Locks the file for exclusive usage, or returns a an error if the file is currently locked (see lock_contended_error).'

fn try_lock_exclusive(&self) -> Result<()>
[−]
Locks the file for shared usage, or returns a an error if the file is currently locked (see lock_contended_error).

https://tantivy-search.github.io/tantivy/fs2/trait.FileExt.html#tymethod.try_lock_exclusive

Fails to build on ARM

[rust@vinipsmaker-pi fs2-rs]$ cargo build
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading tempdir v0.3.4
   Compiling winapi-build v0.1.1
   Compiling libc v0.2.4
   Compiling winapi v0.2.5
   Compiling kernel32-sys v0.2.1
   Compiling fs2 v0.2.2 (file:///home/rust/Projects/fs2-rs)
src/unix.rs:117:26: 117:39 error: mismatched types:
 expected `*const u8`,
    found `*const i8`
(expected u8,
    found i8) [E0308]
src/unix.rs:117         if libc::statvfs(cstr.as_ptr(), &mut stat) == -1 {
                                         ^~~~~~~~~~~~~
src/unix.rs:117:26: 117:39 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error
Could not compile `fs2`.

To learn more, run the command again with --verbose.

Is it time for a 1.0.0 release?

Seems like this API is pretty stable now. Would you be interested in reflecting that with a 1.0.0 release?