webdesus / fs_extra Goto Github PK

How can I create pull request to start contributing to this repo?

Realize the function copy/move list items. Where item its will be path to folder or file.

Currently fs_extra::remove_items does not act on symlinks that point to not-existing targets. It is not clear from the documentation what the intended behavior is, so I am not sure if this asks for fixing, or just documenting.

Reproduce

Assume a directory that contains symlinks, both to existing and non-existing targets:

mkdir dir/
touch dir/file-1
ln -s dir/file-1 dir/link-1
ln -s dir/file-2 dir/link-2

A minimal rust program for reproduction:

fn main() {
    for arg in std::env::args().skip(1) {
        let path = Path::new(&arg);
        println!("Removing path {:?}", path);
        match fs_extra::remove_items(&[path]) {
            Ok(_) => println!(" -> OK"),
            Err(err) => println!(" -> Err: {}", err)
        }
    }
}

And then:

$ tree dir
dir
├── file-1
├── link-1 -> file-1
└── link-2 -> file-2

1 directory, 3 files


$ cargo run -- dir/link*
Removing path "dir/link-1"
 -> OK
Removing path "dir/link-2"
 -> OK


$ tree dir              
dir
├── file-1
└── link-2 -> file-2

1 directory, 2 files

(above tested on MacOS and Linux)

Expected

Up to the maintainers of the library:

1. Remove All: Delete all provided paths. Here: link-1 and link-2 are both removed. file-1 stays (unchanged).
2. Document Symlink: Be clear that remove_items has caveat (or maybe more broadly: recommend not to use with symlinks?)
3. Error: Return an error in case symlink is "unclean" (is that a thing?)

I think (1) makes the most sense. Alas this was my expectation when using remove_items.

Causal

• remove_items uses file::remove, in case the item is not a directory
• file::remove only removes when the path exists
• Check whether path exists is done via std::path::Path::exists, that evaluates the symlink target (which here is missing) and not the symlink itself

It's pretty hard to display what is the result of the ls command. It would be cool to be able to print it.

        let mut path = result_path.as_os_str().to_os_string().into_string()?;
        if path.find("\\\\?\\") == Some(0) {
            path = path[4..].to_string();
        }

First, this is relatively inefficient, as it creates a copy of the path and then searches entire path.

But most importantly, this isn't safe. UNC paths can change their meaning if they're reinterpreted as DOS paths.

Consider using https://lib.rs/crates/dunce which strips the prefix only when it is safe to do so.

The latest release produces forwards-compatibility errors, fixed in 5f54a1a (Feb 2021) but never released. Could another release be published soon-ish?

Hi, I have this code:

Code

static DIR_MOVE_OPTIONS: fs_extra::dir::CopyOptions = fs_extra::dir::CopyOptions {
	overwrite: false,
	skip_exist: true,
	buffer_size: 64000,
	copy_inside: true,
	content_only: true,
	depth: 0
};

static FILE_MOVE_OPTIONS: fs_extra::file::CopyOptions = fs_extra::file::CopyOptions {
	overwrite: false,
	skip_exist: true,
	buffer_size: 64000
};

...

fn move_items(&self, from: &Path, to: &Path) -> Result<(), String> {
	/*let mut renamed_from = from.to_owned();
	if from.file_name().unwrap() != to.file_name().unwrap() {
		// move_dir doesn't rename directories when moving(?), so we'll have to do it ourselves
		renamed_from.pop();
		renamed_from.push(&to.file_name().unwrap());
		match fs::rename(from, &renamed_from) {
			Err(error) => { return Err(String::from(format!("{:?}", &error))) },
			Ok(_) => {}
		}
	}*/

	match match from.is_dir() {
		true => fs_extra::dir::move_dir(from, to, &DIR_MOVE_OPTIONS),
		false => fs_extra::file::move_file(from, to, &FILE_MOVE_OPTIONS)
	} {
		Err(error) => Err(String::from(format!("{:?}", &error))),
		Ok(_) => Ok(())
	}
}

(#24 is why I have that comment block there)

Error

Error { kind: PermissionDenied, message: \"Access is denied. (os error 5)\" }

thread 'main' panicked at 'Blackhole PANIC: Failed to move file/folder to Blackhole ("Error { kind: PermissionDenied, message: \"Access is denied. (os error 5)\" }")
"D:\\Servers\\Garry\'s Mod\\GarrysMod\\addons\\rrrrrrrrrrrrrrrrrrrr\\_bkeypads-4.0.3" -> "C:\\Users\\billy\\$BLACKHOLE\\_bkeypads-4.0.3 (14)"', src\show.rs:7:13     
stack backtrace:
   0:     0x7ff6270dff49 - std::backtrace_rs::backtrace::dbghelp::trace
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\..\..\backtrace\src\backtrace\dbghelp.rs:98
   1:     0x7ff6270dff49 - std::backtrace_rs::backtrace::trace_unsynchronized
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66    
   2:     0x7ff6270dff49 - std::sys_common::backtrace::_print_fmt
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\sys_common\backtrace.rs:79
   3:     0x7ff6270dff49 - std::sys_common::backtrace::_print::{{impl}}::fmt
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\sys_common\backtrace.rs:58
   4:     0x7ff6270f69ab - core::fmt::write
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\core\src\fmt\mod.rs:1080
   5:     0x7ff6270db8f8 - std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\io\mod.rs:1516
   6:     0x7ff6270e2a44 - std::sys_common::backtrace::_print
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\sys_common\backtrace.rs:61
   7:     0x7ff6270e2a44 - std::sys_common::backtrace::print
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\sys_common\backtrace.rs:48
   8:     0x7ff6270e2a44 - std::panicking::default_hook::{{closure}}
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:208
   9:     0x7ff6270e2628 - std::panicking::default_hook
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:227
  10:     0x7ff6270e32ff - std::panicking::rust_panic_with_hook
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:577
  11:     0x7ff6270e2e65 - std::panicking::begin_panic_handler::{{closure}}
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:484
  12:     0x7ff6270e082f - std::sys_common::backtrace::__rust_end_short_backtrace<closure-0,!>
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\sys_common\backtrace.rs:153
  13:     0x7ff6270e2e19 - std::panicking::begin_panic_handler
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:483
  14:     0x7ff6270e2dcc - std::panicking::begin_panic_fmt
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:437
  15:     0x7ff62705044e - blackhole::show::show::Show::panic
                               at C:\Users\billy\Documents\GitHub\blackhole\src\show.rs:7
  16:     0x7ff627040761 - blackhole::blackhole::blackhole::Blackhole::send
                               at C:\Users\billy\Documents\GitHub\blackhole\src\blackhole.rs:149
  17:     0x7ff62703c731 - blackhole::main
                               at C:\Users\billy\Documents\GitHub\blackhole\src\main.rs:70
  18:     0x7ff627038b2b - core::ops::function::FnOnce::call_once<fn(),tuple<>>
                               at C:\Users\billy\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:227
  19:     0x7ff62704472b - std::sys_common::backtrace::__rust_begin_short_backtrace<fn(),tuple<>>
                               at C:\Users\billy\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\sys_common\backtrace.rs:137
  20:     0x7ff6270451f1 - std::rt::lang_start::{{closure}}<tuple<>>
                               at C:\Users\billy\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:66
  21:     0x7ff6270e3503 - core::ops::function::impls::{{impl}}::call_once
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\library\core\src\ops\function.rs:280
  22:     0x7ff6270e3503 - std::panicking::try::do_call
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:381
  23:     0x7ff6270e3503 - std::panicking::try
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panicking.rs:345
  24:     0x7ff6270e3503 - std::panic::catch_unwind
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\panic.rs:382
  25:     0x7ff6270e3503 - std::rt::lang_start_internal
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\rt.rs:51
  26:     0x7ff6270451c3 - std::rt::lang_start<tuple<>>
                               at C:\Users\billy\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:65
  27:     0x7ff62703c940 - main
  28:     0x7ff6270fb394 - invoke_main
                               at d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  29:     0x7ff6270fb394 - __scrt_common_main_seh
                               at d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  30:     0x7ffe75ec7c24 - BaseThreadInitThunk
  31:     0x7ffe77e6d721 - RtlUserThreadStart

Screenshot of the directory

What works

dir::move_dir successfully creates all the directories and nested directories:

The only file copied is the first file in .git/objects/00:

It does appear to contain data:

Notes

• My program is running with administrative privileges
• I initially thought it had something to do Windows Subsystem for Linux, but using cmd and Windows git to initialize .git still causes the issue
• Deleting the .git folder stops the issue from happening
• Moving the directory myself using Windows Explorer works with no issue, and does not prompt for anything like a file being in use

Summary

If the directory contains symlinks, then dir::get_size follows the symlinks and counts the sizes of those files too. If the targets of those symlinks are inside that same directory structure, then it effectively means the file sizes of some files are counted twice. Or if the targets of those symlinks are not inside that directory structure, then it means the returned size isn't really that of the requested directory, but also includes that of arbitrary other locations on the filesystem, which doesn't seem desirable.

Steps to reproduce

1. mkdir test-dir
2. truncate -s 1M test-dir/large-file
3. ln -sr test-dir/large-file test-dir/symlink-to-large-file
4. Run the following (using fs_extra v1.2.0):

   fn main() {
       let size = fs_extra::dir::get_size("test-dir").expect("IO error calculating size");
       println!("size = {size}");
   }

Expected

fs_extra::dir::get_size() returns the same size as ls would, eg:

ls -al test-dir/
total 0
drwxr-xr-x 4 emorley staff     128 Jul  1 11:42 .
drwxr-xr-x 9 emorley staff     288 Jul  1 11:42 ..
-rw-r--r-- 1 emorley staff 1048576 Jul  1 11:41 large-file
lrwxr-xr-x 1 emorley staff      10 Jul  1 11:42 symlink-to-large-file -> large-file

...so 1048576 + 10 (for the symlink) + 128 (for the directory itself) = 1048714

ie something like:

$ cargo run -q
size = 1048714

Actual

fs_extra::dir::get_size() follows the symlink, so counts the size of that file twice. It also doesn't seem to count the size of the directory entry itself.

$ cargo run -q
size = 2097152

Notes

I believe the cause is that fs_extra::dir::get_size() uses std::fs::metadata not std::fs::symlink_metadata, and that the former follows symlinks. Also, use of std::fs::metadata is actually unnecessary since the std::fs::DirEntry returned by std::fs::read_dir already has the metadata we need.

Currently DirInfo.directories returns all directories recursively, but I need only to loop through children directories. I thought it was weird that it returns a recursive result by default. Can you add child_directories or change it to get_directories(recursive: bool)?

In my application, I need to get the size of a directory, and in order to avoid having to re-invent the wheel I was planning to use the dir::get_size method provided by this library (thanks for your work!).

Unfortunately though, the results I am getting differ from the ones expected (I am targeting a Linux machine). This is because the implementation does not return the size of a directory, but the sum of all the sizes of its files, without taking into account the size of the directory structure itself.

You can quickly test this behaviour on a Linux machine by running the mention method on a given directory, and then running for the same directory the command du --bytes -s {dir_path}.

If this is the expected behaviour, I would simply suggest to explicitly state it in the documentation in a clear way.

If this is not the expected behaviour, it can be easily fixed by slightly changing the implementation:

pub fn get_size<P>(path: P) -> Result<u64>
where
    P: AsRef<Path>,
{
    let mut result = 0;

    if path.as_ref().is_dir() {
        for entry in read_dir(&path)? {
            let _path = entry?.path();
            if _path.is_file() {
                result += _path.metadata()?.len();
            } else {
                result += get_size(_path)?;
            }
        }
        // add the directory data structure size
        result += path.as_ref().metadata()?.len();
    } else {
        result = path.as_ref().metadata()?.len();
    }
    Ok(result)
}

I understand this will also require some work to update the tests to reflect the new implementation. Please let me know if support is needed for this (can't promise when I'll be able to have a deeper look into this in case it's required).

Hello! 🐙
While scanning crates.io. we noticed a document-code mismatch issue in this crate, we think the doc examples are generally meant to illustrate why someone would want to use the item.

Location
documentation for fs_extra::file::move_file_with_progress
(https://docs.rs/fs_extra/1.3.0/fs_extra/file/fn.move_file_with_progress.html)

Summary
The example of the document does not mention the move_file_with_progress function, but mention move_file, is it a doc-code mismatch?

Thank you for checking out this issue! 👍🤗

If there's any reason you don't think this is a mismatch issue, please let me know, we're doing an empirical study of documentation in the Rust ecosystem. Your answers will help us and better promote the development of the Rust ecosystem.

hey. I want to copy all files and folders from "a" folder to "b" folder. I tried to use dir::get_dir_content and iterate with ::copy_item, but I couldn't came up with a simple solution. I was able to copy "a" folder inside the "b" folder, but this is not what I want to do.

Is there a simple solution to this problem? If yes, I would love to see it in examples. If no, an additional function would be great.

Add new option for copydeep how much more files will be copy by levels of directory.
This options will be disabled for move function, cuz it's imposible.

Standard library std::fs not have method receiving information about folder size.

Inside /my/source/.git there is a non-bare git repo with objects, history, etc... Then I want to copy the contents of that directory to a whole other directory at /my/dest/.git, which is empty.

let source = PathBuf::from("/my/source/.git");
let dest = PathBuf::from("/my/dest/.git");

fs_extra::dir::create(&dest, true); // create with erase

let mut copts = fs_extra::dir::CopyOptions::new();
copts.overwrite = true;
copts.copy_inside = true;

let _ = fs_extra::dir::copy(&source, &dest, &copts);

After this I would expect to see the following inside /my/dest/.git:

/
    my/
        dest/
            .git/
                refs/
                objects/
                ...

But instead I see (note nested .git inside .git)

/
    my/
        dest/
            .git/
                .git/
                    refs/
                    objects/
                    ...

This looks like the dest path should be the parent directory to where I want to copy the first directory? Documentation states:

Copies the directory contents from one place to another using recursive method.

I would assume this means that the contents from source directory become the contents of destination directory, while the directory itself is not copied anywhere?

I'm using fs_extra 1.1.0 and latest stable Rust.

I'm getting the following error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: NotFound, message: "No such file or directory (os err
or 2)" }', src/main.rs:4:82

although it should work (I think).

Reproduction steps

Create test directory:

mkdir -p /tmp/rofl1/yeet
touch /tmp/rofl1/test.txt
touch /tmp/rofl1/yeet/bruh.txt

Cargo.toml

[package]
name = "rust_tmp"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
fs_extra = "1.2.0"

src/main.rs

use fs_extra::dir::CopyOptions;

fn main() {
    fs_extra::dir::copy("/tmp/rofl1", "/tmp/rofl2", &CopyOptions::new()).unwrap();
}

What am I doing wrong?

is

fs_extra::dir_copy("a/b", "c/d");

goint to create c/d/b, or just c/d? Does it depend on c/d existing or not? The documentation is very unclear.

"Copies the directory contents from one place to another using recursive method. This function will also copy the permission bits of the original files to destionation files (not for directories)."

Explains non of it.

this might already be supported, but to be honest from the docs I can't tell. The equivalent of what I want to do on the command line is.

❯ ln -s shared/.gitattributes
❯ cp --dereference .gitattributes gitattr
❯ ls -la
lrwxrwxrwx   - xeno 25 Jan 13:54 .gitattributes -> shared/.gitattributes
.rw-r--r--  68 xeno 25 Jan 13:54 gitattr

I'd appreciate it if you could make a new point release containing #38. This would help in evaluating the prevalence of macro trailing semicolons in the Rust ecosystem, as I'll be able to bump the fs_extra dependency in various places.

Thanks!

Hello, thanks for the great crate! It really helps cut down on boilerplate for common fs tasks.

I was wondering if you would be open to me adding the builder pattern for fs_extra::dir::CopyOptions and fs_extra::file::CopyOptions? They should both be valid for the full time they're being built so that means it should be rather simple to add and it allows for

use fs_extra::dir;
let mut opts = dir::CopyOptions::new();
opts.overwrite = true;

to be represented as

use fs_extra::dir;
let opts = dir::CopyOptions::new()
    .overwrite(true);

There's not too much difference here, but getting to drop the mut is slightly nicer since it means that refactoring away whatever uses the thing using opts will let the rust compiler pick up that opts is unused in the second and not in the first.

Nothing too major, but might be a bit of a nicety that people are used to. Let me know what you think!

Hi I have some questions regarding this crate.

1. Does it use shell commands at all?
2. Lets just say I wanted to remove everything recursively and I don't want to delete any files with a specific name (lets call the file name "Rust"). So do you happen to know how I would delete everything but the name "Rust"?

I have been using the fs_crate for file copying in my project, and I've found it to be a valuable tool. However, I noticed that the crate currently provides options to either overwrite or skip existing files/folders, but it lacks the functionality to copy files with sequential naming.

This feature would be particularly useful in scenarios where users want to duplicate files to a destination directory but need to avoid overwriting existing files. The sequential naming option ensures that each copied file has a unique name without the risk of overwriting.

Hey! Thx for lib, but I have noticed that directory copy behaves in non-idempotent way.

Suppose we have:

let pathbufs = [PathBuf::from("directory1"), PathBuf::from("directory2"), PathBuf::from("directory3")];
let options = fs_extra::dir::CopyOptions {
        overwrite: true,
        copy_inside: true,
        ..Default::default()
    };
let to = PathBuf::new("current");

    for p in pathbufs {
        if let Err(e) = fs_extra::dir::copy(&p, &to, &options) {
            eprintln!(
                "Failed to copy directory: {:?} path: {}",
                e,
                p.display()
            );
        }
    }

First time running copying content of the directory1 in the root of the current path, and the second time it works as I want - I have all 3 dirs there but with directory1 files from previous run. Is there a way to copy all 3 directories if there are no current/ directory created previously?

Result after 1st run:

current/
|-- file1.txt
|-- file11.txt
|-- directory2
|   |-- ...
|-- directory3
`   |--...

Result after 2nd run

current/
|-- directory1
|   |-- file1.txt
|   `-- file11.txt
|-- file1.txt
|-- file11.txt
|-- directory2
|   |-- ...
|-- directory3
`   |--...

I understand that this is exactly the behavior of cp -r on Linux, but maybe there could be a flag for this case?

I hope fs_extra's copy for directory can simulate cp command. e.g., cp d1 d2, if d2 does not exist it copy the whole directory d1 to d2 (in this ase d1 and d2 in same folder with same substructure); if d2 exists, copy d1 into d2 (as what has been implemented). The default behavior can be changed by modifying CopyOptions.

Move should use rename, not copy and remove.
Especially, that now, virtually whole code of move is a copy paste of copy with minor differences that even if it to stay in current form should be refactored to get rid of these duplicate codes.

Add support for copying files while preserving timestamps (do not change the last modified time), just like the cp command -p or --preserve option: http://man7.org/linux/man-pages/man1/cp.1.html

       -p     same as --preserve=mode,ownership,timestamps

       --preserve[=ATTR_LIST]
              preserve the specified attributes (default:
              mode,ownership,timestamps), if possible additional attributes:
              context, links, xattr, all

I am currently working on a tool to reuse cargo build artifacts from the "target" directory, and it turns out that cargo checks the last modified time on files and triggers a rebuild if it changed.

          if let Err(e) = fs_extra::dir::move_dir(&thing, &destination_file, &fs_extra::dir::CopyOptions::new()) {
                messages += format!("Failed to move folder {}, reason {}\n", thing, e).as_str();
                continue 'next_result;
            }

such code prints(error contains file name)

Failed to move file /home/rafal/TESTTT/TT/AutoSave_0091.sav, reason Path "/home/rafal/TESTTT/TT/AutoSave_0091.sav" is exist

but such code from std::fs don't print name of file/folder which is I think better, because I can put this name in any other place

        if let Err(e) = fs::metadata("/hehe") {
            messages += format!("HEHE {}, reason {}\n", thing, e).as_str();
            continue 'next_result;
        }

HEHE /hehe reason No such file or directory (os error 2)

tests/file.rs and test/dir.rs rewrite with use Path.

A move should be instantaneous when files/folders are on the same filesystem.
But instead, a full copy/remove is done when doing move_*

Not sure if this is within the scope of this project, but I'll make an issue anyways

Hey, what's the difference between SkipExisting and Overwrite? I mean doesn't overwrite selected mean don't skip existing and and not checked means skip existing?
fs_extra::dir::CopyOptions

Error { kind: InvalidFile, message: "Path \"/tmp/state/0/data/ipc\" is not a file!" }

I wish it would just skip copying it...

By default, the cp command on Linux will attempt to reflink on Copy-on-Write filesystems (--reflink=auto). I think the CopyOptions struct should have similar option.

For example, if CopyOptions::reflink is set to RefLink::Auto, the copy function will use the reflink_or_copy function from the reflink crate to copy files.

The advantages of reflink:

• It is faster to reflink big files than to copy big files.
• It is faster to reflink many files recursively than to copy many files recursively.
• It saves disk spaces.

Summary

If one of the file entries being copied by fs_extra::dir::copy happens to be a symlink to a directory, then that symlink is converted to a real directory as part of copying, resulting in (a) duplicate file contents in the destination directory, (b) non-identical contents between source and destination (which defeats the point of copying a directory verbatim).

Note: Symlinks to files are preserved, this is just about symlinks that target directories.

Steps to reproduce

Using fs_extra = "=1.2.0" with rustc 1.63.0-beta.3 on macOS 12.4, run the following:

use fs_extra::dir::CopyOptions;
use std::fs::{self, File};
use std::path::{Path, PathBuf};

fn main() {
    let source_dir = PathBuf::from("source");
    let destination_dir = PathBuf::from("destination");
    if source_dir.exists() {
        fs::remove_dir_all(&source_dir).unwrap();
    }
    if destination_dir.exists() {
        fs::remove_dir_all(&destination_dir).unwrap();
    }

    let subdir = source_dir.join("subdir");
    fs::create_dir_all(&subdir).unwrap();
    File::create(&subdir.join("file.txt")).unwrap();
    create_dir_symlink("subdir", &source_dir.join("symlink-to-subdir")).unwrap();

    let options = &CopyOptions {
        content_only: true,
        ..CopyOptions::default()
    };
    fs_extra::dir::copy(source_dir, destination_dir, options).unwrap();
}

#[cfg(target_family = "unix")]
fn create_dir_symlink<P: AsRef<Path>, Q: AsRef<Path>>(original: P, link: Q) -> std::io::Result<()> {
    std::os::unix::fs::symlink(original.as_ref(), link.as_ref())
}

#[cfg(target_family = "windows")]
fn create_dir_symlink<P: AsRef<Path>, Q: AsRef<Path>>(original: P, link: Q) -> std::io::Result<()> {
    std::os::windows::fs::symlink_dir(original.as_ref(), link.as_ref())
}

Expected

The contents of source/ and destination/ to be identical, and for any symlinks targetting directories to be preserved as symlinks.

Actual

The source/ and destination/ directories have different contents, with the symlinked directory now being a real directory containing duplicate files.

Source:

$ ls -alR source/
source/:
total 0
drwxr-xr-x  4 emorley staff 128 Jul  6 14:03 .
drwxr-xr-x 10 emorley staff 320 Jul  6 14:03 ..
drwxr-xr-x  3 emorley staff  96 Jul  6 14:03 subdir
lrwxr-xr-x  1 emorley staff   6 Jul  6 14:03 symlink-to-subdir -> subdir

source/subdir:
total 0
drwxr-xr-x 3 emorley staff  96 Jul  6 14:03 .
drwxr-xr-x 4 emorley staff 128 Jul  6 14:03 ..
-rw-r--r-- 1 emorley staff   0 Jul  6 14:03 file.txt

Destination:

$ ls -alR destination/
destination/:
total 0
drwxr-xr-x  4 emorley staff 128 Jul  6 14:03 .
drwxr-xr-x 10 emorley staff 320 Jul  6 14:03 ..
drwxr-xr-x  3 emorley staff  96 Jul  6 14:03 subdir
drwxr-xr-x  3 emorley staff  96 Jul  6 14:03 symlink-to-subdir

destination/subdir:
total 0
drwxr-xr-x 3 emorley staff  96 Jul  6 14:03 .
drwxr-xr-x 4 emorley staff 128 Jul  6 14:03 ..
-rw-r--r-- 1 emorley staff   0 Jul  6 14:03 file.txt

destination/symlink-to-subdir:
total 0
drwxr-xr-x 3 emorley staff  96 Jul  6 14:03 .
drwxr-xr-x 4 emorley staff 128 Jul  6 14:03 ..
-rw-r--r-- 1 emorley staff   0 Jul  6 14:03 file.txt

I believe there is a race condition when recursively copying:

create_all will fail when the directory already exists. Having two threads copying files to the same directory, this may result in an a File exists error.

The my understanding tokio compensates for this by ignoring this case as an error: https://docs.rs/tokio/latest/tokio/fs/fn.create_dir_all.html

Notable exception is made for situations where any of the directories specified in the path could not be created as it was being created concurrently. Such cases are considered to be successful. That is, calling create_dir_all concurrently from multiple threads or processes is guaranteed not to fail due to a race condition with itself.

I think it makes sense to implement the same logic here.

I am using standard example from documentation.

dir::copy(&src, &dst, &dir_options)?;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::convert::From<fs_extra::error::Error>` is not implemented for `std::io::Error`

Same for file::copy.

It is Result<u32>, not Result<()>, but the documentation does not tell what that number (u32) means.

Some files can be bigger but we want to change only one little part of this file. Will be a stupid load all full file data in a memory.
It's will be interesting for people who work on own format.
I want to make two operations at the current moment:

• change_block. This operation will change only one information block
• change_blocks. This function will get many blocks for changes and will be made it more optimize
• Optimize change_blocks function.
• Write docs for both functions

It would very nice to have an exclude paths in CopyOptions. For example, I want to copy one folder recursively but excluding some files / directories.

Thanks in advance for your work 😊

Hi, we use fs_extra in Nushell; thanks for writing it!

We recently had a scary bug report, and it looks like the root cause is fs_extra::dir::move_dir(). If you use move_dir() to change the case of a directory name on Windows (ex: "Test" -> "test"), the directory is deleted and fs_extra reports success.

For example:

// This deletes the directory named Test and does not panic!
fs_extra::dir::move_dir("Test", "test", &options).unwrap();

Steps to Reproduce

I've put together a minimal repro: https://github.com/rgwood/fs-extra-repro

Version Info

fs-extra v1.2.0
Windows 11 (can't reproduce on Linux)
rustc v1.63.0

I would like to see a special copy option to transform symbolic links using absolute file paths to symbolic links relative to another path. The reason I am looking for such an option is that I am working on a tool to create sysroots for the purpose of cross-compiling based on the contents of docker containers (https://github.com/devolutions/albatross-rs). Just like chroot environments, the absolute symbolic links inside the docker container will not work when copied as-is for usage outside of the container.

For instance, if I copy the contents of a 32-bit Linux container, I get symbolic links like this:
/this-is-my-new-sysroot-dir/usr/lib/i386-linux-gnu/libz.so -> /lib/i386-linux-gnu/libz.so.1.2.11
where /lib/i386-linux-gnu/libz.so.1.2.11 is obviously something that only exists relative to the root of the original file system contained in the docker container. The same issue occurs with trying to copy files from a chroot for usage on the host.

The easiest way to work around the issue would be to provide the old root and modify all symbolic links to relative links (libz.so -> libz.so.1.2.11, no absolute path). Another option would be to simply "rebase" all symbolic links to another absolute path based on where the files will be copied, but I would rather go with the relative symbolic links option.

What is the meaning of those flags?

I'm responsible for maintaining the Fedora Linux packages for many Rust crates, the fs_extra crate among them. I tried updating our package to the recently released v1.3.0, but it appears that the test suite produces a lot of errors (all of which originating in tests/dir.rs):

failures:

---- it_copy_progress_work stdout ----
thread 'it_copy_progress_work' panicked at 'assertion failed: `(left == right)`
  left: `55`,
 right: `57`', tests/dir.rs:853:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- it_copy_with_progress_inside_no_overwrite_work_target_dir_exist_with_source_dir_exist stdout ----
thread 'it_copy_with_progress_inside_no_overwrite_work_target_dir_exist_with_source_dir_exist' panicked at 'assertion failed: `(left == right)`
  left: `57`,
 right: `59`', tests/dir.rs:3970:21

---- it_copy_with_progress_inside_work_target_dir_exist_with_no_source_dir_named_sub_dir stdout ----
thread 'it_copy_with_progress_inside_work_target_dir_exist_with_no_source_dir_named_sub_dir' panicked at 'assertion failed: `(left == right)`
  left: `57`,
 right: `59`', tests/dir.rs:3881:21

---- it_copy_with_progress_inside_work_target_dir_not_exist stdout ----
thread 'it_copy_with_progress_inside_work_target_dir_not_exist' panicked at 'assertion failed: `(left == right)`
  left: `55`,
 right: `57`', tests/dir.rs:3800:21

---- it_copy_with_progress_inside_overwrite_work_target_dir_exist_with_source_dir_exist stdout ----
thread 'it_copy_with_progress_inside_overwrite_work_target_dir_exist_with_source_dir_exist' panicked at 'assertion failed: `(left == right)`
  left: `57`,
 right: `59`', tests/dir.rs:4058:21

---- it_get_dir_content stdout ----
thread 'it_get_dir_content' panicked at 'assertion failed: `(left == right)`
  left: `56`,
 right: `58`', tests/dir.rs:2936:5

---- it_copy_with_progress_work_dif_buf_size stdout ----
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `56`,
 right: `58`', tests/dir.rs:980:13
thread 'it_copy_with_progress_work_dif_buf_size' panicked at 'assertion failed: `(left == right)`
  left: `56`,
 right: `58`', tests/dir.rs:1002:9

---- it_get_dir_content_many_levels stdout ----
thread 'it_get_dir_content_many_levels' panicked at 'assertion failed: `(left == right)`
  left: `140`,
 right: `202`', tests/dir.rs:2998:5

---- it_move_dir_with_progress_inside_no_overwrite_work_target_dir_exist_with_source_dir_exist stdout ----
thread 'it_move_dir_with_progress_inside_no_overwrite_work_target_dir_exist_with_source_dir_exist' panicked at 'assertion failed: `(left == right)`
  left: `57`,
 right: `59`', tests/dir.rs:4632:21

---- it_move_dir_with_progress_inside_work_target_dir_exist_with_no_source_dir_named_sub_dir stdout ----
thread 'it_move_dir_with_progress_inside_work_target_dir_exist_with_no_source_dir_named_sub_dir' panicked at 'assertion failed: `(left == right)`
  left: `57`,
 right: `59`', tests/dir.rs:4533:21

---- it_move_dir_with_progress_inside_work_target_dir_not_exist stdout ----
thread 'it_move_dir_with_progress_inside_work_target_dir_not_exist' panicked at 'assertion failed: `(left == right)`
  left: `55`,
 right: `57`', tests/dir.rs:4446:21

---- it_move_dir_with_progress_inside_overwrite_work_target_dir_exist_with_source_dir_exist stdout ----
thread 'it_move_dir_with_progress_inside_overwrite_work_target_dir_exist_with_source_dir_exist' panicked at 'assertion failed: `(left == right)`
  left: `57`,
 right: `59`', tests/dir.rs:4727:21

---- it_move_progress_work stdout ----
thread 'it_move_progress_work' panicked at 'assertion failed: `(left == right)`
  left: `55`,
 right: `57`', tests/dir.rs:2357:21

---- it_move_with_progress_work_dif_buf_size stdout ----
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `56`,
 right: `58`', tests/dir.rs:2501:13
thread 'it_move_with_progress_work_dif_buf_size' panicked at 'assertion failed: `(left == right)`
  left: `56`,
 right: `58`', tests/dir.rs:2523:9


failures:
    it_copy_progress_work
    it_copy_with_progress_inside_no_overwrite_work_target_dir_exist_with_source_dir_exist
    it_copy_with_progress_inside_overwrite_work_target_dir_exist_with_source_dir_exist
    it_copy_with_progress_inside_work_target_dir_exist_with_no_source_dir_named_sub_dir
    it_copy_with_progress_inside_work_target_dir_not_exist
    it_copy_with_progress_work_dif_buf_size
    it_get_dir_content
    it_get_dir_content_many_levels
    it_move_dir_with_progress_inside_no_overwrite_work_target_dir_exist_with_source_dir_exist
    it_move_dir_with_progress_inside_overwrite_work_target_dir_exist_with_source_dir_exist
    it_move_dir_with_progress_inside_work_target_dir_exist_with_no_source_dir_named_sub_dir
    it_move_dir_with_progress_inside_work_target_dir_not_exist
    it_move_progress_work
    it_move_with_progress_work_dif_buf_size

test result: FAILED. 68 passed; 14 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

I originally saw these failures in our containerized build environment (which has limited OS access in some ways), but I also reproduced it by just cloning this git repo and just running cargo test.

Tracing back the errors to where they originate in test sources, it appears that the calculation of expected total directory size is off - the assertion failures seem to point to lines like assert_eq!(get_dir_size() * 2 + 17, process_info.total_bytes);. I don't understand what this assertion does, but in most cases, it seems to be off by 2 (or in one case, by 62).

Additional information:

• Operating system: Fedora Linux 37 and Rawhide on Linux 6.1.9 (or newer)
• rust 1.67.0 stable and 1.69.0-nightly (f3126500f 2023-02-02)

thread 'it_copy_exist_overwrite' panicked at 'called Result::unwrap() on an Err value: Error { kind: AlreadyExists, message: "entity already exists" }', /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libcore/result.rs:837

Err(Error { kind: NotFound, message: "Path "/Users/edison/Downloads/123\ aaaa/归档(5)/panda" does not exist or you don't have access!" })