Failure to read creation time on ZFS about fclones

[patrickwolf]

It's not actually ZFS specific as I get the same issues with BTRFS. And based on this post it's not even ZFX/BTRFS but a Linux issue with the stat system call https://askubuntu.com/questions/470134/how-do-i-find-the-creation-time-of-a-file/980750#980750

[pkolaczk]

Thank you for reporting this. Fclones uses Rust's fs::symlink_metadata to read creation time of a file.

[fryfrog]

And pull request #8509 is why this isn't in ZFS yet. :(

[pkolaczk]

Do you know any workaround?
What is the most important is the modification time for checking if the file hasn't been modified since the time the report was generated. I wonder if I could special-case it for zfs.

If not possible to get at least the modification time, I can add a new flag to ignore modification time check.

[fryfrog]

There is stat output in my first post and it does show modify time, just not birth time. So if modify is all you care about, maybe just check that? Or maybe that is all you're looking for, but for some reason the underlying fs::symlink_metadata errors if birth isn't available, even if not needed?

Edit: To be clear, I'm not a developer so I'm not sure if any of the above is accurate or possible! :)

[pkolaczk]

Yeah, sure, np I'll investigate. So far I've found there is a third party crate for getting file times more portably, I'll give it a try.

Anyways, this probably looks like an issue to be filed against the Rust stdlib.

[pkolaczk]

I can't reproduce on zfs-fuse. Trying with kernel-based zfs...

Ok, so it looks like the issue is not really the statx call but just the fact that somehow the metadata returned from it doesn't have the creation time populated on the Rust side. I'll need to dig into the Rust stdlib source code...

So overall, fclones is usable on zfs, the only option that doesn't work is prioritizing by file creation time:

pkolaczk@p5520:/zfs/fs1$ fclones remove --priority newest <dupes.txt
[2021-06-06 07:49:59.307] fclones:  info: Started deduplicating
[2021-06-06 07:49:59.313] fclones:  warn: Failed to read creation time of file /zfs/fs1/foo3.txt: creation time is not available for the filesystem
[2021-06-06 07:49:59.313] fclones:  warn: Failed to read creation time of file /zfs/fs1/foo2.txt: creation time is not available for the filesystem
[2021-06-06 07:49:59.313] fclones:  warn: Failed to read creation time of file /zfs/fs1/foo2.txt: creation time is not available for the filesystem
[2021-06-06 07:49:59.313] fclones:  warn: Failed to read creation time of file /zfs/fs1/foo1.txt: creation time is not available for the filesystem
[2021-06-06 07:49:59.313] fclones:  warn: Could not determine files to drop in group with hash 6109f093b3fd5eb1060989c990d1226f and len 4: Metadata of some files could not be read.
[2021-06-06 07:49:59.313] fclones:  info: Processed 0 files and reclaimed 0 B space

But reading the modification date works properly:

pkolaczk@p5520:/zfs/fs1$ fclones remove --priority most-recently-modified <dupes.txt
[2021-06-06 07:50:24.263] fclones:  info: Started deduplicating
[2021-06-06 07:50:24.269] fclones:  info: Processed 2 files and reclaimed 8 B space

[pkolaczk]

Not a bug in the stdlib either, because the documentation for reading the created time says:

    /// The returned value corresponds to the `btime` field of `statx` on
    /// Linux kernel starting from to 4.11, the `birthtime` field of `stat` on other
    /// Unix platforms, and the `ftCreationTime` field on Windows platforms.

[fryfrog]

Yeah, I really think it is on ZFS for not doing statx. :(

[pkolaczk]

To summarize:

• fclones correctly uses statx birthtime to determine the creation time of a file, I double checked with the docs and this looks like the correct way of reading the creation time on linux; there is no bug in fclones nor Rust stdlib
• birthtime is not supported on ZFS currently, so technically this is a ZFS limitation
• statx itself works fine on ZFS and other metadata are fetched properly
• fclones usability is not hugely affected by this problem; it only affects --priority newest / oldest option, the other functionality works fine, including checking the modification timestamp

I think I can add another priority values for using ctime, so lack of birthtime on ZFS wouldn't be so limiting, but ctime is not exactly the same as birthtime. WDYT?

[fryfrog]

Isn't zfs's crtime birth time? If I were the boss of everything, I'd make zfs fix it. :)

[pkolaczk]

Well, if I only knew how to access crtime...
I can easily access ctime (change time), but not crtime.
Anyway, I'll check that - maybe there is some way to get down to the raw statx result ;)

[pkolaczk]

I know nothing about zfs, but that crtime looks pretty non-standard.
It is not even present in the raw (deprecated) stat structure:

https://doc.rust-lang.org/nightly/std/os/linux/raw/struct.stat.html
https://docs.rs/libc/0.2.95/libc/struct.statx.html

[pkolaczk]

I'm closing this because:

• It is not an fclones bug.
• The amount of functionality affected is tiny.
• I don't know any workaround, and if it exists it must be something very ZFS-specific, which would add a lot of complexity.
• Error messages clearly describe what is going on.