Comments (13)
This is equally bizarre output. I grabbed just the --file-hash
section:
--file-hash arg (=xxh3-128) choice of file hashing function (none,
xxh3-128, xxh3-64)
from dwarfs.
I used the -clang.tar.zst
version and at least the mkdwarfs
worked perfectly. I first checked to see the hash sections and it showed the following list:
$ ./bin/mkdwarfs -H | grep -A6 file-hash
--file-hash arg (=xxh3-128) choice of file hashing function (none,
blake2b512, blake2s256, md5, md5-sha1,
ripemd160, sha1, sha224, sha256,
sha3-224, sha3-256, sha3-384, sha3-512,
sha384, sha512, sha512-224, sha512-256,
shake128, shake256, sm3, xxh3-128,
xxh3-64)
Well done! 🏁 🚀
from dwarfs.
Interesting! Can you do me a favour and run:
$ mkdwarfs -H
The output should contain something like this:
--file-hash arg (=xxh3-128) choice of file hashing function (none,
blake2b512, blake2s256, md5, md5-sha1,
ripemd160, sha1, sha224, sha256,
sha3-224, sha3-256, sha3-384, sha3-512,
sha384, sha512, sha512-224, sha512-256,
shake128, shake256, sm3, xxh3-128,
xxh3-64)
DwarFS uses EVP_sha512_256()
to compute digests for each file system block and I assume this is what triggers the assertion.
Unfortunately, I have no experience with FIPS mode; from the OpenSSL documentation I would assume that SHA512/256 would even work in FIPS mode as it states "CONFORMING TO NIST FIPS 180-4".
from dwarfs.
The top of -H
looks like the following:
mkdwarfs (v0.9.7 [2024-04-10])
built for x86_64, Linux-5.15.0-101-generic, Clang 18.1.0
using: FLAC++-1.4.3, boost-1.83.0, brotlidec-1.1.0, brotlienc-1.1.0,
crypto-3.0.13, fmt-10.2.1, jemalloc-5.3.0, lz4-1.9.4, lzma-5.4.5,
xxhash-0.8.2, zstd-1.5.5
Which seems to be missing openssl 🤔
from dwarfs.
This is equally bizarre output. I grabbed just the
--file-hash
section:--file-hash arg (=xxh3-128) choice of file hashing function (none, xxh3-128, xxh3-64)
Yeah, I kinda expected something like this. Didn't quite expect that none of the OpenSSL algorithms would be present, though.
The top of
-H
looks like the following:mkdwarfs (v0.9.7 [2024-04-10]) built for x86_64, Linux-5.15.0-101-generic, Clang 18.1.0 using: FLAC++-1.4.3, boost-1.83.0, brotlidec-1.1.0, brotlienc-1.1.0, crypto-3.0.13, fmt-10.2.1, jemalloc-5.3.0, lz4-1.9.4, lzma-5.4.5, xxhash-0.8.2, zstd-1.5.5
Which seems to be missing openssl 🤔
That's actually expected; the OpenSSL is not (yet) part of this list. It's nonetheless linked into the binary.
I wonder if OpenSSL has some dependency on kernel functions and the (rather old) kernel and OpenSSL aren't getting along to well.
from dwarfs.
I wonder if OpenSSL has some dependency on kernel functions and the (rather old) kernel and OpenSSL aren't getting along to well.
I don't think it's that. I just installed Fedora 22 and tried:
[root@localhost ~]# uname -a
Linux localhost.localdomain 4.4.14-200.fc22.x86_64 #1 SMP Fri Jun 24 21:19:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# ./dwarfs-universal-0.9.7-Linux-x86_64-clang --tool=mkdwarfs -H | grep -A 10 file-hash
warning: failed to set user default locale: locale::facet::_S_create_c_locale name not valid
--file-hash arg (=xxh3-128) choice of file hashing function (none,
blake2b512, blake2s256, md5, md5-sha1,
ripemd160, sha1, sha224, sha256,
sha3-224, sha3-256, sha3-384, sha3-512,
sha384, sha512, sha512-224, sha512-256,
shake128, shake256, sm3, xxh3-128,
xxh3-64)
That kernel is even older. The binary seems to work just fine:
[root@localhost ~]# ./dwarfs-universal-0.9.7-Linux-x86_64-clang --tool=mkdwarfs -i /usr/ -o /dev/null --force -l4
warning: failed to set user default locale: locale::facet::_S_create_c_locale name not valid
I 19:01:53.486009 scanning "/usr"
I 19:01:54.263798 assigning directory and link inodes...
I 19:01:54.267666 waiting for background scanners...
I 19:01:55.724926 scanning CPU time: 4.714s
I 19:01:55.724941 finalizing file inodes...
I 19:01:55.734015 saved 15.55 MiB / 990.2 MiB in 2504/42710 duplicate files
I 19:01:55.734257 assigning device inodes...
I 19:01:55.734591 assigning pipe/socket inodes...
I 19:01:55.734846 building metadata...
I 19:01:55.734862 building blocks...
I 19:01:55.734915 saving names and symlinks...
I 19:01:55.735250 waiting for segmenting/blockifying to finish...
I 19:01:55.748615 updating name and link indices...
I 19:02:00.024608 total ordering CPU time: 256.7us
I 19:02:00.024765 total segmenting CPU time: 4.212s
I 19:02:00.027036 saving chunks...
I 19:02:00.034627 saving directories...
I 19:02:00.039707 saving shared files table...
I 19:02:00.054576 saving names table... [9.325ms]
I 19:02:00.056605 saving symlinks table... [1.809ms]
I 19:02:00.080138 waiting for compression to finish...
I 19:02:00.082730 compressed 990.2 MiB to 316.3 MiB (ratio=0.319461)
I 19:02:00.087192 compression CPU time: 20.67s
I 19:02:00.087370 filesystem created without errors [6.602s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish
4653 dirs, 1244/3649 soft/hard links, 42710/42710 files, 0 other
original size: 990.2 MiB, hashed: 144.4 MiB (29093 files, 60.95 MiB/s)
scanned: 974.6 MiB (36557 files, 158.5 MiB/s), categorizing: 0 B/s
saved by deduplication: 15.55 MiB (2504 files), saved by segmenting: 91.95 MiB
filesystem: 882.7 MiB in 221 blocks (46290 chunks, 36556/36556 fragments, 36557 inodes)
compressed filesystem: 221 blocks/316.3 MiB written
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏100% 🌒
[root@localhost ~]#
from dwarfs.
I was able to build it inside a ubuntu:23:10 container it works fine, so it's probably something about FIPS not being linked in and so you get no hashes.
from dwarfs.
My only guess is machine has FIPS enabled on it.
That's definitely the root cause. I also installed Fedora 29 in a VM. mkdwarfs
worked fine right after the system booted. Then I followed the instructions to enable FIPS mode and got:
[root@localhost ~]# uname -a
Linux localhost.localdomain 5.3.11-100.fc29.x86_64 #1 SMP Tue Nov 12 20:41:25 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
[root@localhost ~]# ./dwarfs-universal-0.9.7-Linux-x86_64-clang --tool=mkdwarfs -H | grep -A 3 file-hash
--file-hash arg (=xxh3-128) choice of file hashing function (none,
xxh3-128, xxh3-64)
--progress arg (=unicode) progress mode (ascii, none, simple,
unicode)
from dwarfs.
Well done @mhx. That certainly proves it.
from dwarfs.
The funny thing is: I can build a statically linked test program on my Gentoo machine, copy it to the FIPS-enabled Fedora 29 VM, and get:
[root@localhost ~]# ./digest
-> md5-sha1
-> sha512-224
-> sm3
-> sha512
-> sha384
-> sha224
-> md4
-> blake2b512
-> ripemd160
-> sha256
-> sha512-256
-> shake128
-> whirlpool
-> blake2s256
-> sha3-256
-> shake256
-> mdc2
-> sha3-224
-> sha3-512
-> sha3-384
-> md5
-> sha1
Digest is: 0686f0a605973dc1bf035d1e2b9bad1985a0bff712ddd88abd8d2593e5f99030
And I think that works because my Gentoo OpenSSL library is built without FIPS support:
[ebuild R ] dev-libs/openssl-3.0.13:0/3::gentoo USE="asm static-libs -fips -ktls -rfc3779 -sctp -test -tls-compression -vanilla -verify-sig -weak-ssl-ciphers" ABI_X86="32 (64) (-x32)" CPU_FLAGS_X86="(sse2)" 0 KiB
from dwarfs.
Reopening as I think this is actually fixable.
from dwarfs.
If you want, you can try dwarfs-universal-0.9.7-2-gdadc2e0254-Linux-x86_64-clang or dwarfs-0.9.7-2-gdadc2e0254-Linux-x86_64-clang.tar.zst. You'll need to be signed in to GitHub to be able to download the build artifacts.
The binaries work fine on my FIPS-enabled VM. The next release will contain the fix.
from dwarfs.
Fixed in v0.9.8. Thanks for your help!
from dwarfs.
Related Issues (20)
- cannot enter subdirs of overlayed dwarfs HOT 10
- Error while building HOT 2
- Fuse Passthrough
- Are the prebuilt binaries affected by xz/liblzma backdoor? HOT 6
- [Feature request] Allow providing dwarfs with a dedup library HOT 4
- official debian package, allow building without git? HOT 4
- [MacOS] DwarFS mount not seen through Finder HOT 5
- [Core Dump] Signal 7 (SIGBUS) (code: nonexistent physical address) on making archive of currently running OS (possibly bad use case) HOT 2
- Homebrew formula HOT 22
- some problem on the README.md files. Please Check up and Fix. HOT 1
- read scalability issues with large archives HOT 9
- Unexpected exception: `inode has no file (any)` HOT 8
- [Feature Request] Mounting multiple archives to the same path HOT 12
- Segfault when using the mold linker HOT 2
- Cannot build v0.9.9 on Ubuntu 22.04 HOT 5
- Vendor fbthrift & folly using vcpkg HOT 2
- exception thrown in worker thread: class dwarfs::runtime_error: lzma_stream_encoder HOT 2
- [Feature Request] Provide non-generic packaging CI for major linux distributions HOT 2
- Document memory behaviour and give tips for dealing with many files HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dwarfs.