Coder Social home page Coder Social logo

gottox / sqsh-tools Goto Github PK

View Code? Open in Web Editor NEW
30.0 5.0 4.0 14.22 MB

a fresh squashfs implementation. BSD-2 licensed.

Home Page: https://gottox.de/sqsh-tools/

License: BSD 2-Clause "Simplified" License

C 95.42% Meson 1.69% Dockerfile 0.05% Shell 1.19% Makefile 0.17% C++ 0.28% Roff 1.19%
compression filesystem c library squashfs libsquashfs linux unix zstd fuse

sqsh-tools's Introduction

sqsh-tools CI codecov Quality Gate Status License

squashfs is an open and free compressed read-only filesystem. It is used in embedded devices, live-CDs, or in packaging. The original implementation resides in the linux kernel, but there are also userspace implementations.

This project provides a userspace implementation of the squashfs filesystem containing a set of command line tools and a C library.

Building

Dependencies

  • libc: Any POSIX compliant libc should work. Open a bug if it doesn't.
  • zlib optional: For gzip compression support.
  • liblz4 optional: For lz4 compression support.
  • liblzma optional: For lzma compression support.
  • libzstd optional: For zstd compression support.
  • fuse3 optional: For mounting squashfs archives.
  • fuse2 optional: For mounting squashfs archives on systems that don't support fuse3. e.g. OpenBSD.
  • libcurl optional: For transparently reading squashfs archives from the internet without downloading them first.

Note that to do anything useful with libsqsh, you need to have at least one of the compression libraries enabled.

Compile & Install

meson setup build
cd build
meson compile
meson install

tools

This project provides a set of command line tools to interact with squashfs archives.

  • sqsh-cat: Prints the content of one or multiple files to stdout.
  • sqsh-ls: Lists the content of a directory.
  • sqsh-stat: Prints the metadata of a file, directory, or the whole archive.
  • sqsh-unpack: Extracts a squashfs archive to a directory.
  • sqsh-xattr: Prints the extended attributes of a file or directory.
  • sqshfs: Mounts a squashfs archive to a directory. There are two versions of this tool, one for fuse3 and one for fuse2.

libsqsh

libsqsh is a purely 2-Clause BSD Licensed implementation of the squashfs filesystem in C11. It covers the complete squashfs feature set, while still having minimal memory footprint.

Note that libsqsh only supports reading squashfs archives. If you want to create squashfs archives, you can either use squashfs-tools, which provides a command line interface, or squashfs-tools-ng, which provides both a command-line interface and a C library interface.

Features

  • Complete feature set: libsqsh supports all features of the squashfs file system.

  • Modern C11 implementation: Written in modern C11, libsqsh is designed for efficiency and compatibility with C++, C, and other languages.

  • High-Level API: The high-level API provides easy-to-use functions to access the contents of a to the contents of a squashfs archive. It is designed to be easy to use and fast to develop with.

  • Low-Level API: The low-level API provides zero-copy interfaces to the squashfs archive. It is designed to be both memory and CPU efficient.

  • Pure 2-Clause-BSD License: libsqsh is licensed under the 2-Clause BSD license. This allows you to use libsqsh in any project, even commercial ones.

Example

This is a simple example that a) prints the content of a file and b) lists the content of a directory.

struct SqshArchive *archive =
		sqsh_archive_open("/path/to/archive.squashfs", NULL, NULL);

uint8_t *contents = sqsh_easy_file_content(archive, "/path/to/file", NULL);
assert(contents != NULL);
const size_t size = sqsh_easy_file_size(archive, "/path/to/file", NULL);
fwrite(contents, 1, size, stdout);
free(contents);

char **files = sqsh_easy_directory_list(archive, "/path/to/dir", NULL);
assert(files != NULL);
for (int i = 0; files[i] != NULL; i++) {
	puts(files[i]);
}
free(files);

sqsh_archive_close(archive);

Find further examples in the examples directory.

LZO2

LZO2 is a fast compression algorithm. Unfortunately the current implementation is GPL licensed and therefore not included in this library. If you want to use LZO2 there's and independent glue library called libsqsh-lzo.

Resource

sqsh-tools's People

Contributors

dr-emann avatar gottox avatar probonopd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

sqsh-tools's Issues

libsqsh should be 0-copy

Currently there are a few points where libsqsh needlessly copies data:

  • accessing entries from a table currently copies the entries. Instead it could be possible to access the data directly through a pointer, although there might be performance penalties due to unaligned access
  • file content are currently always copied to a buffer owned by the SqshFileContext. If the files are uncompressed in the archive they could just directly point to the Mapping or if compressed to the decompressed buffer. #8
  • Same for fragment. If the file only consists of a fragment, just point to the fragment directly instead coping to the buffer.
  • If we hand over a block to the decompressor, it is always a continues buffer in memory. If the memory region stretches multiple blocks, they are stitched together in a SqshBuffer. This is still needed for uncompressed blocks, but the decompressors implement streaming APIs, so we can hand them multiple slices without stitching them together. #5

Rename TreeWalker to PathResolver

In #124 it came up, that TreeWalker could be mistaken for a recursive visitor for all files/directory. While it has some similarities, TreeWalker works in a non-recursive manner which leads to some unexpected side effects, see #124.

To mitigate this issue, let's consider renaming the module to something like PathResolver.

Fails to build on macos

Describe the bug

Build fails on macos because #include <endian.h>

To Reproduce

meson setup \
  -Ddefault_library=static \
  -Dexamples=true \
  -Db_lundef=false \
  -Dtest=false \
  -Ddoc=true \
  -Dfuzzer_timeout=10 \
  -Dcurl=enabled \
  -Dzlib=enabled \
  -Dlz4=enabled \
  -Dlzma=enabled \
  -Dzstd=enabled \
  -Db_coverage=true \
  -Db_sanitize=address,undefined \
  -Dwerror=true \
  "build" && ninja -C build all
Full output
The Meson build system
Version: 1.2.1
Source dir: /Users/zach/Development/tmp/libsqsh
Build dir: /Users/zach/Development/tmp/libsqsh/build
Build type: native build
Project name: libsqsh
Project version: 1.0.0
C compiler for the host machine: sccache cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
C linker for the host machine: cc ld64 857.1
Host machine cpu family: aarch64
Host machine cpu: aarch64
Run-time dependency threads found: YES
Found pkg-config: /opt/homebrew/bin/pkg-config (0.29.2)
Run-time dependency libcurl found: YES 7.79.1
Found CMake: /opt/homebrew/bin/cmake (3.27.4)
Run-time dependency fuse3 found: NO (tried pkgconfig, framework and cmake)
Dependency fuse skipped: feature fuse-old disabled
Run-time dependency liblz4 found: YES 1.9.4
Run-time dependency liblzma found: YES 5.4.4
Run-time dependency zlib found: YES 1.2.11
Run-time dependency libzstd found: YES 1.5.5

Executing subproject cextras 

cextras| Project name: cextras
cextras| Project version: 0.1
cextras| C compiler for the host machine: sccache cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
cextras| C linker for the host machine: cc ld64 857.1
cextras| Dependency threads found: YES unknown (cached)
cextras| Build targets in project: 2
cextras| Subproject cextras finished.

Configuring sqsh-cat.1 using configuration
Configuring sqsh-ls.1 using configuration
Configuring sqsh-stat.1 using configuration
Configuring sqsh-unpack.1 using configuration
Configuring sqsh-xattr.1 using configuration
Program doxygen found: YES (/opt/homebrew/bin/doxygen)
Program git found: YES (/opt/homebrew/bin/git)
Configuring Doxyfile using configuration
Build targets in project: 15

libsqsh 1.0.0

  Subprojects
    cextras        : YES

  User defined options
    default_library: static
    werror         : true
    b_coverage     : true
    b_lundef       : false
    b_sanitize     : address,undefined
    curl           : enabled
    doc            : true
    examples       : true
    fuzzer_timeout : 10
    lz4            : enabled
    lzma           : enabled
    test           : false
    zlib           : enabled
    zstd           : enabled

Found ninja-1.11.1 at /opt/homebrew/bin/ninja
WARNING: Need gcovr or lcov/genhtml to generate any coverage reports
ninja: Entering directory `build'
[1/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_buffer.c.o
[2/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_rc_hash_map.c.o
[3/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/concurrency_future.c.o
[4/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_lru.c.o
[5/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/collection_rc_map.c.o
[6/84] Compiling C object lib/libsqsh.1.dylib.p/reader_reader.c.o
[7/84] Compiling C object lib/libsqsh.1.dylib.p/archive_trailing_context.c.o
[8/84] Compiling C object subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a.p/concurrency_threadpool.c.o
[9/84] Compiling C object lib/libsqsh.1.dylib.p/archive_archive.c.o
[10/84] Compiling C object lib/libsqsh.1.dylib.p/archive_compression_options.c.o
[11/84] Linking static target subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/lib/libcextras.a
[12/84] Compiling C object lib/libsqsh.1.dylib.p/archive_superblock.c.o
[13/84] Compiling C object lib/libsqsh.1.dylib.p/archive_inode_map.c.o
[14/84] Compiling C object lib/libsqsh.1.dylib.p/data_directory_data.c.o
FAILED: lib/libsqsh.1.dylib.p/data_directory_data.c.o 
sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_directory_data.c.o -MF lib/libsqsh.1.dylib.p/data_directory_data.c.o.d -o lib/libsqsh.1.dylib.p/data_directory_data.c.o -c ../lib/data/directory_data.c
../lib/data/directory_data.c:41:11: fatal error: 'endian.h' file not found
#       include <endian.h>
                ^~~~~~~~~~
1 error generated.
[15/84] Compiling C object lib/libsqsh.1.dylib.p/data_fragment_data.c.o
FAILED: lib/libsqsh.1.dylib.p/data_fragment_data.c.o 
sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_fragment_data.c.o -MF lib/libsqsh.1.dylib.p/data_fragment_data.c.o.d -o lib/libsqsh.1.dylib.p/data_fragment_data.c.o -c ../lib/data/fragment_data.c
../lib/data/fragment_data.c:41:11: fatal error: 'endian.h' file not found
#       include <endian.h>
                ^~~~~~~~~~
1 error generated.
[16/84] Compiling C object lib/libsqsh.1.dylib.p/data_compression_options_data.c.o
FAILED: lib/libsqsh.1.dylib.p/data_compression_options_data.c.o 
sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_compression_options_data.c.o -MF lib/libsqsh.1.dylib.p/data_compression_options_data.c.o.d -o lib/libsqsh.1.dylib.p/data_compression_options_data.c.o -c ../lib/data/compression_options_data.c
../lib/data/compression_options_data.c:41:11: fatal error: 'endian.h' file not found
#       include <endian.h>
                ^~~~~~~~~~
1 error generated.
[17/84] Compiling C object lib/libsqsh.1.dylib.p/data_metablock_data.c.o
FAILED: lib/libsqsh.1.dylib.p/data_metablock_data.c.o 
sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_metablock_data.c.o -MF lib/libsqsh.1.dylib.p/data_metablock_data.c.o.d -o lib/libsqsh.1.dylib.p/data_metablock_data.c.o -c ../lib/data/metablock_data.c
../lib/data/metablock_data.c:41:11: fatal error: 'endian.h' file not found
#       include <endian.h>
                ^~~~~~~~~~
1 error generated.
[18/84] Compiling C object lib/libsqsh.1.dylib.p/data_superblock_data.c.o
FAILED: lib/libsqsh.1.dylib.p/data_superblock_data.c.o 
sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_superblock_data.c.o -MF lib/libsqsh.1.dylib.p/data_superblock_data.c.o.d -o lib/libsqsh.1.dylib.p/data_superblock_data.c.o -c ../lib/data/superblock_data.c
../lib/data/superblock_data.c:41:11: fatal error: 'endian.h' file not found
#       include <endian.h>
                ^~~~~~~~~~
1 error generated.
[19/84] Compiling C object lib/libsqsh.1.dylib.p/data_inode_data.c.o
FAILED: lib/libsqsh.1.dylib.p/data_inode_data.c.o 
sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_inode_data.c.o -MF lib/libsqsh.1.dylib.p/data_inode_data.c.o.d -o lib/libsqsh.1.dylib.p/data_inode_data.c.o -c ../lib/data/inode_data.c
../lib/data/inode_data.c:41:11: fatal error: 'endian.h' file not found
#       include <endian.h>
                ^~~~~~~~~~
1 error generated.
[20/84] Compiling C object lib/libsqsh.1.dylib.p/data_xattr_data.c.o
FAILED: lib/libsqsh.1.dylib.p/data_xattr_data.c.o 
sccache cc -Ilib/libsqsh.1.dylib.p -Ilib -I../lib -Isubprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I../subprojects/cextras-887bee6835d5fa8213a8f9ffbdf3a26f119983a3/include -I/opt/homebrew/Cellar/lz4/1.9.4/include -I/opt/homebrew/Cellar/xz/5.4.4/include -I/opt/homebrew/Cellar/zstd/1.5.5/include -fcolor-diagnostics -fsanitize=address,undefined -fno-omit-frame-pointer --coverage -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -std=c11 -O0 -g -DCONFIG_CURL -DCONFIG_ZLIB -DCONFIG_LZ4 -DCONFIG_LZMA -DCONFIG_ZSTD -MD -MQ lib/libsqsh.1.dylib.p/data_xattr_data.c.o -MF lib/libsqsh.1.dylib.p/data_xattr_data.c.o.d -o lib/libsqsh.1.dylib.p/data_xattr_data.c.o -c ../lib/data/xattr_data.c
../lib/data/xattr_data.c:41:11: fatal error: 'endian.h' file not found
#       include <endian.h>
                ^~~~~~~~~~
1 error generated.
[21/84] Compiling C object lib/libsqsh.1.dylib.p/directory_directory_index_iterator.c.o
[22/84] Compiling C object lib/libsqsh.1.dylib.p/directory_directory_iterator.c.o
[23/84] Compiling C object lib/libsqsh.1.dylib.p/easy_directory.c.o
[24/84] Compiling C object lib/libsqsh.1.dylib.p/extract_extract_manager.c.o
[25/84] Compiling C object lib/libsqsh.1.dylib.p/easy_file.c.o
ninja: build stopped: subcommand failed.

Expected behavior

Build successful

Platform (please complete the following information):

  • Operating System: macos
  • Version: Ventura 13.4.1

uname -a output:

Darwin zach-mbp.lan 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000 arm64

Additional context
I was able to hackily limp along by removing the includes, and defining no-ops for le32toh (and the like), and it seemed to succeed, so it seems like it's close to supporting macos.

Deadlock when looking up files with fast directory index.

f8bab56 introduced a deadlock when a directory item is looked up through directory_iterator_index_lookup() in src/iterator/directory_iterator.c. The error must be somewhere in the hsqs_inode_directory_index_iterator_* functions in src/iterator/directory_index_iterator.c

unaligned data access on arm

Describe the bug

libsqsh runs into unaligned data access issues when running on arm.

To Reproduce
Enable usan and run fuzzer

Expected behavior
running on arm no unaligned data accesses should happen.

Platform (please complete the following information):

  • Operating System: Ubuntu
  • Version 22.04

Additional context
none

valgrind reports read to uninitialised memory

when running sqsh-mount:

==31801== Thread 5:
==31801== Syscall param writev(vector[1]) points to uninitialised byte(s)
==31801==    at 0x4B5C6DD: __writev (writev.c:26)
==31801==    by 0x4B5C6DD: writev (writev.c:24)
==31801==    by 0x4869E01: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486D740: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486DD64: fuse_reply_data (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x4861559: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486A8F5: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486CEA2: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x4868378: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x4AE7AF9: start_thread (pthread_create.c:442)
==31801==    by 0x4B679AF: clone (clone.S:100)
==31801==  Address 0x6a0f394 is 112,468 bytes inside a block of size 114,688 alloc'd
==31801==    at 0x48427B5: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==31801==    by 0x486129F: fuse_fs_read_buf (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x48613E1: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486A8F5: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486CEA2: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x4868378: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x4AE7AF9: start_thread (pthread_create.c:442)
==31801==    by 0x4B679AF: clone (clone.S:100)
==31801==  Uninitialised value was created by a heap allocation
==31801==    at 0x48426C5: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==31801==    by 0x408319: sqsh__buffer_add_capacity (buffer.c:62)
==31801==    by 0x40ADD6: sqsh__compression_decompress_to_buffer (compression.c:131)
==31801==    by 0x405164: sqsh_file_read (file_context.c:174)
==31801==    by 0x402ABE: sqshfuse_read (sqsh-mount.c:362)
==31801==    by 0x48612FC: fuse_fs_read_buf (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x48613E1: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486A8F5: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x486CEA2: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x4868378: ??? (in /usr/lib/libfuse3.so.3.13.1)
==31801==    by 0x4AE7AF9: start_thread (pthread_create.c:442)
==31801==    by 0x4B679AF: clone (clone.S:100)

make doc doesn't seem to work

Describe the bug

Trying to make doxygen docs, I found make doc seems to exist, but doesn't work.

To Reproduce

Clone libsqsh, run make doc. It appears to exist as a makefile target, but doesn't work

Full output
[ "" ] && meson wrap update-db || true
[ -d "build" ] && rm -rf "build" || true
CC=gcc meson setup -Ddefault_library=static -Dexamples=true -Db_lundef=false -Dtest=true -Ddoc=internal -Dfuzzer_timeout=10 -Dcurl=enabled -Dzlib=enabled -Dlz4=enabled -Dlzma=enabled -Dzstd=enabled -Dfuse=enabled -Dfuse-old=enabled -Db_coverage=true -Db_sanitize=address,undefined -Dwerror=true "build"
The Meson build system
Version: 1.2.1
Source dir: /home/zach/Development/tmp/libsqsh
Build dir: /home/zach/Development/tmp/libsqsh/build
Build type: native build
Project name: libsqsh
Project version: 1.0.0
C compiler for the host machine: gcc (gcc 11.4.0 "gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: gcc ld.bfd 2.38
Host machine cpu family: aarch64
Host machine cpu: aarch64
Run-time dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29.2)
Run-time dependency libcurl found: YES 7.81.0
Run-time dependency fuse3 found: YES 3.10.5
Run-time dependency fuse found: YES 2.9.9
Run-time dependency liblz4 found: YES 1.9.3
Run-time dependency liblzma found: YES 5.2.5
Run-time dependency zlib found: YES 1.2.11
Run-time dependency libzstd found: YES 1.4.8

Executing subproject cextras

cextras| Project name: cextras
cextras| Project version: 0.1
cextras| C compiler for the host machine: gcc (gcc 11.4.0 "gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
cextras| C linker for the host machine: gcc ld.bfd 2.38
cextras| Dependency threads found: YES unknown (cached)
cextras| Build targets in project: 2
cextras| Subproject cextras finished.

Configuring sqsh-cat.1 using configuration
Configuring sqsh-ls.1 using configuration
Configuring sqsh-stat.1 using configuration
Configuring sqsh-unpack.1 using configuration
Configuring sqsh-xattr.1 using configuration
Configuring sqshfs.1 using configuration
C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C++ linker for the host machine: c++ ld.bfd 2.38
Program mksquashfs found: YES (/usr/bin/mksquashfs)
Program doxygen found: YES (/usr/bin/doxygen)
Program git found: YES (/usr/bin/git)
Configuring Doxyfile using configuration
Build targets in project: 50

libsqsh 1.0.0

  Subprojects
    cextras        : YES

  User defined options
    default_library: static
    werror         : true
    b_coverage     : true
    b_lundef       : false
    b_sanitize     : address,undefined
    curl           : enabled
    doc            : internal
    examples       : true
    fuse           : enabled
    fuse-old       : enabled
    fuzzer_timeout : 10
    lz4            : enabled
    lzma           : enabled
    test           : true
    zlib           : enabled
    zstd           : enabled

Found ninja-1.10.1 at /usr/bin/ninja
Ubuntu LLVM version 14.0.0

  Optimized build.
  Default target: aarch64-unknown-linux-gnu
  Host CPU: neoverse-n1
WARNING: Need gcovr or lcov/genhtml to generate any coverage reports
ninja -C build doc
ninja: Entering directory `build'
ninja: error: unknown target 'doc', did you mean 'all'?
make: *** [Makefile:66: doc] Error 1

Expected behavior

Expected documentation to be generated successfully

Platform (please complete the following information):

  • Operating System: Ubuntu
  • Version 22.04

uname -a output:

Linux arm-oracle-vm 5.15.0-1040-oracle #46-Ubuntu SMP Fri Jul 14 21:47:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Additional context

I'm not very familiar with meson, so I'm not super sure how I should be running the commands to generate docs other than the makefile

Implement chunked compression

Currently a compressed block needs to be continuous in memory to be decompressed. if an compressed chunk spans between two memory mapped blocks, libsqsh currently stitches the blocks together in order to decompress them.

Most compression APIs support an other approach where they internally do the buffering. If we would rely on that, we could do a zero copy approach for all data inside of libsqsh.

implementing this solves #2.

Error with sqsh-mount

Describe the bug

I encountered an error while recompressing an archive through sqsh-mount.

To Reproduce

mksquashfs ~/src/k0s /tmp/http/k0s.squashfs  -comp gzip -all-root
build_dir/tools/sqsh-mount /tmp/http/k0s.squashfs /tmp/mnt -f -s
...
mksquashfs /tmp/mnt /tmp/http/k0s2.squashfs  -comp gzip -all-root
...
open flags: 0x8000 /.git_1/hooks/fsmonitor-watchman.sample
   open[0] flags: 0x8000 /.git_1/hooks/fsmonitor-watchman.sample
   unique: 5136, success, outsize: 32
unique: 5138, opcode: LISTXATTR (23), nodeid: 54, insize: 48, pid: 21935
listxattr /.git_1/hooks/commit-msg.sample 0
   unique: 5138, success, outsize: 24
unique: 5140, opcode: READ (15), nodeid: 55, insize: 80, pid: 21936
read[0] 8192 bytes from 0 flags: 0x8000
   unique: 5140, error: -22 (Invalid argument), outsize: 16
unique: 5142, opcode: READ (15), nodeid: 55, insize: 80, pid: 21936
read[0] 4096 bytes from 0 flags: 0x8000
   read[0] 4655 bytes from 0
fuse: read too many bytes
   unique: 5142, success, outsize: 4671
fuse: writing device: Invalid argument
double free or corruption (!prev)

Expected behavior

The resulting squashfs image should have the same exact checksum.

Platform (please complete the following information):

  • Operating System: VoidLinux
  • Version 2023-03-17

curl_mapper: first request always loads at offset 0, even if archive_offset is set

Describe the bug
To determine the size of the archive, curl_mapper loads the first page and returns it, when it is mapped. Unfortunately that means that if archive_offset is set, libsqsh will never request this mapping.

To Reproduce
TODO

Expected behavior
TODO

Platform (please complete the following information):
TODO

Additional context
TODO

Cannot extract data from squashfs file with offset if data is in a symlinked file

Describe the bug
Cannot extract data from squashfs file with offset if data is in a symlinked file

To Reproduce

This file contains a .DirIcon which is a symlink. The squashfs offset is 193728.

Expected behavior
The data of the .DirIcon gets extracted,

Platform (please complete the following information):

  • Operating System: FreeBSD
  • Version 13.1

Additional context
I am using code similar to https://github.com/Gottox/libsqsh/blob/main/examples/read_file_ll.c.

If this is not possible with ll, then either I need to know how to (recursively) resolve symlinks on my own, or it should become possible to use an offset in the high level version.

replace `metablock_stream_context` by a double buffer implementation

Currently directory listings and xattr traversals use SqshMetablockStreamContext to get a continuous view into a sequence of metablock. The reason why this continuous view is needed is that objects inside of a metablock are allowed cross block boundaries. So the end of one metablock could contain the first half of a data object that will continue in the next block. This behavior is documented at 2.2. Packing Metadata of @Dr-Emann's excellent squashfs documentation:

Individual entries are allowed to cross the block boundary, so e.g. an inode may be located at the end of a metadata block with some part of it located at the start of the next block. Both have to be read and decompressed when reading this inode. If an entry is written across block boundaries, there MUST NOT be any gap between the compressed metadata blocks on-disk.

SqshMetablockStreamContext simply reserves more memory everytime a new block is added. This is not actually needed as there's a maximum of two sequential blocks that read used at the same time.

The idea is to have two buffer of uncompressed datablock right next to eachother in memory (2*4096byte) and alternating writing/reading between them. In case an object crosses boundaries, it'll be either already in continuous memory, or if it'll be at the end of double buffer can simply be memcpy'ed to the beginning of the passive buffer and served from there.

As this code path is used in directory tree traversals and therefore is called recursively, it might be a good idea to store the buffers on the heap to reduce stack pressure.

The downside of this algorithm is, that it might create some new error cases when a field spans more than two blocks. One candidate is the xattr values that can be as big as 64kb (according to the xattr manpage). For this cases, the implementation needs to take care to never read more than one full datablock (4096 bytes).

Another case is the link target, that may be limited to 4096 bytes, but the size of this string is saved in an uint32_t and therefore could be bigger if the archive is programmatically built. For such cases extra error handling needs to be in place.

There are multiple other cases that need to handled on a higher level then. It's a good idea to search for the []-string in the squash-docs to see where variable sized arrays are used in metablocks.

Implement generic reader

While developing the readers it turns out, they are mostly c&p from each other. Let's try out a generic implementation.

*_reader: Only recreate the buffer if necessary

Currently all *_reader implementations clean the current state and create the next state. This adds redundant memcpys for every *_advance call. The readers should only recreate their state if they actually cross segment boundaries.

  • ./mapper/map_reader.c
  • ./file/file_reader.c
  • ./metablock/metablock_reader.c

Downgrade build system requirements to allow building on Ubuntu 20.04

Is your feature request related to a problem? Please describe.
I frequently build binaries that are supposed to run on all still-supported LTS releases of Ubuntu.
Currently, libsqsh can only be built on 22.04 but not on 20.04, due to it needing a newer meson.

Describe the solution you'd like
It would be nice if build system requirements could be downgraded to allow building on Ubuntu 20.04.

Describe alternatives you've considered
Get a newer meson (and whatnot) onto Ubuntu 20.04, but it is not a great solution because it adds work for every such distribution (e.g., CentOS)

Additional context
I consider building binaries on the oldest still-supported LTS releases of Ubuntu best practice, so that the binaries can run on more target systems and not only on the latest and greatest ones.

Compile to wasm

This libary can be useful on the web too. Investigate how we could compile it to wasm.

Problems so far identified:

  • in order to be useful, libsqsh needs at least one compression algorithm. There are two ways to do that:

No good way to tell when TreeWalker is at the end of the directory

Describe the bug

sqsh_tree_walker_next returns 0 when it reaches the end of the directory.

I believe the best way currently to tell if we're at the end is to save off the last name, and compare to the name after calling sqsh_tree_walker_next, or open a file from the tree walker, and compare e.g. inode or inode_ref.

Maybe exposing a sqsh_tree_walker_inode_ref to get the inode ref of the current item would at least doing that sort of check more efficient?

To Reproduce

Call sqsh_tree_walker_next on a new walker repeatedly, and observe the return value is always 0.

Expected behavior

There should be an easier way to tell if the walker is not actually advancing to the next entry because there are none left.

clang reports `_Bool` to be a c99 feature in ansi test

libsqsh aims to support ansi compatible public headers. Unfortunately it makes use of the bool type which is c99.

../include/sqsh_xattr.h:100:1: warning: '_Bool' is a C99 extension [-Wc99-extensions]
bool sqsh_xattr_iterator_is_indirect(struct SqshXattrIterator *iterator);

Options:

  • Make c99 the new baseline
  • Remove the bool type.

Make `sqsh_tree_walker_up` position the walker at the entry of the previous directory

Is your feature request related to a problem? Please describe.

I want to be able to use a walker to navigate to every file in the archive, recursively.

I'd like to use something like:

Code Example
void
visit_all_entries(struct SqshTreeWalker *walker) {
    int rc = 0;
    while (sqsh_tree_walker_next2(walker, &rc)) {
        char *name = sqsh_tree_walker_name_dup(walker);
        printf("Visiting %s\n", name);
        free(name);
        if (sqsh_tree_walker_type(walker) == SQSH_FILE_TYPE_DIRECTORY) {
            printf("BEGIN DIR\n");
            rc = sqsh_tree_walker_down(walker);
            assert(rc == 0);
            visit_all_entries(walker);
            rc = sqsh_tree_walker_up(walker);
            assert(rc == 0);
            printf("END DIR\n");
        }
    }
    assert(rc == 0);
}

However, this doesn't work, because sqsh_tree_walker_up sets the walker to be at the start of the parent directory, not at the entry of the inner directory it was just in.

I'm not sure if such a thing is possible to do efficiently.

Describe the solution you'd like

Ideally, it would be better to have sqsh_tree_walker_up would return to the entry of the directory just left rather than the beginning of the parent directory.

Describe alternatives you've considered

It is probably possible to save the name, and use sqsh_tree_walker_lookup to get back to the directory. But presumably that will inefficient.

Alternatively, it might require keeping our own stack of SqshDirectoryIterators to do such visitation of files.

Additional context

reduce memory copies

Currently libhsqs heavily relies on memcpy's, even if they aren't neccessary. For example, HsqsBuffer always copies its contents, even if it maps to a single metablock of uncompressed content.

Also, uncompressed ranges of datablocks can be handed over to the user without and mutation as they don't contain any headers at all.

This is less memory safe, as memory canaries aren't triggered on overflows as they are just pointers into the mmapped archive.

metablock: add test for skipping

Currently skipping metablocks in metablock_iterator is not covered by the unit tests. This could be done in both, the iterator and the reader.

image

Write support for squashfs archives

Is your feature request related to a problem? Please describe.

libsqsh was intended to be read only. There are already libraries and tools out there that do a pretty good job in creating squashfs images. The reason why libsqsh was even started was, that there is no decent non-GPL implementation which is a showstopper for many use cases. Now that the reading part is in decent shape we're facing new issues: Creating new test cases is often clumsy and requires deep knowledge of the squashfs file system. Having a way to tailor custom squashfs archives for test cases would be awesome.

Describe the solution you'd like

  • have an independent builder API to create squashfs images and write them to a FILE* (seekable) pointer.
  • The code sharing between the reader and the writer API should be minimal. There are certain things that could be reused (like the SqshMapper APIs and the on-disk data structures) but there are certain runtime characteristics that benefit from the fact, that most of the data in libsqsh are immutable.

Describe alternatives you've considered

Not do it at all and rely on squashfs-tools-ng to generate test cases on the fly.

Implement caching of decompression

Like we do with the archive mapping, we need some kind of caching system. In contrast to the archive mapping we can use blocksizes, as the data is byte aligned. We need instead to safe the memory locations as a key of a hashmap

  • implement a hash map with uint64_t as a key type
  • Implement a compression_manager similar to the one in the file mapping
  • In contrast to the file mapping, It makes more sense to have specific CompressionManagers in the different types. So SqshTable will get its own, SqshFileContext too, ...

only spawn lzo-helper as needed

Currently we spawn one lzo-helper per CPU at the first request. This is not needed most of the time. Especially when using the helper in a single threaded environment, there's at most one worker active. So we should only spawn new workers when all otheres are blocked.

Make `SqshRcHashMap` extendable

Is your feature request related to a problem? Please describe.
Currently SqshRcHashMap needs to knowy an upper limit of the elements in the HashMap. This is not always possible to achieve and even when, it might introduce a lot of unused memory.

Describe the solution you'd like
In contrast to other hashmap implementations, in libsqsh there might be pointers directly into the HashMap, so relocation/rebalancing won't work. Instead the hash map implementation should consist of several internal hash maps, that are created on demand.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Need more docs on sqsh_file_iterator_skip

Is your feature request related to a problem? Please describe.

The docs aren't clear enough for me to understand the expected use of sqsh_file_iterator_skip.

  1. What is the offset relative to? The beginning of the current data, the end of the current data, the beginning of the archive?
  2. The offset parameter is documented as @param[in] offset, but it is taken as sqsh_index_t *offset, and does appear to be written to. Need to document what will be in *offset after calling the function.
  3. Is sqsh_index_t (size_t) the right type? It seems like an offset into a file could be much larger than a size_t on a 32 bit system.

Describe the solution you'd like

Docs on the above questions.

Support OpenBSD

OpenBSD is a good baseline for supporting other platforms. There are currently two issues:

  • The library does not compile due to multithreading is unable to detect the amount of processors.
  • sqshfs does not compile. OpenBSD only supports fuse2. sqshfs is developed for fuse3. Backport it to fuse2?

sqsh_inode.h does not get installed

Describe the bug
sqsh_inode.h does not get installed

To Reproduce
meson install

Expected behavior
All headers get installed

Platform (please complete the following information):

  • Operating System: helloSystem
  • Version: FreeBSD 13.1 based

Additional context

Insatllation does:

Installing lib/libsqsh.so.0.5.0 to /usr/local/lib
Installing lib/libsqsh.a to /usr/local/lib
Installing tools/sqsh-cat to /usr/local/bin
Installing tools/sqsh-ls to /usr/local/bin
Installing tools/sqsh-stat to /usr/local/bin
Installing tools/sqsh-xattr to /usr/local/bin
Installing tools/sqsh-unpack to /usr/local/bin
Installing tools/sqshfs to /usr/local/bin
Installing /tmp/libsqsh/include/sqsh.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_archive.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_chrome.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_common.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_data_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_directory.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_directory_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_error.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_extract_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_file.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_file_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_mapper.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_mapper_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_metablock_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_primitive_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_table.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_table_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_thread_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_xattr.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/include/sqsh_xattr_private.h to /usr/local/include/sqsh/
Installing /tmp/libsqsh/build/man/sqsh-cat.1 to /usr/local/share/man/man1
Installing /tmp/libsqsh/build/man/sqsh-ls.1 to /usr/local/share/man/man1
Installing /tmp/libsqsh/build/man/sqsh-stat.1 to /usr/local/share/man/man1
Installing /tmp/libsqsh/build/man/sqsh-unpack.1 to /usr/local/share/man/man1
Installing /tmp/libsqsh/build/man/sqsh-xattr.1 to /usr/local/share/man/man1
Installing /tmp/libsqsh/build/man/sqshfs.1 to /usr/local/share/man/man1
Installing /tmp/libsqsh/build/meson-private/libsqsh.pc to /usr/local/libdata/pkgconfig
Installing symlink pointing to libsqsh.so.0.5.0 to /usr/local/lib/libsqsh.so.0
Installing symlink pointing to libsqsh.so.0 to /usr/local/lib/libsqsh.so

Manually running sudo cp ../include/* /usr/local/include/sqsh from the build directory does improve the situation.

iterators: _skip function should take the offset in bytes, not in blocks

Is your feature request related to a problem? Please describe.

Currently block iterators have by contract two functions to forward the block window: _skip() and _next(). The _next() function always returns the next block in the iterator, whereas _skip can skip multiple _next() calls.

The idea of _skip() is to allow optimizations for skipped blocks, for example not to map blocks into memory that are skipped anyway. Unfortunately this may introduce a constraint to _next that it always need map the same data-size in one iteration, otherwise the amount of skipped blocks can't be easily calculated.

Currently we have two cases for _skip: There are iterator that just call _next until the desired amount of blocks has been skipped (like SqshFileIterator) and there are ones, that have optimisations and predict the mapped area directly (like SqshMapIterator). The first class of implementations are allowed to vary their block size while iterating, for the second class, it'll introduce misbehavior.

Describe the solution you'd like

To harmonize the implementations and allow skipping optimisation for all implementations, _skip should take the amount of bytes to be skipped instead of the amount of blocks and skip forward to that block where the requested offset can be found. It's up to the caller to determine the offset inside of the block if it doesn't align with the block itself.

This way we allow _skip optimisation for both classes of iterators and make the API more consistent.

Describe alternatives you've considered

none

Additional context

none

Examples

Is your feature request related to a problem? Please describe.

I'd like to consume libsqsh in a Qt based application to extract files from squashfs to files and/or memory.
For this reason, I'd like to have it wrapped in a C++ class.

Describe the solution you'd like

Something along those lines with a working .cpp implementation

#ifndef SQSH_FSEXTRACTOR_H
#define SQSH_FSEXTRACTOR_H

#include <QString>
#include <QByteArray>
#include <QList>
#include <QDir>

#include "SqshFSInfo.h" // Include the previously defined SqshFSInfo class

class SqshFSExtractor
{
public:
    SqshFSExtractor(const QString &imagePath);
    ~SqshFSExtractor();

    SqshFSInfo getFileInfo(const QString &filePath) const;
    QList<SqshFSInfo> listFilesRecursively() const; // Recursive listing; like find /
    QList<SqshFSInfo> listFiles() const; // Directory listing, like ls /

    bool extractToFile(const QString &filePath, const QString &outputFilePath);
    bool extractRecursively(const QString &sourceDir, const QString &targetDir);
    QByteArray extractToMemory(const QString &filePath);

private:
    // Add any private members or helper functions here as needed
};

#endif // SQSH_FSEXTRACTOR_H

#ifndef SQSH_FSINFO_H
#define SQSH_FSINFO_H

#include <QString>
#include <QVector>

enum class FileType {
    File,
    Directory,
    Symlink,
    Device,
    Other
};

class SqshFSInfo
{
public:
    SqshFSInfo(const QString &name, FileType type, qint64 size = 0);
    ~SqshFSInfo();

    QVector<SqshFSInfo> getChildren() const;
    QString getName() const;
    FileType getType() const;
    qint64 getSize() const;

private:
    QString name;
    FileType type;
    qint64 size;
    QVector<SqshFSInfo> children;
};

#endif // SQSH_FSINFO_H

Additional context

Unfortunately, I am not able to do this with the current documentation.
Do you think it can be done?
Would you be able to present a sample implementation?

Error SQSH_ERROR_NOT_A_FILE is unused

Describe the bug

Error SQSH_ERROR_NOT_A_FILE is never created. It seems like it might be a good error for e.g. sqsh_easy_file_content(archive, "path/to/directory", &err) (which seems to currently error with SQSH_ERROR_OUT_OF_BOUNDS)

use cextras for primitives

Is your feature request related to a problem? Please describe.

cextras is currently used for the test runner of libsqsh. But cextras original purpose was to encapsulate common reusable functionality between projects. Currently almost all of the lib/primitive functionality is already in cextras.

Describe the solution you'd like

Remove the lib/primitive functionality and replace them with their cextras counterparts.

Describe alternatives you've considered

None

Additional context

This will break the buildsystemless approach that libsqsh currently had, so bootstrapping becomes harder. With muon there exists easy to setup buildsystem that could be used as a replacement.

Tests are broken on openbsd

Running tests on OpenBSD is currently non-functional. The issue is, that the setfattr tool cannot be found as openbsd does not support extended attributes. We need a different way to define attributes. OpenBSDs pkg currently ships an old version of squashfs-tools that doesn't support reading xattrs from pseudo files.

chrome API

The API of libsqsh is powerful but quite complex. Adding an API layer that eases the most common operations would solve this problem.

  • path resolver should resolve softlinks too
  • add functionality to resolve a path, open the inode and load the file content
  • add functionality to resolve a path, open the inode and load specific metadata
  • if there's no export table, dynamicly fill the export table as we walk through the file tree

improve code sharing between fs3 and fs2

Currently libsqsh implements two implementations of fuse filesystems. One uses the lowlevel fuse3 one, the other uses the default fuse2 API. The latter is useful for OpenBSD.

Currently those two implementations share a few bits in fs-common.c, but there's more to generalize in these implementations.

Rust frontend

It can be useful to make this library available via Rust. Investigate how to build an API binding.

Support squashfs files that start at an offset

Is your feature request related to a problem? Please describe.

The AppImage file format conists of an ELF to the end of which a squashfs file is appended.

Describe the solution you'd like

I would like to use libsqsh on such squashfs files that don't start at offset 0 but at another offset.
Hence, I am looking for

Additional context

This library could become really, really useful in the context of the AppImage ecosystem (e.g., file managers that want to show the proper icons for AppImages, etc.); for this the offset functionality is essential. Thanks for making this library, for licensing it under a permissive license, and for your support! ๐Ÿ‘

Here is an example file: appimagetool-x86_64.AppImage - the squashfs starts at offset 631200.

References

reiterate on streaming extractor APIs

Is your feature request related to a problem? Please describe.

One of the pieces where libsqsh isn't zero-copy is the uncompression of non-continious blocks. While this is an edge case it could end up in situation where up to 1 MiB get copied just to uncompress them afterwards and unnecessarily cache them.

Describe the solution you'd like

The better solution is to write the partial blocks to the extractor and let it handle internal state. This would reduce the amount of (large) copies inside libsqsh.

Describe alternatives you've considered

None

Additional context

The issue is that the implementation of extractors must be capable to always process incoming data completely.

Replace `SqshFileContext` with `SqshFileReader`

Is your feature request related to a problem? Please describe.
The current SqshFileContext implementation does not follow the Sqsh*Reader API pattern. It feels clumsy to use and it does not make sense to the user why they need to first open a file (internal: the inode of a file) to open then a file (internal: the file contents).

Describe the solution you'd like
With this change, libsqsh follows the common pattern found in many other languages where you open one object and use a second one to read it. It's a bit cheating, as we do not open the file, but the Inode, but this pattern is more understandable by the library user.

Describe alternatives you've considered
none

Additional context
none

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.