fluent / chunkio

Simple library to manage chunks of data in memory and file system

fluent

chunkio's Issues

Allow for errors to be reported to a file instead of stderr

The Problem

Today, any errors in chunkio file operations will be reported to stderr. This is not a big deal when running an application using this library as a direct binary, since there are methods to redirect stderr when running directly. When running as a systemd service, you can set a StandardError configuration to report stderr to a file. The main situation where there is no recourse for this is when running as a Windows Service, which does not retain stderr and does not have a standard way to reconfigure it.

Potential Solution

Allow configuring a log file for any chunkio logs.

Alternatives Considered

Since the problem really shines as a Windows Service, this could be solved on Windows specifically by using the SetStdHandle function in the Win32 Console API. However, with all the work that would need to be done to allow configuration of this file just for Windows, you might as well do the implementation in a cross platform manner that may still benefit users on other platforms even if they have other escape hatches.

Additional Context

This issue was discussed in person at the Fluent Bit Developer's Meeting on July 12,2023.

There is a wrong target name in shared build.

I met a compilation error on shared build.

CMake Error at src/CMakeLists.txt:44 (target_link_libraries):
  Cannot specify link libraries for target "chunkio-static" which is not
  built by this project

It seems to be caused by wrong target name on shared build.
https://github.com/fluent/chunkio/blob/v1.1.6/src/CMakeLists.txt#L44

I think it should be target_link_libraries(chunkio-shared ${libs}).

Allow for more file system support by making a choice to use posix_fallocate instead of fallocate

In the chunkio code there are system calls to expand files by using the system call "fallocate" that limits usage to certain file system types, this is mainly due to performance reasons.

To support more file systems apart from XFS, ext4 etc, it would be good if there could be an option to use posix_fallocate and let the decision on performance be up to usage and configurable.

Proposal: Let's do versioning

This is just a casual proposal that (I think) would help the development.

Backgrond

We use chunkio as a library in Fluen Bit. Right now the workflow is that
(1) we do some development on chunkio and then (2) occationally sync the
result into fluent/fluent-bit.

The synchronization is done by manually copying files and adding a commit
like below:

commit 0e000ca45cd70731896008d1cf1a34c3caf5a88f
Date:   Thu Sep 26 11:59:29 2019 -0600

    lib: chunkio: sync changes, details below:
    
    - validate if fallocate is supported
    - new trace mode
    - new api call to retrieve metadata size
    - fix crash when opening a corrupted file
    - new unit test issue_51

But it's kinda opaque from the commit which version of chunkio is in use.
We need to look into the history and figure out is which is corresponding
to which.

If there is a versioning, we can simply say:

commit 0e000ca45cd70731896008d1cf1a34c3caf5a88f
Date:   Thu Sep 26 11:59:29 2019 -0600

    lib: chunkio: update version from v1.0 to v1.1

In my opinion, this is much clearer. The commit message clearly tells us
that the synchronization is done in a well-defined point.

Proposal

Let's call 4f197e0, say, v1.0.0 and do versioning thereafter.

improve error messages with more context

There are some error messages that could be more helpful with more context:
No space left on device => No space left on device. Space requested {} space available {} device/path {}
No such file or directory => No such file or directory. Path {}

config file option for dir/file permissions

All elements created by chunkio.c:check_root_path and cio_stream.c:check_stream_path are created with hard-coded mode 0755. Therefore anyone can read the data; it is not confidential.

Provide a config file option allowing a user-configurable permission to be defined, eg 0700.

#include <fts.h> not supported in UClibc

While porting fluent-bit (which uses chunkio), I found out that it couldn't be compiled against UClibc.
For now, I've added a guard to enforce glibc: https://github.com/buildroot/buildroot/blob/master/package/fluent-bit/Config.in#L3.

It would be nice to have UClibc and e.g. also Musl support. Then, fluent-bit can be used on all supported platforms on buildroot.

[ 16%] Building C object lib/monkey/mk_server/CMakeFiles/monkey-core-static.dir/mk_vhost.c.o
/home/thomas/br-test-pkg/bootlin-armv5-uclibc/build/fluent-bit-2.0.8/lib/chunkio/src/cio_utils.c:29:10: fatal error: fts.h: No such file or directory
   29 | #include <fts.h>
      |          ^~~~~~~
compilation terminated.
make[4]: *** [lib/chunkio/src/CMakeFiles/chunkio-static.dir/build.make:174: lib/chunkio/src/CMakeFiles/chunkio-static.dir/cio_utils.c.o] Error 1
make[4]: *** Waiting for unfinished jobs....