Impossible to disable compression by default about nanolog

[syang0]

Hi digriz33,

It seems like you're looking for two features within NanoLog:

1. Having the log files be in a human-readable format for log-based monitoring to parse line by line and
2. Performing log rotations.

For the first request, while one can disable compression in the library, this will not make the log file human-readable; the NanoLog system will still output a binary log file at runtime. To obtain a human-readable file that's line-delineated, one must use the decompressor to post-process the binary log files.

Outputting the log messages in a human-readable format at runtime would defeat most of the performance benefits of NanoLog, so it's not supported out of the box. Instead, I would recommend you attempt to incorporate the decompressor into your log ingest pipeline.

For the second request, while there are no automatic log rotation mechanisms implemented into NanoLog itself, one can perform a rotation manually by invoking NanoLog::setLogFile(const char* filename) as documented in the NanoLog Header. This will cleanly split the log file such that each split is independently interpretable by the decompressor application.

[syang0]

Hi digriz33,

That is correct, setLogFile() should be thread safe.

For your second question, yes and no. See Issue #10 for more information about the structure of the log file and how you may parse it gradually.

But at a high level, the NanoLog log file consists of three entities: a header, dictionary entries, and bufferExtents (i.e. a chunk log messages from a single thread). The header and dictionary entries are needed to interpret the bufferExtents, so if you had a mechanism of keeping this state readily available, the bufferExtents can be interpreted independently.

What setLogFile() does is it duplicates the header + dictionary entries into the beginning of the new file, and this is what allows the decompressor to read the new log messages.

Hope that helps.

[digriz33]

Thank you very much for extensive feedback!

Am I right that setLogFile call is thread-safe?

Instead, I would recommend you attempt to incorporate the decompressor into your log ingest pipeline.

Does this mean that the logger compresses its entries in binary format, but it's still possible to somehow parse file's content gradually without using decompressor on the whole file?

[digriz33]

Got it!
Thank you very much for your efforts in developing the library and for the help.
Have a nice weekend!