Transmit Binary Information over UDP! about nanolog

[syang0]

Hello Balaji,

I think the answer depends on what you mean by a "truncated'' file. If you mean that the file has all bytes present and in order except for the very tail, then the decompressor should be able to interpret most of the log and then error out towards the very end.

If "truncated" means that bytes are missing in the middle (like say form a UDP packet drop), then you need to make sure that at least the NanoLog "dictionary" is sent in full to your remote location and that the log data present is aligned with the start a "BufferExtent". Once you have those two requirements, you should be able to hack the NanoLog Decoder accept the two separately and interpret the binary stream. Below is some information that may be helpful in your endeavor.

Good Luck.

Log File Structure and The Dictionary

How the log file looks and how you go about obtaining the dictionary are different depending on whether you’re working with Preprocessor NanoLog or C++17 NanoLog.

For Preprocessor NanoLog, the log file looks similar to Figure 4 in the NanoLog Paper. The log file contains a header, the dictionary for the entire application, and “Buffer Extents” which are groupings of log messages. The entire dictionary is built statically at compile-time and can be obtained by invoking the insertCheckpoint() function in runtime/Log.cc:118.

For C++17 NanoLog, the dictionary is built up piece-wise as the application runs. The format again resembles Figure 4 from the NanoLog Paper, except the “Dictionary” segment can appear in-between Buffer Extents, and each segment contains a fragment of the full dictionary. For your purposes, if you are missing dictionary entries, then you need to run logic similar to what’s found in runtime/RuntimeLogger.cc:393 to build it. And again, the dictionary is built up incrementally in C++17 NanoLog, so you will need to keep updating your remote dictionary as log messages are encountered during runtime. This is different from Preprocessor NanoLog which has the complete dictionary at application start.

Building Buffer Extents

To make sure log messages aren't cut randomly, I'd recommend you send only complete BufferExtents in UDP packets. The BufferExtents are built in this function. You can change the logic to build smaller extents that fit within the MTU of a UDP packet.

Of course, all this means you’ll also need to change the Decoder to accept the dictionary and BufferExtents separately, but it should be doable. Good Luck.

[bkannadassan]

Thanks that helps. Will check the same. I am using the preprocessor version were use python parser. Btw I see C++17 a must, is it possible to get it working with C++11 ?.

[syang0]

The Preprocessor version of NanoLog will work with C++11.

The C++17 version of NanoLog, as the name implies, will need something newer.

[bkannadassan]

Thanks
root@localhost ~/nanolog/NanoLog/sample_preprocessor (master) $ git diff

diff --git a/NanoLogMakeFrag b/NanoLogMakeFrag
index 404103b..cdaa5cc 100644
--- a/NanoLogMakeFrag
+++ b/NanoLogMakeFrag
@@ -39,7 +39,7 @@ COMWARNS := -Wall -Wformat=2 -Wextra \
 CWARNS   := $(COMWARNS) -Wmissing-prototypes -Wmissing-declarations -Wshadow \
                -Wbad-function-cast
 CXXWARNS := $(COMWARNS) -Wno-non-template-friend -Woverloaded-virtual \
-               -Wcast-qual -Wcast-align -Wconversion -Weffc++
+               -Wcast-qual -Wcast-align -Weffc++
 
 .PHONY: all
 all:
diff --git a/sample_preprocessor/GNUmakefile b/sample_preprocessor/GNUmakefile
index f8aaf2a..3b9633a 100644
--- a/sample_preprocessor/GNUmakefile
+++ b/sample_preprocessor/GNUmakefile
@@ -41,7 +41,7 @@ include $(NANOLOG_DIR)/NanoLogMakeFrag
 # User Section
 ####
 # -DNDEBUG and -O3 should always be passed for high performance
-CXXFLAGS= -std=c++17 -DNDEBUG -O3 -g
+CXXFLAGS= -std=c++11 -DNDEBUG -O3 -g
 CXX=g++

I did above two changes and got it working..