hbpvis / zerobuf

zero-copy, zero-serialize, zero-hassle protocol buffers

ZeroBuf

Overview

ZeroBuf implements zero-copy, zero-serialize, zero-hassle protocol buffers. It is a replacement for FlatBuffers, resolving the following shortcomings:

• Direct get and set functionality on the defined data members
• A single memory buffer storing all data members, which is directly serializable
• Usable, random read and write access to the data members
• Zero copy of the data used by the (C++) implementation from and to the network

Features

• Storage of (u)int[8,16,32,64,128]_t, float, double and enum single elements, static and dynamic sub-structures, fixed size and dynamic arrays of static-sized elements
• Access to arrays using raw pointers, iterators, std::array, std::string and std::vector
• Conversion to and from a JSON representation

Extensions to flatbuffers grammar

• Arrays can have an optional fixed size specified as part of the type, e.g., matrix:[float:16] for a 16 value float array
• byte and ubyte data is base64 encoded in JSON, int8_t and uint8_t are represented as value arrays

Building from Source

ZeroBuf is a cross-platform library, designed to run on any modern operating system, including all Unix variants. It requires a C++11 compiler and uses CMake to create a platform-specific build environment. The following platforms and build environments are tested:

• Linux: Ubuntu 16.04, RHEL 6.8 (Makefile, Ninja)
• Mac OS X: 10.9 (Makefile, Ninja)

Building from source is as simple as:

git clone --recursive https://github.com/HBPVIS/ZeroBuf.git
mkdir ZeroBuf/build
cd ZeroBuf/build
cmake -GNinja -DCLONE_SUBPROJECTS=ON ..
ninja

This work has been partially funded by the European Union Seventh Framework Program (FP7/2007­2013) under grant agreement no. 604102 (HBP).

zerobuf's Issues

Qt updated property signaling

• one property in the object called 'update' to signal if the buffer was updated (on ZeroEQ receive for instance)
• emit signals for changed properties after update: compare the buffer contents respectively (code generated) only for those properties

Add support for arrays of dynamics zerobufs

The python fbs generator stumbles upon this:

table DataSource {
  url: string;
}

table Scene {
  data_sources: [DataSource];
}

>>> Dynamic arrays of empty ZeroBuf (field data_sources) not supported

Probably the allocator and the C++ side of things is not ready yet either, but hopefully @eile knows more about this.

Problem with ZeroBuff and [string] type

I'm Cristian Rodriguez, from the group working back in URJC/UPM with Pablo Toharia. Right now I'm trying to use a custom ZeroBuff fbs file that required a std::vector of char* (or vector) but zerobufCxx generates a std::vector when I use string or [string].
My fbs file contains:

namespace zeroeq.gmrv;

table SelectionStr
{
 ids:[string];
}

I've seen in the python script (zerobufCxx) it converts string to zerobuf::Vector< char >.
With the following fbs, I get the same result in the two attributes:

namespace zeroeq.gmrv;

table SelectionStr
{
 fooStrArr:[string];
 fooStr:string;
}

I had been trying to modify your Python script to generate std::vector (string) and std::vector<char*> ([string]), but I prefer to ask you how can I resolve this without change the script.
Any ideas on this?
Thank you!