Binary File Inputs about vowpal_wabbit

[rajan-chari]

There is a nice CPU efficient input format implemented in VW (Flatbuffers)

Command line example:
vw --cb_force_legacy --cb 2 -d train-sets/rcv1_raw_cb_small.fb --flatbuffer

Unfortunately, this feature is off by default. You can turn it on when you build VW.

vowpal_wabbit/CMakeLists.txt

Line 53 in de52303

if(VW_FEAT_FLATBUFFERS)

Here is the schema:
https://github.com/VowpalWabbit/vowpal_wabbit/blob/master/vowpalwabbit/fb_parser/schema/example.fbs

Helpful reference. Converting to flatbuffers
https://github.com/VowpalWabbit/vowpal_wabbit/blob/master/test/runtests_flatbuffer_converter.py

There are extensive tests for this feature:

vowpal_wabbit/test/core.vwtest.json

Line 3100 in de52303

"vw_command": "--cb_force_legacy --cb 2 -d train-sets/rcv1_raw_cb_small.fb --flatbuffer",

If you feel you can contribute some time to the projects, we welcome your involvement. The following link is a PR that's almost ready to go. It defines a more compact Flatbuffer format with similar performance. I am happy to guide you through it if you would like to push forward with it. C++ experience would be very helpful.
https://github.com/VowpalWabbit/vowpal_wabbit/pulls?q=is%3Apr+is%3Aclosed+flatbuffer

Please note that the input format in the PR above is where we want to eventually end up.

[rajan-chari]

Feel free to reach out to me if you would like to push this item forward for the benefit of the community.

[olgavrou]

Closing for now, please feel free to re-open