brycebaril / multibuffer

Package Arrays of Buffers into a single buffer that they can be later unpacked from.

multibuffer

Package Arrays of Buffers into a single buffer that they can be later unpacked from.

One place this library can be useful is if you want to stream tuples of Buffer data without entering an objectMode stream.

Each input buffer is prefixed with a varint prefix encoding how long the buffer is. The resulting buffer is the same length as all the concatenated buffers, plus a few bytes per buffer for the encodings.

Because encoding is at the beginning and varints can be consumed without back-tracking, this encoding is safe to nest upon itself.

var multibuffer = require("multibuffer")

var input = [
  new Buffer("Hi there"),
  new Buffer("BYE NOW!!!!!!!!!")
]

/*
[ <Buffer 48 69 20 74 68 65 72 65>,
  <Buffer 42 59 45 20 4e 4f 57 21 21 21 21 21 21 21 21 21> ]
 */

var packed = multibuffer.pack(input)

/*
<Buffer 08 48 69 20 74 68 65 72 65 10 42 59 45 20 4e 4f 57 21 21 21 21 21 21 21 21 21>
 */

var unpacked = multibuffer.unpack(packed)
/*
[ <Buffer 48 69 20 74 68 65 72 65>,
  <Buffer 42 59 45 20 4e 4f 57 21 21 21 21 21 21 21 21 21> ]
 */

API

.pack(buffers, extra)

Pack the Array[Buffer] buffers into a single encoded Buffer. extra is an optional integer specifying how many leading empty bytes to leave in the returned Buffer.

.unpack(multibuffer)

Unpack the Array[Buffer] buffers that were encoded into the multibuffer.

.encode(buffer, extra)

Encode a single buffer. extra is an optional integer specifying how many leading empty bytes to leave in the returned Buffer.

.readPartial(multibuffer)

Attempt to read the first encoded buffer from a multibuffer. Will return a two-element array of [Buffer, Buffer] which is [firstBuffer, rest]. If the multibuffer is incomplete, it will return [null, multibuffer] where the second element is the passed incomplete multibuffer.

NOTES

This library currently only supports packing buffers that are each a maximum of 4294967295 octets. My guess is you'll have memory issues before this is a bottleneck.

LICENSE

MIT

multibuffer's Issues

Feature: 16 bit version

For lots of small multibuffers, 32 bits is too wide, and can vastly bloat the total data size.

I'm thinking about exposing a 16 bit interface as well, which will use 2 byte prefixes for the buffers vs. 4 bytes, thus possibly saving a lot of total size. The downside being :

1. a smaller maximum encoded buffer length (64kB)
2. two possible multibuffer encodings

One possible idea would be to add a prefix to the start of the multibuffer with the encoding width, though there would be backwards compatability issues, and it could complicate the streamed model.

Most likely at this point it would be up to the user to know which flavor multibuffer the data was stored in when unpacking it.