Question: is there a ways to determine byte size of the packed message without actually decoding it? about msgpack-lite

[royaltm]

Yes exactly. The dryrun would do the trick. Is there any chance you would consider this feature?

[kawanet]

@royaltm

The byte length of a msgpack item is a variable. Detecting number of items or byte lengths would take almost same seconds to decode the msgpack stream. What is an use case which needs to determine boudaries of the msgpack or count sequenced msgpacked items?

---

// hypothetical api: encode(data[, buffer[, offset[, maxLength]]])
encode("some data", new Buffer(100), 10, 50);

I could guess an use case for above. It may allow less memory copy.
Here is msgpack-lite's Endoder object which is not well documented.
This allows a similar thing as below:

var msgpack = require("msgpack-lite");
var e = new msgpack.Encoder();
e.on("data", console.log); // debug only
var buf = e.buffer = new Buffer(100);
var start = e.offset = 10;
e.encode("some data");
var length = e.offset - start;
msgpack.decode(buf.slice(10, e.offset)); // debug only

It requests enough length of buffer.

[royaltm]

Yes I know that detecting the size would require to recursively iterate over maps and arrays but decoding data means also allocating Objects and later gc'ing them. I need to scan over large stream of msgpacked objects only cherrypicking n'th of them. For now I can just throw away whatever items I don't need but it's an obvious waste.

Thanks for the hint regarding 2nd question. This is a little bit verbose, but that will do :).

[kawanet]

decoding data means also allocating Objects and later gc'ing them.

Right.

I need to scan over large stream of msgpacked objects only cherrypicking n'th of them.

It would need a kind of dryrun feature which returns the byte length instead of the object decoded.

function getNth(source, n, callback) {
  var dry = msgpack.createCodec({dryrun: true}); // proposal
  var wet = msgpack.createCodec({dryrun: false}); // normal behavior codec
  var codec = n ? dry : wet; // first codec
  var decoder = msgpack.createDecodeStream({codec: codec});
  var cnt = 0;
  source.pipe(decoder).on("data", function (data) {
    if (cnt === n) callback(null, data);
    decoder.codec = (cnt === n-1) ? wet : dry; // next codec
    cnt++;
  }).on("error", callback);
}