Comments (4)
Yes exactly. The dryrun
would do the trick. Is there any chance you would consider this feature?
from msgpack-lite.
The byte length of a msgpack item is a variable. Detecting number of items or byte lengths would take almost same seconds to decode the msgpack stream. What is an use case which needs to determine boudaries of the msgpack or count sequenced msgpacked items?
// hypothetical api: encode(data[, buffer[, offset[, maxLength]]])
encode("some data", new Buffer(100), 10, 50);
I could guess an use case for above. It may allow less memory copy.
Here is msgpack-lite
's Endoder object which is not well documented.
This allows a similar thing as below:
var msgpack = require("msgpack-lite");
var e = new msgpack.Encoder();
e.on("data", console.log); // debug only
var buf = e.buffer = new Buffer(100);
var start = e.offset = 10;
e.encode("some data");
var length = e.offset - start;
msgpack.decode(buf.slice(10, e.offset)); // debug only
It requests enough length of buffer.
from msgpack-lite.
Yes I know that detecting the size would require to recursively iterate over maps and arrays but decoding data means also allocating Objects and later gc'ing them. I need to scan over large stream of msgpacked objects only cherrypicking n'th of them. For now I can just throw away whatever items I don't need but it's an obvious waste.
Thanks for the hint regarding 2nd question. This is a little bit verbose, but that will do :).
from msgpack-lite.
decoding data means also allocating Objects and later gc'ing them.
Right.
I need to scan over large stream of msgpacked objects only cherrypicking n'th of them.
It would need a kind of dryrun
feature which returns the byte length instead of the object decoded.
function getNth(source, n, callback) {
var dry = msgpack.createCodec({dryrun: true}); // proposal
var wet = msgpack.createCodec({dryrun: false}); // normal behavior codec
var codec = n ? dry : wet; // first codec
var decoder = msgpack.createDecodeStream({codec: codec});
var cnt = 0;
source.pipe(decoder).on("data", function (data) {
if (cnt === n) callback(null, data);
decoder.codec = (cnt === n-1) ? wet : dry; // next codec
cnt++;
}).on("error", callback);
}
from msgpack-lite.
Related Issues (20)
- Require cycle: node_modules/msgpack-lite/lib... HOT 1
- question: how would I en/decode js set type?
- usemaps gives undefined instead
- docs: msgpack-lite is much more efficient for large strings HOT 1
- decode only first item HOT 1
- here is the CDN:
- how to decode 4 digit buffers. HOT 1
- Slow moving, would funding help? HOT 1
- es modules for modern apps
- better error message - BUFFER_SHORTAGE HOT 2
- Is it possible for msgpack-lite to decode data as it is streamed in?
- use native array.isArray instead
- Report amount of decoded data HOT 1
- Discrepancy for binary keys between browser and nodejs
- Compression issues with large Inputs HOT 4
- Unable to serialise maps and UInt8Arrays at the same time HOT 3
- BigInt / BigInt64Array / BigUInt64Array HOT 2
- encoding number types should not use both Int and Float families
- [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead. HOT 2
- Some tests fails when minified by [email protected] HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from msgpack-lite.