ssbc / bipf Goto Github PK

In @staltz' FOSDEM2021 talk we had the feedback that the encoding is basically DER, which is also type, length, value.

While my guess is that most DER libraries don't offer a seek to label and decode as type kind of API that bipf offers, it might short cut some documentation needs.

The new version includes bounds checking, which is nice. It would be best to wait for the upgrade until chrisdickinson/varint#24 is merged so we can benefit from the decode performance optimization.

I've playing with sending LoRa packets over SSB, using ssb-db2. I'm getting this error related to the values in the LoRa packet which I'm translating to json as they are, Ints:

Received onTextPacket
NEW TEXT PACKET {
  packet: {
    from: 4066789380,
    to: 4294967295,
    channel: 0,
    payloadVariant: { oneofKind: 'decoded', decoded: [Object] },
    id: 3731283723,
    rxTime: 1635936589,
    rxSnr: 5.75,
    hopLimit: 3,
    wantAck: false,
    priority: 0,
    rxRssi: 0
  },
  data: 'Salve6'
}
Error sending message RangeError [ERR_OUT_OF_RANGE]: The value of "value" is out of range. It must be >= -2147483648 and <= 2147483647. Received 3731283723
    at new NodeError (node:internal/errors:371:5)
    at checkInt (node:internal/buffer:72:11)
    at writeU_Int32LE (node:internal/buffer:692:3)
    at Buffer.writeInt32LE (node:internal/buffer:861:10)
    at Array.Integer (/home/dev/node_modules/bipf/index.js:28:12)
    at encode (/home/dev/node_modules/bipf/index.js:201:24)
    at Array.Object (/home/dev/node_modules/bipf/index.js:47:10)
    at encode (/home/dev/node_modules/bipf/index.js:201:24)
    at Array.Object (/home/dev/node_modules/bipf/index.js:47:10)
    at encode (/home/dev/node_modules/bipf/index.js:201:24) {
  code: 'ERR_OUT_OF_RANGE'
}

Is turning them to strings the best solution?

In Python, it is fine to have bytes (buffer in JS), or integers, as keys for dictionaries (object in JS).

The current implementation in JS, which somehow serves as spec, insists on object keys being strings (lines 120 and 121 of index.js):

// JavaScript only allows string-valued and Symbol keys for objects
if (tag & TAG_MASK) throw new Error('required type:string')

Maybe this is not meant to be a restriction of BIPF - just a JS-specific precaution. Then this GitHub issue is void.

Otherwise, I suggest to not have this restriction being part of the BIPF spec in order to let other languages use BIPF for data struct serialization. The spec should enumerate the allowed key types: string, int, buffer/bytes, bool and even null can make sense in some programming language (while having arrays or objects as keys is more doubtful and needs more thoughts before allowing them in a spec).

In order to let JS handle incoming "buffer keys", a convention could be adopted for JS land, for example that buffer keys are automatically converted to strings, using base64 or similar.

Note: I do not advocate that objects with buffer keys are used in any SSB public data structure. But I want to use BIPF also for serializing internal Python data structures, instead of having to add redundant serialization libraries (CBOR etc). The quite ubiquitious use case in my code are dictionaries (that I persist) whose keys are feed IDs, which I keep in binary format throughout the implementation for space and comparison speed reasons.

returning the favor to @arj03 and providing a test vector for BIPF:

f50418666f6f8c02127fff0a800a810aff0a000a010a7f12800012007f1a0080000e002179656168061862616695014046726564686f6c6d4305413da6832fbc3f186261722868656c6c6f1862617a06

which should turn, in Python speak, into
{'foo': [-129, -128, -127, -1, 0, 1, 127, 128, 32512, 32768, False, b'yeah', None], 'baf': {'Fredholm': 0.1101000100000001}, 'bar': 'hello', 'baz': None}

does this compute?

This is the memory view (key val pairs) when iterating:

> 18666f6f 8c02127fff0a800a810aff0a000a010a7f12800012007f1a0080000e00217965616806
  > 0 127fff
  > 1 0a80
  > 2 0a81
  > 3 0aff
  > 4 0a00
  > 5 0a01
  > 6 0a7f
  > 7 128000
  > 8 12007f
  > 9 1a008000
  > 10 0e00
  > 11 2179656168
  > 12 06
> 18626166 95014046726564686f6c6d4305413da6832fbc3f
  > 4046726564686f6c6d 4305413da6832fbc3f
> 18626172 2868656c6c6f
> 1862617a 06

The library encode undefined as value 2 for BOOLNULL tag while nothing defined in spec.

Note that seems a deviation from JSON compatibility and is not very portable as differentiation between undefined and null does not exist in most language.

Did a query where I just got the following error:

Uncaught Error: invalid boolnull
    at decode_boolnull (bundle-ui.js:43553)
    at decode_type (bundle-ui.js:43630)
    at decode_object (bundle-ui.js:43536)
    at decode_type (bundle-ui.js:43629)
    at Object.decode (bundle-ui.js:43642)
    at bundle-ui.js:102787
    at bundle-ui.js:148005
    at getData (bundle-ui.js:25400)
    at bundle-ui.js:25411
    at getBlock (bundle-ui.js:25380)

It came up as part of an ssb-db2 query I ran. Not sure how to track it down, but it happened in response to this query:

SSB.db.query(SSB.db.operators.type("channel"), SSB.db.operators.toCallback((err, msgs) => { console.log("Error: ", err); console.log("Messages: ", msgs) }))

Well, this is a new one:

Uncaught ReferenceError: decode_reserved is not defined
    at decode_type (bundle-ui.js:43642)
    at Object.decode (bundle-ui.js:43654)
    at bundle-ui.js:80851
    at bundle-ui.js:122305
    at getData (bundle-ui.js:25412)
    at bundle-ui.js:25423
    at getBlock (bundle-ui.js:25392)
    at get (bundle-ui.js:25421)
    at bundle-ui.js:25583
    at Object.log.get (bundle-ui.js:122301)

Evidently in this case, type == 7, which I found out by adding a decode_reserved function, sticking a breakpoint on it, and walking back up the call stack to decode_type.

Continuing from jerive/bipf-napi#1 (comment)

@jerive I noticed that in test/perf.js, sometimes seekKey(buffer, start, target) is called with buffer = 0 and that's what causes a deopt. We could do two things: (1) check why is buffer = 0 being passed to this function in the perf.js, and not pass that (invalid) arg, (2) put a if (Buffer.isBuffer(buffer)) check in the implementation, although this will likely impact performance a tiny bit.

From #1 (comment)

btw, i didn't quite consider this production ready yet. it currently makes small numbers bigger, because a double is 8 bytes. was considering making it into two varints as in scientific notation 10e1000 or something like that, but need to figure out a performant way to encode/decode them

Hello,

I don't know if it is the good place for this, but I just wanted to let you know that I started an implementation of bipf in rust. It is just a toy project for me to learn rust, so there is certainly a lot of room for improvement, but so far I observe a x2/x3 perf gain on both serialization and deserialization.

Looking forward to hearing from you,

Jerome