drewmcarthur / msgpack-bal

an implementation of the msgpack spec in ballerina. / msgpack.org[Ballerina]

msgpack-bal

an implementation of the msgpack spec in ballerina.

!!! this library is still a WIP, see the progress tracker below.

how to use

import it from your ballerina project by tossing this into your Ballerina.toml file

[[dependency]]
org="drewmca"
name="msgpack"
version="0.0.7"

and your editor should see the module and let you use the library, something like this:

import ballerina/io;
import drewmca/msgpack;
json obj = {"hello": "world!"}
byte[] encoded = msgpack:encode(obj);
json decoded = msgpack:decode(encoded);
io:println(decoded);

Formats implemented

from the msgpack format spec

• nil format
  • nil
• bool format family
  • true
  • false
• int format family
  • PositiveFixInt
  • NegativeFixInt
  • Uint8
  • Uint16
  • Uint32
  • Uint64
  • Int8
  • Int16
  • Int32
  • Int64 (not implemented bc ballerina cannot do 1 << 64)
• float format family
  • Float32
  • Float64
• str format family
  • FixStr
  • Str8
  • Str16
  • Str32 (implemented but untested)
• bin format family
  • Bin8
  • Bin16
  • Bin32 (implemented but untested)
• array format family
  • FixArray
  • Array16
  • Array32 (implemented but untested)
• map format family
  • FixMap
  • Map16
  • Map32 (implemented but untested)
• ext format family
  • Ext8
  • Ext16
  • Ext32
  • FixExt1
  • FixExt2
  • FixExt4
  • FixExt8
  • FixExt16
• Timestamp extension type

Roadmap

This is a quick and dirty implementation, generally building out functionality first and will refactor after.

• v0.3: implementation of some, but not all of the spec. quick & dirty implementation.
• v0.4: added benchmarks, coreutil for int->byte[] map16, str, & int impl
• v0.5: implemented maps and arrays, also refactored decoding to pop bytes off as it goes
• v0.6: implemented binary byte arrays, introduced some better error handling
• v0.7: version bump for updating docs and pushing to bal central
• v1.0: full compatibility with the msgpack spec, including a full test suite and benchmarks

TODO:

• create a big json file of test cases
• put a checkbox of msgpack format types here
• separate functions into different files
• refactor decoding to pop bytes off the array
• test long array/map values

msgpack-bal's Issues

Cannot decode large signed Int64

from testCases.bal:67

// saving this test case for later: encoding works, 
// but currently trying to run decode([0xd3, ...]) will fail, 
// because 0xd3 == signed Int64, and our algorithm fails.
// this is because we calculat an int, then subtract (1 << 8*nBytes), and 
// if nBytes == 8, we would try to 1 << 64, which doesn't work with 64-bit ints.
TestCase[] SignedInt64TestCase = [
    {"input": -11111111111, "output": [0xd3, 0xff, 0xff, 0xff, 0xfd, 0x69, 0xb9, 0xe6, 0x39]}
];

Should constrain int subtypes for functions

ballerina has builtin integer subtypes that we should use to add further constraints to our int encoding/decoding functions

`Float` type not yet implemented

Main blocker on this is messing with number -> binary conversions, endianness, etc. feels like this should be another ballerina library that we can lean on.