Symbols are converted to Strings on pack; can't modify this behaviour about msgpack-ruby

[tagomoris]

#114 was merged into v1.

[tagomoris]

The reason of it is just performance. It makes makes packer/unpacker slower to make it possible to pack strings/symbols/numbers (or some others) into different representations.
I'm ok to make it modify-able by using ext types if there's any way without performance regression.

[simonrussell]

Okay, I might try and have a look at this. It seems like it should be fairly straightforward to do it with minimal performance impact (i.e. done in the native code with an 'if').

[christopheraue]

I'm having the same problem. What about giving the (un)packer an allow_core_ext option on initialization? It had to be checked once for every (un)pack.

• If false (default) it selects the (fast) current algorithm.
• If true it selects the (slow) modified algorithm looking for an ext type before defaulting to the current algorithm.

[tagomoris]

It looks an option to make it implemented.

[christopheraue]

I had some time to take a closer look at it.

1. Allowing all core types (number, strings, …) to be custom serialized in the way I proposed above imposes the performance penalty to every serialization of every type. And this even if we only want to use an ext type for just one core type (like with symbols). This seems to be too much. Especially when most of the time (if not always) there is no need to use an ext type for the general core types (nil, bool, number, string, array, hash). The only remaining exception are symbols.

2. So, I think it's best to only support an ext type for symbols. Once an ext type is registered for them the intention is clear:

packer = MessagePack::Packer.new
packer.register_type(0x00, Symbol, :to_msgpack_ext)

packer.pack(:symbol).to_s # => "\xc7\x06\x00symbol"

No further option/configuration needed.

3. For an implementation see #114.

[tagomoris]

IMO that approach looks reasonable... @frsyuki how do you think about it?

My understanding:

• there's no case to want to pack core types into ext types - except for symbols
• add just a flag/hook to check whether Symbol is registered into registry for ext types
• pack Symbols as ext types if it's already registered into ext types

[tagomoris]

and @simonrussell ?

[simonrussell]

That suits me; that's more-or-less what I was thinking for an implementation; thanks @christopheraue for following this up. I guess the only other part of my problem was that Symbols get encoded and there's no way to disable that (i.e. throw an exception) -- I suppose I could register a handler that just throws an exception? (My issue here is the non-round-trip nature of encoding/decoding with Symbols).

Personally I think the pattern works for all core types; maybe as a future extension those could be implemented -- I doubt that would impact performance that much in the case where the extension types weren't registered.

[frsyuki]

Concept of #114 looks very good. That is because I agree that there are some use cases where Symbol needs a custom serialization format so that receiver of the serialized data can distinguish it from strings. And the other core types don't need such customization.

A missing thing is that there're no pretty ways to configure Packer to throw exceptions when a Symbol is given. But registering a method that throws an exception as ext type handler is possible as a workaround. I think it's good enough for its limited use case. So I think we can go with #114.

[tagomoris]

@frsyuki What you meant as "a missing thing" is: we need a way to prohibit serialization of symbols, and it's be done by hack to raise an exception in ext type handler. Is it right?

[simonrussell]

Yes, I think that was the same comment I was making -- you can't disable Symbol packing completely; so you can't be sure that you'll get a clean round-trip (your Symbols will become Strings).

[frsyuki]

@tagomoris right.

[tagomoris]

OK, let's go forward with #114.