Proposal for a new API to get type and size info without unpack about msgpack-ruby

[nurse]

Just FYI, I'm implementing similar feature (but C layer) https://github.com/nurse/msgpack-ruby/blob/94af4e565a1a849740acd11ba7623df2faa0f84c/ext/msgpack/unpacker.c#L918

[tagomoris]

I found that what I really want is a method to know how many msgpack objects exist in a binary. Hmm.

[tagomoris]

@nurse I updated the description above. Is it still similar to yours?

[nurse]

Your proposal needs a cursor (or an API to really skip objects without allocating objects). Mime contains such implementation as a cursor, but yours seems to want to move the unpacker's position.

[frsyuki]

Doesn't Unpacker#read_array_header or Unpacker#read_map_header work?
But anyways it sounds good idea to add methods in addition to above methods to tell:

• type of next value
• length of next str or bin
• type and length of next ext

[frsyuki]

Returning a class instance (immutable struct) is better than Hash in terms of performance.
I think it should be 2 methods because checking types needs only 1 byte but checking length need more bytes:

class Unpacker
  #
  # Returns type of next value.
  # This peeks 1 byte from the underlaying internal buffer.
  #
  # :nil, :string. :integer, :float, :array, :map, ...
  #
  # @return Symbol
  def peek_next_type
  end

  #
  # Returns ValueInfo of next value.
  # This peeks 1 - 5 bytes from the underlaying internal buffer.
  #
  # @return ValueInfo or its subclass
  def peek_next_value_info
  end
end

class ValueInfo
  # Returns type of the value (:string, :binary, :integer, ...)
  def type
  end

  # Returns of size of string, binary, or extention value
  def size
  end

  alias_method :length, :size

  # Returns type id of extension value
  def ext_type_id
  end

  # Returns true if type is :string or :binary
  def raw?
  end

  # Returns true if type is :integer or :float
  def number?
  end
end

These are optional (My bet is that NOT adding them until some actually need them):

class Unpacker
  #
  # Returns format of next value
  #
  # :nil, :str8, :str16, :str32, :positive_fixint, :negative_fixint, ...
  # 
  def peek_next_format
  end
end

This is how msgpack-java designs data model around types:

• ValueType
• MessageFormat
• ExtensionTypeHeader

[frsyuki]

I think it's NOT good idea to add read_next_type. Because moving cursor makes internal state of the Unpacker unusable. There're no methods to read payload only.

Adding such method as like msgpack-java does is another idea but it will need read_next_format instead of read_next_type to return MessageFormat class. Because read_next_type doesn't tell length of the payload users need to read next. It's another topic.

[chrisseaton]

Is anyone actively requesting this API today? Does someone need it and they're having to do a hack to get around it? Anything like that?