protobuf-ruby / beefcake

A sane Google Protocol Buffers library for Ruby

Beefcake

A pure-Ruby Google Protocol Buffers library. It's all about being Buf; ProtoBuf.

Installation

gem install beefcake

Usage

require 'beefcake'

class Variety
  include Beefcake::Message

  # Required
  required :x, :int32, 1
  required :y, :int32, 2

  # Optional
  optional :tag, :string, 3

  # Repeated
  repeated :ary,  :fixed64, 4
  repeated :pary, :fixed64, 5, :packed => true

  # Enums - Simply use a Module (NOTE: defaults are optional)
  module Foonum
    A = 1
    B = 2
  end

  # As per the spec, defaults are only set at the end
  # of decoding a message, not on object creation.
  optional :foo, Foonum, 6, :default => Foonum::B
end

# You can create a new message with hash arguments:
x = Variety.new(:x => 1, :y => 2)

# You can set fields individually using accessor methods:
x = Variety.new
x.x = 1
x.y = 2

# And you can access fields using Hash syntax:
x[:x] # => 1
x[:y] = 4
x # => <Variety x: 1, y: 4>

Encoding

Any object responding to << can accept encoding

# see code example above for the definition of Variety
x = Variety.new(:x => 1, :y => 2)

# For example, you can encode into a String:
s = ""
x.encode(s)
s # => "\b\x01\x10\x02)\0"

# If you don't encode into anything, a new Beefcake::Buffer will be returned:
x.encode # => #<Beefcake::Buffer:0x007fbfe1867ab0 @buf="\b\x01\x10\x02)\0">

# And that buffer can be converted to a String:
x.encode.to_s # => "\b\x01\x10\x02)\0"

Decoding

# see code example above for the definition of Variety
x = Variety.new(:x => 1, :y => 2)

# You can decode from a Beefcake::Buffer
encoded = x.encode
Variety.decode(encoded) # => <Variety x: 1, y: 2, pary: [], foo: B(2)>

# Decoding from a String works the same way:
Variety.decode(encoded.to_s) # => <Variety x: 1, y: 2, pary: [], foo: B(2)>

# You can update a Beefcake::Message instance with new data too:
new_data = Variety.new(x: 12345, y: 2).encode
Variety.decoded(new_data, x)
x # => <Variety x: 12345, y: 2, pary: [], foo: B(2)>

Generate code from .proto file

protoc --beefcake_out output/path -I path/to/proto/files/dir path/to/file.proto

You can set the BEEFCAKE_NAMESPACE variable to generate the classes under a desired namespace. (i.e. App::Foo::Bar)

About

Ruby deserves and needs first-class ProtoBuf support. Other libs didn't feel very "Ruby" to me and were hard to parse.

This library was built with EventMachine in mind. Not just blocking-IO.

Source: https://github.com/protobuf-ruby/beefcake

Support Features

• Optional fields
• Required fields
• Repeated fields
• Packed Repeated Fields
• Varint fields
• 32-bit fields
• 64-bit fields
• Length-delimited fields
• Embedded Messages
• Unknown fields are ignored (as per spec)
• Enums
• Defaults (i.e. optional :foo, :string, :default => "bar")
• Varint-encoded length-delimited message streams

Future

• Imports
• Use package in generation
• Groups (would be nice for accessing older protos)

Further Reading

http://code.google.com/apis/protocolbuffers/docs/encoding.html

Testing

rake test

Beefcake conducts continuous integration on Travis CI. The current build status for HEAD is .

All pull requests automatically trigger a build request. Please ensure that tests succeed.

Currently Beefcake is tested and working on:

• Ruby 1.9.3
• Ruby 2.0.0
• Ruby 2.1.0
• Ruby 2.1.1
• Ruby 2.1.2
• JRuby in 1.9 mode

Thank You

• Keith Rarick (kr) for help with encoding/decoding.
• Aman Gupta (tmm1) for help with cross VM support and performance enhancements.

beefcake's Issues

ArgumentError when using protoc with addressbook.proto

Using the protoc plugin with the addressbook.proto example from the Google's protobuf project on Google Code fails with:

fijabakk :: ~/src/protobuf-read-only/examples> protoc --beefcake_out /tmp addressbook.proto                                                                                                                         
/Users/fijabakk/.rvm/gems/ruby-1.9.2-head/gems/beefcake-0.3.1/bin/protoc-gen-beefcake:267:in `ns!': wrong number of arguments (0 for 1) (ArgumentError)
    from /Users/fijabakk/.rvm/gems/ruby-1.9.2-head/gems/beefcake-0.3.1/bin/protoc-gen-beefcake:260:in `compile'
    from /Users/fijabakk/.rvm/gems/ruby-1.9.2-head/gems/beefcake-0.3.1/bin/protoc-gen-beefcake:145:in `block in compile'
    from /Users/fijabakk/.rvm/gems/ruby-1.9.2-head/gems/beefcake-0.3.1/bin/protoc-gen-beefcake:143:in `map'
    from /Users/fijabakk/.rvm/gems/ruby-1.9.2-head/gems/beefcake-0.3.1/bin/protoc-gen-beefcake:143:in `compile'
    from /Users/fijabakk/.rvm/gems/ruby-1.9.2-head/gems/beefcake-0.3.1/bin/protoc-gen-beefcake:293:in `'
    from /Users/fijabakk/.rvm/gems/ruby-1.9.2-head/bin/protoc-gen-beefcake:19:in `load'
    from /Users/fijabakk/.rvm/gems/ruby-1.9.2-head/bin/protoc-gen-beefcake:19:in `'
--beefcake_out: protoc-gen-beefcake: Plugin failed with status code 1.

Problem with field number greater than 31

$ #------------------------------------------------------------------------------------------------
$ uname -a
Linux localhost.localdomain 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:35 EDT 2010 i686 i686 i386 GNU/Linux

$ cat /etc/redhat-release
CentOS release 5.5 (Final)

$ ruby -v
ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-linux]

$ gem -v
1.8.4

$ gem list|grep beefcake
beefcake (0.3.4)

$ #------------------------------------------------------------------------------------------------
$ cat foo.proto
package foo;
message Foo {
message Thing {
required string something = 31;
}
repeated Thing things = 1;
}

$ #------------------------------------------------------------------------------------------------
$ cat bar.rb
require 'beefcake'
require 'foo.pb'

class Bar
include Beefcake::Message

def toProtobuf
a_thing = Foo::Thing.new(
:something => 'Value for "something"'
)
p a_thing.encode
end
end

$ #------------------------------------------------------------------------------------------------
$ cat blat.rb
require 'bar'

class Blat
@a_bar = Bar.new
@a_bar.toProtobuf
end

$ #------------------------------------------------------------------------------------------------
$ protoc --beefcake_out . foo.proto && ruby blat.rb
#<Beefcake::Buffer:0xb7ea35f0 @buf="\372\025Value for "something"">

$ #------------------------------------------------------------------------------------------------
$ #### edit foo.proto and change 31 to 32
$ protoc --beefcake_out . foo.proto && ruby blat.rb
/usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake/buffer/base.rb:80:in <<': can't convert Fixnum into String (TypeError) from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake/buffer/base.rb:80:in <<'
from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake/buffer/encode.rb:17:in append_info' from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake/buffer/encode.rb:10:in append'
from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake.rb:102:in encode!' from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake.rb:90:in each'
from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake.rb:90:in encode!' from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake.rb:79:in encode'
from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake.rb:72:in each' from /usr/local/lib/ruby/gems/1.8/gems/beefcake-0.3.4/lib/beefcake.rb:72:in encode'
from ./bar.rb:11:in `toProtobuf'
from blat.rb:5

.proto file generator

Feature Request

Currently you have proto to ruby. I'd love it to have a generator the other way as well. Write first in Ruby, then export to .proto files for other devs to use.

force_encoding doesn't work on sockets

How do you feel about relaxing Buffer#buf= to pass through objects which don't have force_encoding? I've got a few projects for which it would be convenient to use a TCP socket as a buffer.

the generator and default enum values don't get scoped correctly

Using the addressbook.proto example with the generator produces a ruby file with a syntax error:
''''
class Person
include Beefcake::Message

module PhoneType
MOBILE = 0
HOME = 1
WORK = 2
end

class PhoneNumber
include Beefcake::Message

required :number, :string, 1
optional :type, Person::PhoneType, 2, :default => HOME

end

required :name, :string, 1
required :id, :int32, 2
optional :email, :string, 3
repeated :phone, Person::PhoneNumber, 4

end
''''

The :default => HOME part is the problematic part. Needs to be Person::PhoneType::HOME.

Problem with boolean required default value

I have this code:

require 'beefcake'

class TestFalse
  include Beefcake::Message
  required :processed, :bool, 1, default: false

end

class TestTrue
  include Beefcake::Message
  required :processed, :bool, 1, default: true

end

puts TestTrue.new.encode

puts TestFalse.new.encode

This results in both getting:
Beefcake::Message::RequiredFieldNotSetError: Field processed is required but nil
They should just store the default value true and false.

0.4.0

Unclear example on README

Instead of

s = ""
x.encode(s)
p [:s, s]
# or (because encode returns the string/stream)
p [:s, x.encode]
# or
open("x.dat") do |f|
  x.encode(f)
end

I’d prefer to see something like this

# You can encode into a given string or stream, like this:
s = ""
x.encode(s)
puts "Encoded into an explicit string: #{s.inspect}"

# If you don’t give anything to encode into, a new string will be returned:
puts "Encoded into an implicit string: #{x.encode.inspect}"

# You can also encode into a stream, like this:
open("x.dat") do |io|
  x.encode(io)
end

The code generator doesn't properly handle some field names

if you create a message with certain names for fields the code generator fails.

message Msg
{
    optional int32  value = 1;
    optional int32 time = 2;
    optional int32 xtime = 3;
}

I don't know if there are any other names that cause problems.

2.1.0 major feature "CREATE TYPE" support

For 2.0.6, I use blob type to restore complicated data structure. Use the code for encode/decode like this:

• ByteBuffer.wrap(entity.toByteArray())

However, 2.1.0 beta1 already provide major feature to present complicate data structure with "CREATE TYPE".

This is the ticket for Cassandra
https://issues.apache.org/jira/browse/CASSANDRA-5590

I think beefcake should support this as well. Thanks.

problems with nested messages caused by bool fix)))

next if o[f.name] == false
This line leads to error, because o[f.name] is a Beefcake::Buffer, so it uses redefined method ==(o).
And false[fld.name] raises NoMethodError

Troubles with required bool

Simple example that doesn't work

class Simple
include Beefcake::Message
required :ok, :bool, 1
end

s = Simple.new :ok => false

Simple.decode(s.encode) rises error
Beefcake::Message::RequiredFieldNotSetError: Field ok is required but nil

question: why use '.pb.rb' extension for generated classes?

i'm curious why beefcake generates ruby classes with an extension .pb.rb.

currently i rename the classes back to just .rb so they work with rails auto-loading. but curious why they're generated this way in the first place.

Guidance for using beefcake with a gRPC server

We are currently using gruf to serve a gRPC endpoint which sits on top of the standard grpc library. However, we have run into the problem that beefcake is trying to solve in providing a more usable ruby protobuf definition. I was wondering if you have any guidance on how we could use beefcake's protobuf generation to serve a gRPC endpoint (with or without the above gruf/grpc libraries).

I know this may not be a primary concern for the authors of beefcake, but was hoping someone here could point me in the right direction. Thank you!

List of supported / unsupported features?

Not a bug, more of a question.. By the looks of it, all the basic protobuf features are already supported.

Q: What's missing? Packages, imports, pass-through of unknown fields, etc? Any of that on the radar?

ruby-protocol-buffers (https://github.com/mozy/ruby-protocol-buffers) has a nice summary in the readme, I think it would be a good idea to add something similar.

ig

P.S. Great work!

Provide examples for reflection functionality and ruby code generation @ runtime

Hello,
I am quite new to beefcake, but I would like to ask for 2 snippets of code, which for sure will be really useful:

• How to exploit beefcake to decode generic messages without having a .proto file
• If possible, how to generate Beefcake::Messages @ runtime from a given .proto file

Does not support extensions

I'm using nested extensions to simulate polymorphic messages as decribed here: http://www.indelible.org/ink/protobuf-polymorphism/

It would be great if beefcake could support them, right now it just ignores the extensions.

Thanks

Buffer.wire_for doesn't cope with packed+repeated fields

The wire type for packed, repeated fields is 2 (reference).

Currently, Buffer.wire_for parses the field type only, regardless of whether the field is packed+repeated. It'll therefore fail when decoding a packed+repeated field.

Quick fix is to add an extra line to Decode.decode in beefcake.rb:

      exp = Buffer.wire_for(fld.type)
      if fld.rule == :repeated && fld.opts[:packed] then exp = 2 end

Can't parse simple .proto composite message

This simple .proto file does not parse:

message Error {
    optional int32 code = 1;
    optional bytes message = 2;
    optional bytes data = 3;
}


message Request {
    optional Error error = 1;
    optional int32 whatever = 2;
}

It fails with a following error:

protoc --beefcake_out . test.proto        
/home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:232:in `gsub': wrong argument type nil (expected Regexp) (TypeError)
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:232:in `field!'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:202:in `block (2 levels) in message!'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:201:in `each'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:201:in `block in message!'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:174:in `call'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:174:in `indent'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:186:in `message!'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:271:in `block (2 levels) in compile'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:270:in `each'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:270:in `block in compile'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:278:in `call'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:278:in `ns!'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:269:in `compile'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:145:in `block in compile'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:143:in `map'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/lib/beefcake/generator.rb:143:in `compile'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/gems/beefcake-0.3.7/bin/protoc-gen-beefcake:8:in `<top (required)>'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/bin/protoc-gen-beefcake:19:in `load'
        from /home/kybu/.rvm/gems/ruby-1.9.3-p0/bin/protoc-gen-beefcake:19:in `<main>'
--beefcake_out: protoc-gen-beefcake: Plugin failed with status code 1.

Thanks.

Scanner/parser for .proto files

You might take a look at this for a short-cut: https://github.com/macks/ruby-protobuf/tree/master/lib/protobuf/compiler

Release 1.0

We're missing one of the Semantic Versioning significant figures. We should make the next release with breaking changes "1.0.0".

Calling force_encoding on strings beefcake didn't create

I had an issue recently where ActiveRecord queries started falling over with Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT errors after an attribute was added to a beefcake call. I tracked it back to beefcake force_encoding the strings we passed in to binary, rather than making binary encoded copies of them. Is this as expected?

beefcake/lib/beefcake/buffer/encode.rb

Line 109 in 31985ea

encoded = actual_string.force_encoding 'binary'

This line reads like it doesn't expect force_encoding to modify the string in place, which is what the method does.

The code generator doesn't handle nested messages.

The protocol buffer definition:

message A
{
  optional int32 f = 1;
}

message B
{
  optional int32 g = 1;
  optional A submsg = 2;
}

Fails to generate ruby code with a "gsub: wrong argument type nil" error.

Beefcake 1.0.1 breaking riemann-client

See riemann/riemann-tools#103 for details. This seems to be a beefcake issue with the latest release.

Message#== doesn't work with non-messages

https://github.com/protobuf-ruby/beefcake/blob/master/lib/beefcake.rb#L237

This raises an exception, expected false:

some_message == :symbol

This works:

:symbol == some_message