golang / protobuf

Go support for Google's protocol buffers

Go support for Protocol Buffers

This module (github.com/golang/protobuf) contains Go bindings for protocol buffers.

It has been superseded by the google.golang.org/protobuf module, which contains an updated and simplified API, support for protobuf reflection, and many other improvements. We recommend that new code use the google.golang.org/protobuf module.

Versions v1.4 and later of github.com/golang/protobuf are implemented in terms of google.golang.org/protobuf. Programs which use both modules must use at least version v1.4 of this one.

See the developer guide for protocol buffers in Go for a general guide for how to get started using protobufs in Go.

See release note documentation for more information about individual releases of this project.

See documentation for the next major revision for more information about the purpose, usage, and history of this project.

Package index

Summary of the packages provided by this module:

• proto: Package proto provides functions operating on protobuf messages such as cloning, merging, and checking equality, as well as binary serialization and text serialization.
• jsonpb: Package jsonpb serializes protobuf messages as JSON.
• ptypes: Package ptypes provides helper functionality for protobuf well-known types.
• ptypes/any: Package any is the generated package for google/protobuf/any.proto.
• ptypes/empty: Package empty is the generated package for google/protobuf/empty.proto.
• ptypes/timestamp: Package timestamp is the generated package for google/protobuf/timestamp.proto.
• ptypes/duration: Package duration is the generated package for google/protobuf/duration.proto.
• ptypes/wrappers: Package wrappers is the generated package for google/protobuf/wrappers.proto.
• ptypes/struct: Package structpb is the generated package for google/protobuf/struct.proto.
• protoc-gen-go/descriptor: Package descriptor is the generated package for google/protobuf/descriptor.proto.
• protoc-gen-go/plugin: Package plugin is the generated package for google/protobuf/compiler/plugin.proto.
• protoc-gen-go: The protoc-gen-go binary is a protoc plugin to generate a Go protocol buffer package.

Reporting issues

The issue tracker for this project is located here.

Please report any issues with a sufficient description of the bug or feature request. Bug reports should ideally be accompanied by a minimal reproduction of the issue. Irreproducible bugs are difficult to diagnose and fix (and likely to be closed after some period of time). Bug reports must specify the version of the Go protocol buffer module and also the version of the protocol buffer toolchain being used.

Contributing

This project is open-source and accepts contributions. See the contribution guide for more information.

Compatibility

This module and the generated code are expected to be stable over time. However, we reserve the right to make breaking changes without notice for the following reasons:

• Security: A security issue in the specification or implementation may come to light whose resolution requires breaking compatibility. We reserve the right to address such issues.
• Unspecified behavior: There are some aspects of the protocol buffer specification that are undefined. Programs that depend on unspecified behavior may break in future releases.
• Specification changes: It may become necessary to address an inconsistency, incompleteness, or change in the protocol buffer specification, which may affect the behavior of existing programs. We reserve the right to address such changes.
• Bugs: If a package has a bug that violates correctness, a program depending on the buggy behavior may break if the bug is fixed. We reserve the right to fix such bugs.
• Generated additions: We reserve the right to add new declarations to generated Go packages of .proto files. This includes declared constants, variables, functions, types, fields in structs, and methods on types. This may break attempts at injecting additional code on top of what is generated by protoc-gen-go. Such practice is not supported by this project.
• Internal changes: We reserve the right to add, modify, and remove internal code, which includes all unexported declarations, the generator package, and all packages under internal.

Any breaking changes outside of these will be announced 6 months in advance to [email protected].

protobuf's Issues

README in Markdown

Is there a reason for not having the README in rich text? Thought I'd ask before submitting a PR on the repo (not sure if its formatted this way intentionally).

how to set/get extension filed in go?

Any reason for not using golang.org/x/protobuf import path?

@dsymonds

As almost all other tools (e.g. oauth2) uses it, why does this repo use the barebone github path instead of the vanity ones? Wouldn't it make more sense to switch over?

gcloud sdk

I'm trying to deploy an application on GAE VM. Although I'm not using the protobuf package in my app the goapp command throws the error below:
/Users/xyz/google-cloud-sdk/bin/goapp deploy ./

github.com/golang/protobuf/proto/decode.go:219: undefined: structPointer
github.com/golang/protobuf/proto/decode.go:219: undefined: field
github.com/golang/protobuf/proto/properties.go:65: undefined: structPointer
github.com/golang/protobuf/proto/properties.go:73: undefined: structPointer
github.com/golang/protobuf/proto/properties.go:82: undefined: structPointer
github.com/golang/protobuf/proto/properties.go:166: undefined: field

two proto files belongto same package

For example:
a.proto
package Test;
message A{
required string name=1;
}

b.proto
package Test;
import "a.proto";
message B{
required A a=1;
}

for this case ,protoc b.proto will see this :
import Test1 "a.pb"

thanks for reading my poor English

Proto3 Nested Message Getter is not useful

Take this generated code:

type FlagId struct {
    Deployment *DeploymentId `protobuf:"bytes,1,opt,name=deployment" json:"deployment,omitempty"`
    Name       string        `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
}

func (m *FlagId) Reset()         { *m = FlagId{} }
func (m *FlagId) String() string { return proto.CompactTextString(m) }
func (*FlagId) ProtoMessage()    {}

func (m *FlagId) GetDeployment() *DeploymentId {
    if m != nil {
        return m.Deployment
    }
    return nil
}

What is the purpose of this Getter? It will not be called on a nil m *FlagId.
I actually hoped it would create an new instance of the type behind the Deployment field, but it seems to not do anything other than a normal m.Deployment would.

I really hoped it would create a new instance if it was nil, this way I could solve the problem in without reflection:
grpc-ecosystem/grpc-gateway#32

README refers to unexisting files

The README file mentions to create a Makefile like this:

    include $(GOROOT)/src/Make.$(GOARCH)

    ...
    include $(GOROOT)/src/Make.pkg

But those files no (longer?) exists in go 1.4
What's the content of those files?

Go lint naming compliance

Is it an aim of protoc-gen-go to produce names which pass golint? In particular, I find it is not capitalizing abbreviations as I expect (Id vs ID).

A contrived example is

message Args {
  int64 id = 1;
  string api_endpoint = 2;
}

which produces

type Args struct {
    Id          int64  `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
    ApiEndpoint string `protobuf:"bytes,2,opt,name=api_endpoint" json:"api_endpoint,omitempty"`
}

I prefer the names ID and APIEndpoint which pass golint. Currently, I must wrap these public fields in getters and be sure to use them to avoid propagating use of the generated names throughout my codebase.

For this project, is passing golint a goal, a loose aspiration, or no stance?

Thanks a bunch for your work on this :)

Related:
golang/lint#89

proto: make RequiredNotSetError.field exported

Would it be possible to add this function to proto/encode.go ?

func NewRequiredNotSetError(field string) *RequiredNotSetError {
    return &RequiredNotSetError{field}
}

That way generated code can also return a RequiredNotSetError with a field string.
Currently I have to:

return new(RequiredNotSetError)

Which is cool since then users can still type check the error, but it is pretty useless for debugging purposes.

support for google/protobuf/timestamp.proto

Hi,

what is the plan for adding support for google/protobuf/timestamp.proto, e.g. generating Time fields in the proto structs?

Thanks

Feature request: A new file option 'go_import_path'

I use -I$GOPATH/src to include protos, but it's path does not match somtimes.
As golang does not work in simillar way with other langauges, an extra option is required.

Example usecase:
golang package: appcore-project/cdt-go/country
protobuf: appcore-project/protos/appcore/cdt/country/country.proto

Generating proto files in multiple directories

Hi, when generating go protos like

protoc --go_out=plugins=grpc:. /.proto

it fails with error:inconsistent package names: blabla

I guess the assumptions is that it will be called only for files in the same directory. Are there plans to fix that?

Empty map is cloned as nil via proto.Clone.

Empty map is cloned as nil via proto.Clone. nil and empty map are different in Go.

protobuf3 default JSON encoding

In the proto3 there are encodings for different types, in particular the Timestamp type. Where is that code implemented? I am having a hard time figuring it out. https://developers.google.com/protocol-buffers/docs/proto3#json

cc @sym3tri

protobuf3 support for "Any"

Is there an idiomatic way to support Any's in messages in Go? I have been struggling with reflect for a couple of days, and I think if I'm having this much trouble, I must be doing it wrong.

MarshalText incorrectly handles unknown bytes

The following program crashes with the following message:

package main

import (
    "bytes"
    "fmt"

    pb "github.com/dvyukov/go-fuzz/examples/protobuf/pb"
    "github.com/golang/protobuf/proto"
)

func main() {
    data := []byte("\n\x010")
    v := new(pb.M25)
    err := proto.Unmarshal(data, v)
    if err != nil {
        panic(err)
    }

    var buf bytes.Buffer
    err = proto.MarshalText(&buf, v)
    if err != nil {
        panic(err)
    }
    v2 := new(pb.M25)
    err = proto.UnmarshalText(buf.String(), v2)
    if err != nil {
        fmt.Printf("failed to UnmarshalText: %q\n", buf.Bytes())
        panic(err)
    }
}

failed to UnmarshalText: "/* 3 unknown bytes */\n1: \"0\"\n"
panic: line 1.0: unexpected byte 0x2f

MarshalText should not produce text that is unparsable by UnmarshalText.
Either MarshalText should write unknown bytes differently, or UnmarshalText accept the current output of MarshalText.

on commit 34a5f24

size of map with string key

Add this to MessageWithMap in proto/testdata/test.proto

map<string, string> stringy = 4;

Add this to size_test.go in proto

{"map with string and string", &pb.MessageWithMap{
        Stringy: map[string]string{"YGeGc18B1eey5ifSdWMn2c21wRAgY90YvH4HmCOeSa9q2pfAIJa9WiGOOZGZKxZgoJo7VXnQumvsch": "RggKn1pmtK98gnMhD2J0GHy9R4IFPZQwjFeyphRQzEqXkVn"},
    }},

You should get this error:

size_test.go:141: map with string and string: Size(stringy:<key:"YGeGc18B1eey5ifSdWMn2c21wRAgY90YvH4HmCOeSa9q2pfAIJa9WiGOOZGZKxZgoJo7VXnQumvsch" value:"RggKn1pmtK98gnMhD2J0GHy9R4IFPZQwjFeyphRQzEqXkVn" > ) = 131, want 132
    size_test.go:142: map with string and string: bytes: []byte{0x22, 0x81, 0x1, 0xa, 0x4e, 0x59, 0x47, 0x65, 0x47, 0x63, 0x31, 0x38, 0x42, 0x31, 0x65, 0x65, 0x79, 0x35, 0x69, 0x66, 0x53, 0x64, 0x57, 0x4d, 0x6e, 0x32, 0x63, 0x32, 0x31, 0x77, 0x52, 0x41, 0x67, 0x59, 0x39, 0x30, 0x59, 0x76, 0x48, 0x34, 0x48, 0x6d, 0x43, 0x4f, 0x65, 0x53, 0x61, 0x39, 0x71, 0x32, 0x70, 0x66, 0x41, 0x49, 0x4a, 0x61, 0x39, 0x57, 0x69, 0x47, 0x4f, 0x4f, 0x5a, 0x47, 0x5a, 0x4b, 0x78, 0x5a, 0x67, 0x6f, 0x4a, 0x6f, 0x37, 0x56, 0x58, 0x6e, 0x51, 0x75, 0x6d, 0x76, 0x73, 0x63, 0x68, 0x12, 0x2f, 0x52, 0x67, 0x67, 0x4b, 0x6e, 0x31, 0x70, 0x6d, 0x74, 0x4b, 0x39, 0x38, 0x67, 0x6e, 0x4d, 0x68, 0x44, 0x32, 0x4a, 0x30, 0x47, 0x48, 0x79, 0x39, 0x52, 0x34, 0x49, 0x46, 0x50, 0x5a, 0x51, 0x77, 0x6a, 0x46, 0x65, 0x79, 0x70, 0x68, 0x52, 0x51, 0x7a, 0x45, 0x71, 0x58, 0x6b, 0x56, 0x6e}

why all generated file is pointer field

I have proto file

message MsgErrCode {
    required int32 err_code = 1;
    optional string reason = 2;
}

After generated, I got go file

type MsgErrCode struct {
     ErrCode *int32 `....`
     Reason *string  `....`
}

Notice that the field generated is pointer, It make me hard to use the struct,

I can't do &MsgErrCode{ErrCode: int32(12), Reason: "hello world"}

I have to do &MsgErrCode{ErrCode: proto.Int32(12), Reason: proto.String("hello world")}

Generation of repeated structs

From http://golang.org/doc/effective_go.html#slices

Slices hold references to an underlying array,
and if you assign one slice to another,
both refer to the same array.

When I generate the following proto holding the following messages

message AmessageWithAString
{
    required string aString = 1;
}

message RepeatedString 
{
    repeated AmessageWithAString PointerToSlice = 1;
}

the generator generates

type RepeatedString struct {
    PointerToSlice   []*AmessageWithAString `protobuf:"bytes,1,rep" json:"PointerToSlice,omitempty"`
    XXX_unrecognized []byte                 `json:"-"`
}

shouldn't it generate
PointerToSlice []AmessageWithAString
instead?

protoc-gen-go compile error "use of internal package not allowed" on go1.[4,5]

dhubler@duck ~/work/conf2/conf2-restconf $ go get -u github.com/golang/protobuf/proto
dhubler@duck ~/work/conf2/conf2-restconf $ go get -u github.com/golang/protobuf/protoc-gen-go
package github.com/golang/protobuf/protoc-gen-go
imports github.com/golang/protobuf/proto
imports github.com/golang/protobuf/protoc-gen-go/descriptor
imports github.com/golang/protobuf/protoc-gen-go/generator
imports github.com/golang/protobuf/protoc-gen-go/plugin
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc: use of internal package not allowed
dhubler@duck ~/work/conf2/conf2-restconf $ env | grep GO
GOROOT=/home/dhubler/work/conf2/go
GOPATH=/home/dhubler/work/conf2/conf2-restconf
dhubler@duck ~/work/conf2/conf2-restconf $ go version
go version devel +71859ef Mon Jun 22 06:57:19 2015 +0000 linux/amd64

Compile google/protobuf/*.proto files into golang structs and add to a new package

I would be happy to do the work and commit it over, but before doing any work on this, wanted to see how open the maintainers were to it. Thoughts?

Also addresses #49

Importing definitions fails because of incorrect package path

When I import some of google's proto definitions the generated code imports a package that does not exist.

Example:

syntax = "proto3";

import "google/protobuf/timestamp.proto";

package bar;

message Foo {
  google.protobuf.Timestamp start = 1;
  google.protobuf.Timestamp finish = 2;
}

Generated code:

package bar

import proto "github.com/golang/protobuf/proto"
import google_protobuf "google/protobuf"

...

The import of google_protobuf fails.

• Should I be building the google_protobuf package and adding it manually to my GOPATH?
• Is there a directive I'm missing to tell the protoc generator to build dependencies?
• Bug in golang/protobuf? Where would i start looking to add a fix?

Go 1.5 internal package not allowed error

$ go version
go version go1.5beta1 darwin/amd64
$ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
package github.com/golang/protobuf/proto
    imports github.com/golang/protobuf/protoc-gen-go
    imports github.com/golang/protobuf/protoc-gen-go/descriptor
    imports github.com/golang/protobuf/protoc-gen-go/generator
    imports github.com/golang/protobuf/protoc-gen-go/plugin
    imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
    imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
    imports github.com/golang/protobuf/protoc-gen-go/internal/grpc: use of internal package not allowed

How to write my own plugin for generator RPC

I notice this tool have support generate grpc output, but I'd like to implement my own RPC for some reasons.

So I wonder if can you provide a tool generate rpc interfaces without dependencies on grpc, or can you give me some information about how to write my own generator?

One more question, to provide flexibility of my RPC framework, to easy change my generator, can I write a plugin use a python but generate code of golang?

Oneof keyword in generated code is unsupported

Aka: https://developers.google.com/protocol-buffers/docs/proto3#oneof

Specifically: protos including oneof do produce working .pb.proto files, but there doesn't seem to be any of the useful bits of Oneof implemented (selecting on which field is set, setting a field resetting the rest).

I'm sure this is being tracked internally somewhere, but I thought it would be useful to have an external issue for it for people to watch.

go get -u github.com/golang/protobuf/{proto,protoc-gen-go}" gives an error - no buildable Go source files in proto3_proto

When I run go get -u github.com/golang/protobuf/{proto,protoc-gen-go} as instructed in README I keep getting the following error (with Go 1.4.2 on Linux and Mac):

$ go get -u -t -v -f github.com/golang/protobuf/{proto,protoc-gen-go}
github.com/golang/protobuf (download)
package github.com/golang/protobuf/proto
    imports github.com/golang/protobuf/proto/proto3_proto
    imports github.com/golang/protobuf/proto/proto3_proto
    imports github.com/golang/protobuf/proto/proto3_proto: no buildable Go source files in /home/dmitris/go/src/github.com/golang/protobuf/proto/proto3_proto

Should this error be ignored? (the protobuf directory does get checked out, and you can go install ./... in it). But ignoring errors seem contrary to the Go spirit 😄 - maybe it would be possible to add a dummy file in https://github.com/golang/protobuf/tree/master/proto/proto3_proto to prevent the error (not sure what is the standard and recommended way to deal with such - would be interested since it comes up in other places as well). Thanks!

call of reflect.Value.Interface on zero Value in MarshalText

The following program crashes with a panic:

package main

import (
    "github.com/golang/protobuf/proto"
    "io/ioutil"
    pb "protorepro/pb"
)

func main() {
    proto.MarshalText(ioutil.Discard, &pb.M2{F: map[string]*pb.M1{"": nil}})
}

panic: reflect: call of reflect.Value.Interface on zero Value

goroutine 1 [running]:
runtime.gopanic(0x52e860, 0xc20800e4a0)
    src/runtime/panic.go:461 +0x401 fp=0xc2080415a0 sp=0xc208041520
reflect.valueInterface(0x0, 0x0, 0x0, 0x542801, 0x0, 0x0)
    src/reflect/value.go:906 +0x82 fp=0xc2080415f8 sp=0xc2080415a0
reflect.Value.Interface(0x0, 0x0, 0x0, 0x0, 0x0)
    src/reflect/value.go:901 +0x4f fp=0xc208041630 sp=0xc2080415f8
github.com/golang/protobuf/proto.writeAny(0xc20800e400, 0x0, 0x0, 0x0, 0xc208070240, 0x0, 0x0)
    src/github.com/golang/protobuf/proto/text.go:465 +0x632 fp=0xc2080417f8 sp=0xc208041630
github.com/golang/protobuf/proto.writeStruct(0xc20800e400, 0x536380, 0xc20800e3e0, 0xd9, 0x0, 0x0)
    src/github.com/golang/protobuf/proto/text.go:295 +0xf89 fp=0xc208041da0 sp=0xc2080417f8
github.com/golang/protobuf/proto.marshalText(0x7f558bbca1e8, 0xc20800a5f0, 0x7f558bbca210, 0xc20800e3e0, 0x62c100, 0x0, 0x0)
    src/github.com/golang/protobuf/proto/text.go:759 +0x440 fp=0xc208041ee0 sp=0xc208041da0
github.com/golang/protobuf/proto.MarshalText(0x7f558bbca1e8, 0xc20800a5f0, 0x7f558bbca210, 0xc20800e3e0, 0x0, 0x0)
    src/github.com/golang/protobuf/proto/text.go:771 +0x57 fp=0xc208041f20 sp=0xc208041ee0
main.main()
    protorepro.go:10 +0x129 fp=0xc208041f90 sp=0xc208041f20

The proto used is:

syntax = "proto2";

package example;

message M1 {
  optional uint32 f = 1;
}

message M2 {
  map<string, M1> f = 1;
}

golang/protobuf is on 16256d3
protobuf is on fe7b566

second order plugins no longer execute after c22ae3cf

I have a second order plugin written in Go and after updating my golang/protobuf code and compiler locally, my plugin is no longer called.

I traced it down to a somewhat innocuous part of c22ae3c:

diff --git a/protoc-gen-go/generator/generator.go b/protoc-gen-go/generator/generator.go
index af0e15f..49f91d7 100644
--- a/protoc-gen-go/generator/generator.go
+++ b/protoc-gen-go/generator/generator.go
@@ -484,7 +484,7 @@ func (g *Generator) CommandLineParameters(parameter string) {
        }

        g.ImportMap = make(map[string]string)
-       pluginList := "" // Default list of plugin names to enable (empty means all).
+       pluginList := "none" // Default list of plugin names to enable (empty means all).
        for k, v := range g.Param {
                switch k {
                case "import_prefix":

Why the change from s/""/"none"/? How (short of striking the above line) can I cause my second order plugin to continue to execute?

The first letter of type/field name is always uppercase in generated pb.go codes.

The first letter of type/field name is always uppercase in generated pb.go codes, making it no way to create non-exported struct/field with protobuf in Go. Is this by design?

proto3 syntax is not identified as proper syntax only proto2

| have installed the plugin in my $GOPATH in order to give it a go, and I have created a .proto file with proto3 syntax where I got rid of all the required keywords and started the file with ' syntax="proto3" ', however I keep getting this compilation error message:

" Unrecognized syntax identifier "proto3". This parser only recognizes "proto2"."

P.S: when I change from proto3 to proto2 the compilation succeeded however I have of course to invoke the required keywords again because this feature is not supported in proto2.

proposal: protoc-gen-go should be able to compile multiple packages per run

Passing multiple packages' worth of *.proto files currently fails when using (protoc-gen-go)[https://github.com/golang/protobuf/blob/master/protoc-gen-go/generator/generator.go#L612:L614].

The same functionality is supported by other generators (at least the C++ generator shipped with protoc). protoc-gen-go should support this as well.

proto3 no-longer generates pointers to fields

In proto2 fields were generated as pointers, which allowed us to check if a field was specified or not. This no longer seems to be the case. For example:

proto2:

syntax = "proto2";

message Foo {
  required int32 ID = 1;
  optional int32 SortOrder = 2;
}

gives

type Foo struct {
    ID               *int32 `protobuf:"varint,1,req" json:"ID,omitempty"`
    SortOrder        *int32 `protobuf:"varint,2,opt" json:"SortOrder,omitempty"`
    XXX_unrecognized []byte `json:"-"`
}

proto3:

syntax = "proto3";

message Foo {
  int32 ID = 1;
  optional int32 SortOrder = 2;
}

gives

type Foo struct {
    ID        int32 `protobuf:"varint,1,opt" json:"ID,omitempty"`
    SortOrder int32 `protobuf:"varint,2,opt" json:"SortOrder,omitempty"`
}

Is this the expected behavior, and if so how are we supposed to differentiate a client making a call with SortOrder = 0 vs one making a call and not specifying the SortOrder?

"go get -u github.com/golang/protobuf/{proto,protoc-gen-go}" doesn't work.

The README suggests using "go get -u" to install protobuf. However, it doesn't work.

$ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
package github.com/golang/protobuf/proto: /Users/paul/Code/go/src/github.com/golang/protobuf is from [email protected]:golang/protobuf, should be from https://github.com/golang/protobuf
package github.com/golang/protobuf/protoc-gen-go: /Users/paul/Code/go/src/github.com/golang/protobuf is from [email protected]:golang/protobuf, should be from https://github.com/golang/protobuf

writeStruct elides zero-valued fields

This seems confusing because the %+v verb would typically include these fields when used with a generic struct.

Is there some additional reason that our Stringer chooses to elide these fields?

Incorrect imports for generated files

I am unable to build a package when a proto file is included from a separate package.

src/foo/foo.proto

package foo;

message Foo
{
  optional string foo = 1;
}

protoc --go_out=src/ --proto_path=src/ src/foo/foo.proto
src/foo/foo.pb.go

// Code generated by protoc-gen-go.
// source: foo/foo.proto
// DO NOT EDIT!

/*
Package foo is a generated protocol buffer package.

It is generated from these files:
    foo/foo.proto

It has these top-level messages:
    Foo
*/
package foo

import proto "github.com/golang/protobuf/proto"
import math "math"

// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf

type Foo struct {
    Foo              *string `protobuf:"bytes,1,opt,name=foo" json:"foo,omitempty"`
    XXX_unrecognized []byte  `json:"-"`
}

func (m *Foo) Reset()         { *m = Foo{} }
func (m *Foo) String() string { return proto.CompactTextString(m) }
func (*Foo) ProtoMessage()    {}

func (m *Foo) GetFoo() string {
    if m != nil && m.Foo != nil {
        return *m.Foo
    }
    return ""
}

func init() {
}

src/bar/bar.proto

package bar;

import "foo/foo.proto";

message Bar
{
  optional foo.Foo foo = 1;
  optional string bar = 2;
}

protoc --go_out=src/ --proto_path=src/ src/bar/bar.proto
src/bar/bar.pb.go

// Code generated by protoc-gen-go.
// source: bar/bar.proto
// DO NOT EDIT!

/*
Package bar is a generated protocol buffer package.

It is generated from these files:
    bar/bar.proto

It has these top-level messages:
    Bar
*/
package bar

import proto "github.com/golang/protobuf/proto"
import math "math"
import foo "foo/foo.pb"

// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf

type Bar struct {
    Foo              *foo.Foo `protobuf:"bytes,1,opt,name=foo" json:"foo,omitempty"`
    Bar              *string  `protobuf:"bytes,2,opt,name=bar" json:"bar,omitempty"`
    XXX_unrecognized []byte   `json:"-"`
}

func (m *Bar) Reset()         { *m = Bar{} }
func (m *Bar) String() string { return proto.CompactTextString(m) }
func (*Bar) ProtoMessage()    {}

func (m *Bar) GetFoo() *foo.Foo {
    if m != nil {
        return m.Foo
    }
    return nil
}

func (m *Bar) GetBar() string {
    if m != nil && m.Bar != nil {
        return *m.Bar
    }
    return ""
}

func init() {
}

go build foo
go build bar
src/bar/bar.pb.go:18:8: cannot find package "foo/foo.pb" in any of:
/usr/local/Cellar/go/1.3/libexec/src/pkg/foo/foo.pb (from $GOROOT)
/Users/patrick/Code/workspace/src/foo/foo.pb (from $GOPATH)

When I change import foo "foo/foo.pb" to import foo "foo" the package will build correctly. Is this an issue with my use of the generated files or an issue with code generation? Thanks.

proto3 and XXX_unrecognized

I know that unknown fields have been removed from proto3, but I am trying to get an explanation about why this change was made and if there is any way to replicate that behavior in proto3.

Thanks so much.

declaring variables in import generation

https://github.com/golang/protobuf/blob/master/protoc-gen-go/internal/grpc/grpc.go#L116

Depending on the plugin order generation this could result in variables being declared in between import statements, which is invalid go.
Luckily golang/protobuf has just one plugin, so this should not be a problem, but in go1.5 this is a problem for gogo/protobuf.
gogo/protobuf#83
I could easily fix this in gogo/protobuf, but I think this might cause problems for you later as well. At which point I'll have to undo my fix and merge in your fix :(

Umarshal crashes with "call of reflect.Value.SetMapIndex on zero Value"

go version devel +87054c4 Wed Apr 22 02:50:48 2015 +0000 linux/amd64
golang/protobuf is on 655cdfa
protoc is on fe7b566

Protobuf is:

syntax = "proto2";

package main;

message M {
  map<int32, int32> f = 1;
}

Program is:

package main

import (
    "github.com/golang/protobuf/proto"
)

func main() {
    m := &M{}
    data := []byte("\v\x00")
    proto.Unmarshal(data, m)
}

Crashes with:

panic: reflect: call of reflect.Value.SetMapIndex on zero Value

goroutine 1 [running]:
runtime.gopanic(0x530860, 0xc2080104e0)
reflect.flag.mustBeExported(0x0)
reflect.Value.SetMapIndex(0x5058e0, 0xc208010440, 0xd5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
github.com/golang/protobuf/proto.(*Buffer).dec_new_map(0xc208052f70, 0xc20806c480, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.(*Buffer).unmarshalType(0xc208052f70, 0x7f70a263e198, 0x5384e0, 0xc208048070, 0xc208010400, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.(*Buffer).Unmarshal(0xc208052f70, 0x7f70a263e2d8, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.UnmarshalMerge(0xc20800a570, 0x2, 0x8, 0x7f70a263e2d8, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.Unmarshal(0xc20800a570, 0x2, 0x8, 0x7f70a263e2d8, 0xc208010440, 0x0, 0x0)
main.main()

Invalid map is successfully decoded

The following program crashes with the panic:

package main

import (
    pb "github.com/dvyukov/go-fuzz/examples/protobuf/pb"
    "github.com/golang/protobuf/proto"
)

func main() {
    data := []byte("\n\x02\n\x00")
    v := new(pb.M24)
    err := proto.Unmarshal(data, v)
    if err != nil {
        return
    }
    _, err = proto.Marshal(v)
    if err != nil {
        panic(err)
    }
}

panic: proto: map has nil element

The proto used is:

message M24 {
  map<string, M2> f = 1;
}

The message should either be successfully encoded or not decoded.

on commit 34a5f24

protoc-gen-go: support go_tag option to specify custom struct tags

Is it possible to define a message with custom tags? For example defining a json name that isn't just the lowercase field name or adding a bson tag.

cannot run on OSX

when I run protoc on Mac's terminal, will stop on

data, err := ioutil.ReadAll(os.Stdin)

proto: MarshalText fails and prints to stderr

The following program fails:

package main

import (
    "bytes"
    "fmt"
    "reflect"

    pb "github.com/dvyukov/go-fuzz/examples/protobuf/pb"
    "github.com/golang/protobuf/proto"
)

func main() {
    data := []byte("\xf5\a\xb3\xc4\xe1\xbf")
    v := new(pb.M18)
    err := proto.Unmarshal(data, v)
    if err != nil {
        panic(err)
    }
    var buf bytes.Buffer
    err = proto.MarshalText(&buf, v)
    if err != nil {
        fmt.Printf("failed to MarshalText: %#v\n", v)
        panic(err)
    }
    v2 := new(pb.M18)
    err = proto.UnmarshalText(buf.String(), v2)
    if err != nil {
        fmt.Printf("failed to UnmarshalText: %q\n", buf.Bytes())
        panic(err)
    }
    if !reflect.DeepEqual(v, v2) {
        fmt.Printf("v0: %#v\n", v)
        fmt.Printf("v2: %#v\n", v2)
        panic(fmt.Sprintf("non idempotent text marshal of %T", v))
    }
}

proto: failed getting extension:  unexpected EOF
v0: &example.M18{F0:(*string)(nil), XXX_extensions:map[int32]proto.Extension{126:proto.Extension{desc:(*proto.ExtensionDesc)(nil), value:interface {}(nil), enc:[]uint8{0xf5, 0x7, 0xb3, 0xc4, 0xe1, 0xbf}}}, XXX_unrecognized:[]uint8(nil)}
v2: &example.M18{F0:(*string)(nil), XXX_extensions:map[int32]proto.Extension(nil), XXX_unrecognized:[]uint8(nil)}
panic: non idempotent text marshal of *example.M18

There are 2 issues here:

1. Part of the message gets lost.
2. MarshalText prints to stderr. It should not. Either fail of be silent.

Definition of the protobuf is:

message M18 {
  optional string f0 = 1;
  extensions 100 to 199;
}

extend M18 {
  optional int32 f1 = 126;
}

on commit 34a5f24

jsonpb doesn't decode proto3 string enums

Using the jsonpb package, I can marshal a proto3 message to JSON with enums as strings.

However, I cannot go in the reverse direction. The Unmarshal function returns an error when I try to unmarshal JSON with string enums.

I expected this to work how it does with proto2 messages.

Failing round-trip example here:

package main

import (
    "github.com/golang/protobuf/jsonpb"
    pb "github.com/golang/protobuf/proto/proto3_proto"
)

func main() {
    msg := &pb.Message{Hilarity: pb.Message_SLAPSTICK}

    marshaller := jsonpb.Marshaller{EnumsAsString: true}
    data, _ := marshaller.MarshalToString(msg)

    if err := jsonpb.UnmarshalString(data, msg); err != nil {
        panic(err)
    }
}

This code panics with json: cannot unmarshal string into Go value of type proto3_proto.Message_Humour.

proto3 map empty bytes

Add this to proto3.proto in proto/proto3_proto

message MessageWithMappy {
  map<bool, bytes> byte_mapping = 3;
}

and run the Makefile in proto/proto3_proto

Then add this to size_test.go

{"map field with zero bytes", &proto3pb.MessageWithMappy{
        ByteMapping: map[bool][]byte{
            false: []byte{},
            true:  []byte{0x84, 0x78, 0x8e, 0x33, 0xcd, 0x12, 0x65, 0xa0, 0xd2, 0xca, 0xc4, 0xcd, 0x12, 0xe2, 0xcf, 0xf1, 0xbf, 0xfa, 0xf7, 0x5a, 0x3c, 0x6d, 0xdc, 0xcf, 0x9c, 0x92, 0x94, 0x3f, 0x4f, 0xb6, 0x31, 0x77, 0xe3, 0xda, 0x3f, 0x1b, 0x8e, 0x39, 0xe6, 0xa, 0x81, 0x61, 0xf0, 0xba, 0x74, 0x6},
        },
    }},

This gives the following failure

size_test.go:138: map field with zero bytes: Size(byte_mapping:<key:false value:"" > byte_mapping:<key:true value:"\204x\2163\315\022e\240\322\312\304\315\022\342\317\361\277\372\367Z<m\334\317\234\222\224?O\2661w\343\332?\033\2169\346\n\201a\360\272t\006" > ) = 56, want 7
size_test.go:139: map field with zero bytes: bytes: []byte{0x1a, 0x0, 0x0, 0x0, 0x0, 0x8, 0x0}

The error only happens with an empty byte slice.

This is probably a mistake in the way the map values try to reuse the setEncAndDec function with a combination of the proto3 syntax

case reflect.Uint8:
            p.enc = (*Buffer).enc_slice_byte
            p.dec = (*Buffer).dec_slice_byte
            p.size = size_slice_byte
            if p.proto3 {
                p.enc = (*Buffer).enc_proto3_slice_byte
                p.size = size_proto3_slice_byte
            }

Maybe this should be

case reflect.Uint8:
            p.enc = (*Buffer).enc_slice_byte
            p.dec = (*Buffer).dec_slice_byte
            p.size = size_slice_byte
            if p.proto3 && notamapvalue {
                p.enc = (*Buffer).enc_proto3_slice_byte
                p.size = size_proto3_slice_byte
            }

Rename Server to Service in the generated pb.go files

When defining a service in a proto file, the result of the compilation generates a MyNameServer instead of MyNameService. I've also opened the issue in the grpc-go repo because I don't know where this should be taken into account (grpc/grpc-go#114) I'll cose whichever is not in the proper place.

Use native map

The current implementation doesn't allow key-based indexing or the use of range. It appears that the current solution is to create a repeated K/V message. Which isn't really the same as a map.

go test failed

go test github.com/golang/protobuf/proto
# github.com/golang/protobuf/proto
proto3_test.go:38:2: no buildable Go source files in C:\go\gopkg.stable\src\github.com\golang\protobuf\proto\proto3_proto
FAIL    github.com/golang/protobuf/proto [setup failed]

Support for output with unexported struct names

This is a request to support generating code that uses lowercase struct names so they are not exported in the package they target. The motivation is to reduce the API footprint exposed to end users. For example, in a project I am working on I define a set of methods that use the protobuf structs internally, but I don't want them to be exposed themselves: http://godoc.org/github.com/chop-dbhi/origins/dal

map to json

Add this function to all_test.go

func TestMapFieldJsonMarshal(t *testing.T) {
    m := &MessageWithMap{
        NameMapping: map[int32]string{
            1: "Rob",
            4: "Ian",
            8: "Dave",
        },
    }
    _, err := json.Marshal(m)
    if err != nil {
        t.Fatal(err)
    }
}

It will fail with
json: unsupported type: map[int32]string

Is this work just not done yet or am I understanding something incorrectly or is this a bug?
I thought that json is the new text standard for proto3 and also that proto3 supports maps?
I would guess that go needs to support marshaling proto3 structures to json or am I interpreting this in the wrong way?

Nested message types

I have a Protobuf message that takes an optional message of a type I created. However, the optional message type is just a wrapper of nested types. The goal was to be able to specify a thing generically by saying that it's simply something of a Type and not a RedType or GreenType. Polymorphism.

I've messed with all of this to make it concise and easier to read.

package interfaces;

message Wrapper {
    optional Type data = 1;
}

message Type {
    message RedType {
        required int64 id;
        required string red_type_property;
    }

    message GreenType {
        required int64 id;
        required string green_type_property;
    }
}

The issue is, when I go to work with this in Go:

mydata := &interfaces.Type_RedType{
    Id: 20,
    RedTypeProperty: "propval",
}

mywrapper := &interfaces.Wrapper{
    Data: mydata,
}

Not surprisingly, you can't use something that is a Type_RedType in place of a more general Type when setting the struct value of Data.

cannot use mydata (type *interfaces.Type_RedType) as type *interfaces.Type in field value

Is there a language feature of Go I can use here? I tried a type conversion from RedType to Type without expecting much and it didn't work. Am I misusing this library? Or is this a terrible idea to begin with, which is what has landed me here?

Any & all help appreciated. Hopefully this is an easy one to answer.