ugorji / go Goto Github PK
View Code? Open in Web Editor NEWidiomatic codec and rpc lib for msgpack, cbor, json, etc. msgpack.org[Go]
License: MIT License
idiomatic codec and rpc lib for msgpack, cbor, json, etc. msgpack.org[Go]
License: MIT License
If I define a map with value type of interface{} (I'm sending data over the wire in data structures including strings and timestamps) then the map is de-serialised as a byte array:
2013/08/06 02:53:33 Buffer is [130 169 115 111 109 101 102 105 101 108 100 170 115 111 109 101 115 116 114 105 110 103 167 116 104 101 116 105 109 101 171 239 82 0 87 29 47 122 62 127 0 60]
2013/08/06 02:53:33 Decoded is map[somefield:[115 111 109 101 115 116 114 105 110 103] thetime:[239 82 0 87 29 47 122 62 127 0 60]]
The program to generate this is:
package main
import (
"github.com/ugorji/go/codec"
"time"
"reflect"
"log"
)
type timeContainer map[string]interface{}
var (
mh codec.MsgpackHandle
timeTyp = reflect.TypeOf(time.Time{})
source = timeContainer {
"somefield": "somestring",
"thetime": time.Now(),
}
)
func main() {
mh.AddExt(timeTyp, 1, mh.TimeEncodeExt, mh.TimeDecodeExt)
// Encode the value
var encode_out []byte
enc := codec.NewEncoderBytes(&encode_out, &mh)
encode_err := enc.Encode(&source)
if encode_err != nil {
log.Fatal("Ooops, error: ", encode_err)
}
log.Println("Buffer is ", encode_out)
// Decode the value
var b timeContainer
dec := codec.NewDecoderBytes(encode_out, &mh)
decode_err := dec.Decode(&b)
if decode_err != nil {
log.Fatal("Ooops, error: ", decode_err)
}
log.Printf("Decoded is %v", b)
}
Any pointers towards getting values defined as interface{} decoded properly would be great.
Is this expected to fail?
package main
import (
"bytes"
"github.com/ugorji/go/codec"
)
var msgpackHandle = new(codec.MsgpackHandle)
type A struct {
anInt int
}
type B struct {
*A
moreInt int
}
func main() {
var buf bytes.Buffer
encoder := codec.NewEncoder(&buf, msgpackHandle)
err := encoder.Encode(&B{&A{5}, 6})
if err != nil {
panic(err)
}
}
I use Go 1.4.1.
MsgpackDecoder produce int64 for unsigned integer while I set SignedInteger = false
.
https://gist.github.com/methane/cfdcce8257a6406dd2ad
Output of this code is 42, int64
.
Trying to get a struct encoded and cannot get this to work.
Haven't found any definitive examples that illustrate encoding structs.
Any ideas why this always returns an empty map?
package main
import (
"bytes"
"log"
"github.com/ugorji/go/codec"
)
type status struct {
rssi uint8 `codec:"rssi"`
lqi uint8 `codec:"lqi"`
ts uint16 `codec:"ts"`
}
func main() {
in := status{
rssi: 12,
lqi: 231,
ts: 12312,
}
//in := map[string]interface{}{"rssi": uint8(12), "lqi": uint8(231), "ts": uint16(12312)}
w := new(bytes.Buffer)
enc := codec.NewEncoder(w,
&codec.MsgpackHandle{RawToString: true, WriteExt: true})
_ = enc.Encode(in)
log.Printf("% x", w.Bytes())
}
Can you please explain how to migrate this code:
var MpackHandle codec.MsgpackHandle
MpackHandle.AddExt(reflect.TypeOf(time.Time{}), 1, MpackHandle.TimeEncodeExt, MpackHandle.TimeDecodeExt)
I tried this which doesn't work (encodeTime
is undefined):
timeEncExt := func(rv reflect.Value) ([]byte, error) {
return encodeTime(rv.Interface().(time.Time)), nil
}
timeDecExt := func(rv reflect.Value, bs []byte) error {
tt, err := decodeTime(bs)
if err == nil {
rv.Set(reflect.ValueOf(tt))
}
return err
}
MpackHandle.AddExt(reflect.TypeOf(time.Time{}), 1, timeEncExt, timeDecExt)
Here is the code I am using to encode/decode the maps. For my tests I am passing in a map with string key and value. The decoded map contains string key but []uint8 as value. I expect it to return the same map structure which was encoded.
Enocde / Decode code
func encode(msg map[string]interface{}) ([]byte, error) {
var encoded []byte
enc := codec.NewEncoderBytes(&encoded, &mh)
err := enc.Encode(msg)
if err != nil {
return nil, err
}
return encoded, nil
}
func decode(encoded []byte) (map[string]interface{}, error) {
var decoded map[string]interface{}
dec := codec.NewDecoderBytes(encoded, &mh)
err := dec.Decode(&decoded)
if err != nil {
return nil, err
}
return decoded, nil
}
Test
import (
. "gopkg.in/check.v1"
"testing"
)
// Hook up gocheck into the "go test" runner.
func Test(t *testing.T) {
TestingT(t)
}
type S struct{}
var _ = Suite(&S{})
func (s *S) TestEncodeDecode(c *C) {
msg := make(map[string]interface{})
msg["Hello"] = string("world")
encoded, err := encode(msg)
c.Assert(err, IsNil)
c.Assert(encoded, NotNil)
decoded, err := decode(encoded)
c.Assert(err, IsNil)
c.Assert(decoded, DeepEquals, msg)
}
Here is the output of the test run.
FAIL: test.go:27: S.TestEncodeDecode
test.go:35:
c.Assert(decoded, DeepEquals, msg) ... obtained map[string]interface {} = map[string]interface
{}{"Hello":[]uint8{0x77, 0x6f, 0x72, 0x6c, 0x64}} ... expected
map[string]interface {} = map[string]interface {}{"Hello":"world"}OOPS: 0 passed, 1 FAILED
--- FAIL: Test (0.00 seconds) FAIL
To create a msgpack-rpc message (http://wiki.msgpack.org/display/MSGPACK/RPC+specification) the message ID must be specified as a uint32, which seems impossible from reading the source code of the current implementation. The encoder chooses by fitting the passed uint64 in the smallest msgpack container.
Is it possible to have the encoder detect the source type and keep the size? Maybe an option in the handler to compress into smaller types could be added?
My request struct looks like this:
type Request struct {
id uint32
Method string
Params []interface{}
}
Packing is done with this method:
func (r *Request) Packed() []byte {
a := make([]interface{}, 0)
a = append(a, 0)
a = append(a, r.id)
a = append(a, r.Method)
a = append(a, r.Params)
var b []byte
var mh codec.MsgpackHandle
enc := codec.NewEncoderBytes(&b, &mh)
err := enc.Encode(a)
if err != nil {
}
return b
}
Unpacking is done as follows:
func UnpackRequest(b []byte) *Request {
var a []interface{}
mh := codec.MsgpackHandle{RawToString: true}
dec := codec.NewDecoderBytes(b, &mh)
err := dec.Decode(&a)
if err != nil {
}
r := &Request{
id: uint32(a[1].(uint64)), // Ugly conversion has to be done here.
// Prefered way:
// id: a[1].(uint32),
Method: a[2].(string),
Params: a[3].([]interface{}),
}
return r
}
Please provide a mechanism to do Steaming RPC as an extension.
I know net/rpc lacks that feature currently.
package main import ( "fmt" "github.com/ugorji/go/codec" "time" ) func main() { bincHandle := new(codec.BincHandle) var b []byte t := time.Now() if e := codec.NewEncoderBytes(&b, bincHandle).Encode(t); e != nil { panic(e) } var t2 time.Time if e := codec.NewDecoderBytes(b, bincHandle).Decode(&t2); e != nil { panic(e) } fmt.Printf("%v became %v\n", t, t2) t = time.Time{} if e := codec.NewEncoderBytes(&b, bincHandle).Encode(t); e != nil { panic(e) } if e := codec.NewDecoderBytes(b, bincHandle).Decode(&t2); e != nil { panic(e) } fmt.Printf("%v became %v\n", t, t2) }
produces
[0:zond@pelican ~]$go run tmp/test.go 2013-08-02 15:25:53.082092351 +0200 CEST became 2013-08-02 15:25:53.082092351 +0200 UTC+02:00 0001-01-01 00:00:00 +0000 UTC became 34843-02-20 00:36:16 +0000 UTC [0:zond@pelican ~]$
I'm sorry I couldn't get a smaller test case for this, I'm short on time this morning. This is trying to decode a payload from Python (hence the ugly string constant).
If you name the Bstruct's *Astruct field instead of embedding it, it decodes as expected. If you don't you get the error message:
2013/11/15 09:01:54 Failed to decode to struct:
codec.decoder: readContainerLen: Unrecognized descriptor byte: hex: 95, dec: 149
package main
import (
"github.com/ugorji/go/codec"
"bytes"
"log"
"github.com/davecgh/go-spew/spew"
)
type Astruct struct {
Abyte1 []byte
Abyte2 []byte
Auint1 uint64
Auint2 uint64
Auint3 uint8
}
type Bstruct struct {
*Astruct
Bint1 int32
Buint1 uint64
Buint2 uint64
Bint2 int32
Bint3 int32
}
type Cstruct struct {
}
type Dstruct struct {
Dmap map[string]*Bstruct
Dslice []*Cstruct
}
var test = []byte("\x92\x81\xa4test\x96\x95\xda\x00 \x13\n\x1e\x12" +
"\x15t\xf6y\x8e\xf6\xcd\xc3\xde)t2\xfc\xb7\xe0\xcf\xbd\xd9" +
"\xbd\xc2$\xdc\x01\xc5=89\xc9\xda\x00 `\xcb1\xbe+\x1aGr" +
"\x1a\x06\xef\xf3)\xc4\xcf\xef\xfcp\xac\xa9-\n\xe1TV\xb0" +
"\xf8\xd5EX\xceH$\x0b\x01\x0b\x00\x00\x00\x00\x90")
func main() {
var m codec.MsgpackHandle
d := Dstruct{}
decoder := codec.NewDecoder(bytes.NewBuffer(test), &m)
if err := decoder.Decode(&d); err != nil {
log.Println("Failed to decode to struct:", err)
} else {
log.Println(d)
}
var i interface{}
decoder = codec.NewDecoder(bytes.NewBuffer(test), &m)
if err := decoder.Decode(&i); err != nil {
log.Println("Failed to decode to interface:", err)
} else {
spew.Dump(i)
}
}
Output:
2013/11/15 09:01:54 Failed to decode to struct: codec.decoder: readContainerLen: Unrecognized descriptor byte: hex: 95, dec: 149
([]interface {}) {
(map[interface {}]interface {}) {
(string) "test": ([]interface {}) {
([]interface {}) {
([]uint8) {
00000000 13 0a 1e 12 15 74 f6 79 8e f6 cd c3 de 29 74 32 |.....t.y.....)t2|
00000010 fc b7 e0 cf bd d9 bd c2 24 dc 01 c5 3d 38 39 c9 |........$...=89.|
},
([]uint8) {
00000000 60 cb 31 be 2b 1a 47 72 1a 06 ef f3 29 c4 cf ef |`.1.+.Gr....)...|
00000010 fc 70 ac a9 2d 0a e1 54 56 b0 f8 d5 45 58 ce 48 |.p..-..TV...EX.H|
},
(int64) 36,
(int64) 11,
(int64) 1
},
(int64) 11,
(int64) 0,
(int64) 0,
(int64) 0,
(int64) 0
}
},
([]interface {}) {
}
}
how can i use msgpack in golang?
The fix to https://github.com/ugorji/go-msgpack/issues/10 that was carried over into the codec implementation was incorrect.
The 4th parameter to msgpack rpc is an array of all the arguments. However, Go requires that methods exposed through net/rpc have only a single argument. By having the "extra" array indirection on read and write, the "single argument" vs "array of arguments" problem is solved by passing an array consisting of a single argument, and both Go and msgpack are happy. This means that when using msgpack-rpc between Go and (say) Python, the more-restrictive calling conventions of Go (namely, only a single parameter) must be respected. The rpcCodec.ReadRequestBody routine can't be shared, it must be specialized. msgpackSpecRpcCodec writeCustomBody just needs to be patched.
You can test this against these trimmed-down client/servers in https://github.com/dgryski/trifles/tree/master/msgrpc
since this library support map[string]interface{} type, I tried to use it for pacakge struct, what I find is that decode/encode data dismatch, my test code:
func TestCodec(t *testing.T) {
var (
mh codec.MsgpackHandle
rw bytes.Buffer
)
mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
enc := codec.NewEncoder(&rw, &mh)
item := make(map[string]interface{})
item["key"] = int32(123)
item["key2"] = "string"
err := enc.Encode(&item)
if err == nil {
t.Log("Encode OK", "type", reflect.TypeOf(item["key"]), reflect.TypeOf(item["key2"]))
}
var item_decoded map[string]interface{}
dec := codec.NewDecoder(&rw, &mh)
err = dec.Decode(&item_decoded)
if err == nil {
t.Log("Decode OK", "type", reflect.TypeOf(item_decoded["key"]), reflect.TypeOf(item_decoded["key2"]))
}
}
the result:
proto_test.go:63: Encode OK type int32 string
proto_test.go:71: Decode OK type int64 []uint8
update:
I change the handle config mh.RawToString = true
, now the int type still not match:
proto_test.go:64: Encode OK type int32 string
proto_test.go:72: Decode OK type int64 string
msgpack spec include int8 int16 int32 many int families, this result make me confuse.
When encoding a structure to an array, an omitted field will return an error:
Error: reflect: call of reflect.Value.Type on zero Value
package main
import (
"bytes"
"fmt"
"github.com/ugorji/go/codec"
)
type SrcIdDate struct {
Sid string
SDate, EDate string
Option string `codec:",omitempty"`
}
func main() {
var (
mh codec.MsgpackHandle
buf bytes.Buffer
)
s1 := SrcIdDate{Sid: "ABC", SDate: "20000101", EDate: "20101031"}
mh.EncodeOptions.StructToArray = true
enc := codec.NewEncoder(&buf, &mh)
err := enc.Encode(s1)
if err != nil {
fmt.Printf("Error: %v\n", err)
} else {
fmt.Printf("Success: %v\n", buf.Bytes())
}
}
This doesn't occur when encoding as a map or if the field is not empty.
I can see some confusion if the optional field wasn't at the end, but I am trying to process some previously generated data.
How to reproduce:
mkdir tmp
cd tmp
cat >foo.go <<EOF
package foo
import _ "github.com/ugorji/go/codec"
EOF
go get
GOARCH=386 go build
# github.com/ugorji/go/codec
../src/github.com/ugorji/go/codec/binc.go:279: constant 4294967295 overflows int
To emphasize importance of this issue: it breaks (at least local) Appengine builds.
ERROR 2013-06-07 08:52:54,896 go_runtime.py:165] Failed to build Go application: /home/temoto/src/github.com/ugorji/go/codec/binc.go:279: constant 4294967295 overflows int
Hi, I've been looking at codec recently and was testing it out but was getting some behavior differences between it and encoding/json. Here is the output I'm getting using the same structs.
Encodings from codec generated selfer code....
{"Key":"thing"}
{"Key":"whynot","Field1":"another"}
Encodings from codec reflection based encoding....
{"_key":"thing"}
{"_id":"another","_key":"whynot"}
Encodings from encoding/json....
{"_key":"thing"}
{"_key":"whynot","_id":"another"}
Below is my example code and directory structure
root
|----main.go
|----model
|--------model.go
package main
import (
"fmt"
"github.com/ugorji/go/codec"
"encoding/json"
"./model"
)
type Test struct {
Key string `json:"_key,omitempty"`
}
type Next struct {
Test
Field1 string `json:"_id,omitempty"`
Field2 string `json:",omitempty"`
}
func main() {
var h codec.Handle = new(codec.JsonHandle)
var b []byte
d := &model.Test{ Key : "thing" }
u := &model.Next{ Field1: "another" }
f := &Test{ Key : "thing" }
t := &Next{ Field1: "another" }
u.Key = "whynot"
t.Key = "whynot" //Edited to add this line since it was missing
fmt.Println( "Encodings from codec generated selfer code...." )
b, _ = encodeCodec( d, h )
fmt.Println( string(b) )
b, _ = encodeCodec( u, h )
fmt.Println( string(b) )
fmt.Println( "Encodings from codec reflection based encoding...." )
b, _ = encodeCodec( f, h )
fmt.Println( string(b) )
b, _ = encodeCodec( t, h )
fmt.Println( string(b) )
fmt.Println( "Encodings from encoding/json...." )
b, _ = encodeJson( d )
fmt.Println( string(b) )
b, _ = encodeJson( u )
fmt.Println( string(b) )
}
func encodeCodec( v interface{}, h codec.Handle ) ( []byte, error ) {
var b []byte
enc := codec.NewEncoderBytes( &b, h )
err := enc.Encode( v )
if err != nil {
fmt.Println( err )
return nil, err
}
return b, nil
}
func encodeJson( v interface{} ) ( []byte, error ){
b, err := json.Marshal( v )
if err != nil {
fmt.Println( err )
return nil, err
}
return b, nil
}
package model
//go:generate codecgen -o codec_gen.go model.go
type Test struct {
Key string `json:"_key,omitempty"`
}
type Next struct {
Test
Field1 string `json:"_id,omitempty"`
Field2 string `json:",omitempty"`
}
This is more of a question/discussion than an issue.
MsgPack supports FixStr, as well as Str8, Str16, Str32 and also Bin8, Bin16, Bin32.
When encoding the sample from issue #22 without the bug inducing bit (specifically: the type declaration of binary).. I noticed that the byte slice is being encoded as FixStr for small lengths, and StrX (X being the accompanying size) instead of the BinX variants.
I think maybe this could be a potential issue between different clients who are expecting a specific schema.
Thoughts on this?
When i use json decode codec, can i use something like:
type A struct {
Hello float64 `json:"hello"`
World string `json:"world"`
}
?
The below snippet produces perhaps sub-optimal msgpack.
Notice that the struct is created from small numbers (even if big types), in serialization with MsgPack we should be able to create these numbers without any standalone type descriptor by using the positive FixInt type from the spec:
positive fixint 0xxxxxxx 0x00 - 0x7f
If you look at the output of the snippet you'll notice several 0xcc bytes in there describing each of the integers as a uint8 when those bytes don't need to exist.
Removing the 0xcc bytes and decoding with this package produces the expected result, so it seems to be an encoding problem only.
package main
import (
"github.com/ugorji/go/codec"
"bytes"
"log"
)
type bstruct struct {
A, B, C uint64
}
func main() {
b := bstruct{1, 2, 3}
log.Println(b)
var m codec.MsgpackHandle
m.StructToArray = true
m.WriteExt = true
var buf bytes.Buffer
encoder := codec.NewEncoder(&buf, &m)
if err := encoder.Encode(b); err != nil {
log.Fatal("Failed to encode:", err)
}
log.Printf("%#v", buf.Bytes())
}
Output:
2013/11/14 17:24:29 {1 2 3}
2013/11/14 17:24:29 []byte{0x93, 0xcc, 0x1, 0xcc, 0x2, 0xcc, 0x3}
It appears that arrays are not able to be serialized.
package main
import (
"github.com/ugorji/go/codec"
"bytes"
"log"
)
type bstruct struct {
B [3]byte
}
func main() {
b := bstruct{
[3]byte{5, 7, 8},
}
log.Println(b)
var m codec.MsgpackHandle
var buf bytes.Buffer
encoder := codec.NewEncoder(&buf, &m)
if err := encoder.Encode(b); err != nil {
log.Fatal("Failed to encode:", err)
}
}
This snippet produces:
2013/11/14 09:01:41 Failed to encode:reflect.Value.Slice: slice of unaddressable array
Are arrays not able to be serialized?
$ go get github.com/ugorji/go/codec
# github.com/ugorji/go/codec
../../ugorji/go/codec/encode.go:514: undefined: sync.Pool
../../ugorji/go/codec/encode.go:1177: undefined: sync.Pool
Should we pull from a different repos? Thanks!
Example code
var mh2 codec.MsgpackHandle
ooo := []byte{
220, 0, 117, //0xdc array 16, len 117
204, 131, 204, 169, 112, 114, 111, 102, 105, 108,
101, 105, 100, 204, 169, 48, 58, 48, 58, 49,
48, 48, 48, 49, 204, 177, 97, 116, 116, 114,
105, 98, 117, 116, 101, 115, 75, 101, 121, 76,
105, 115, 116, 204, 147, 204, 164, 110, 97, 109,
101, 204, 167, 115, 97, 118, 101, 118, 101, 114,
204, 163, 108, 118, 108, 204, 171, 112, 97, 115,
115, 116, 104, 114, 111, 117, 103, 104, 204, 217,
36, 49, 49, 51, 100, 55, 102, 51, 52, 45,
54, 100, 48, 97, 45, 52, 48, 55, 57, 45,
98, 51, 50, 53, 45, 98, 101, 51, 50, 101,
52, 98, 51, 101, 48, 97, 97,
}
dec4 := codec.NewDecoderBytes(ooo, &mh2)
var abcn []byte
dec4.Decode(&abcn)
fmt.Println("---%v", abcn)
Output abcn is empty.
i have a file, with a batch of records, each record is a map with string key and string/int/float value. then i decode this file with code below:
type MsgpackReader struct {
fp *os.File
dec *codec.Decoder
}
func NewMsgpackReader(filename string) *MsgpackReader {
fp, err := os.Open(filename)
if err != nil {
return nil
}
r := bufio.NewReader(fp)
mh := codec.MsgpackHandle{}
mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
mh.RawToString = true
return &MsgpackReader{fp: fp, dec: codec.NewDecoder(r, &mh)}
}
func (reader *MsgpackReader) Read() (record TRecord, err error) {
err = reader.dec.Decode(&record)
return
}
func (reader *MsgpackReader) Close() {
reader.fp.Close()
}
func main() {
reader := NewMsgpackReader(filename)
defer reader.Close()
for {
_, err := reader.Read()
if err != nil {
if err == io.EOF {
break
} else {
continue
}
}
}
}
this file contains 874589 records, is there any wrong? i am confused.
The following code:
package main import ( "fmt" "github.com/zond/go/codec" ) func main() { bincHandle := new(codec.BincHandle) var b []byte sl := make([]string, 0) if e := codec.NewEncoderBytes(&b, bincHandle).Encode(sl); e != nil { panic(e) } var sl2 []string if e := codec.NewDecoderBytes(b, bincHandle).Decode(&sl); e != nil { panic(e) } fmt.Printf("%#v became %#v\n", sl, sl2) }
Produces:
[]string{} became []string(nil)
Empty slices are not always nil..
I need to do a "canonical" encoding in CBOR -- much as described in http://tools.ietf.org/html/rfc7049#section-3.9. In other words, I need to make some calls to the encoder that when given the same inputs, have identical binary outputs. (Though it's probably not relevant, in the bigger picture, this is for feeding into a cryptographic hash, so yes, the input->binaryoutput mapping has to be identical every time.)
How can I do this?
I think codec already mostly does the right thing with ints and so forth, but maps are a sticking point. Here's what I considered so far:
cborEncDriver
has enough exported methods on it that I might be able to get the trick done by hand (and that'd be fine; since this is finicky work with byte layouts, it's fair if my code has to do fairly manual byte concatenations, as long as it's not totally re-copying everything down to the magic constants and int/type layouts)... except it (and a bunch of other important magic constants) are not exported, so I can't just initialize it.fast-path.go.tmpl
seemed like it offered a glimmer of hope for a second, because it's doing the things I need to do... but it looks like that code only works from within the codec package (lo and behold it's using those unexported encoder drivers).GenHelperEncoder
to get access to all the fun stuff. It's documented as not for use, but...Option 4 is the only thing I've found that's capable of doing what's needed to output a consistent/canonical form. Is there anything I've missed that's a better way of addressing this? Are there any code changes I could help PR that would make this better?
hi, I have a simple struct pass from cpp to golang, struct such as
struct { std::string A; int B };
here's problem:
msgpack cpp lib will encode such struct to array, but golang lib will decode map to struct automatically, not array.
here's what I want:
I hope golang lib will decode such array to golang struct, just omit field name.
is there any way to do such thing?
or what's your suggestion?
The new changes to fast-path slices and maps has caused nil values to be replaced with non-nil zero values of the fields. This has caused lots of pain for us as the input is no longer equal to the output when serializing / deserializing. We've forked and rolled back to 71c2886
When running codecgen against the ffjson test set I got this error:
c:\GoPath\src\github.com\pquerna\ffjson\tests>codecgen -o ff_codec.go ff.go
codecgen error: Error running go run codecgen-1424087427033171900.generated.go.
Error: exit status 1. stdout/err: panic: 27217:22: expected '}', found 'IDENT' i
nt (and 10 more errors)
goroutine 1 [running]:
main.main()
c:/GoPath/src/github.com/pquerna/ffjson/tests/codecgen-14240874270331719
00.generated.go:422 +0x1103f
exit status 2
go get -u github.com/pquerna/ffjson
cd github.com/pquerna/ffjson/tests (in gopath)
codecgen -o ff_codec.go ff.go
Also, you might want to update the information on your blog - all of the points you mention about ffjson is no longer true. Decoding is now supported, struct renaming works, and generating values.go:TestStruct works.
Running codecgen results in the following output:
func init() {
if codec1978.GenVersion != 1 {
_, file, _, _ := runtime.Caller(0)
err := fmt.Errorf("codecgen version mismatch: current: %v, need %v. Re-generate file: %v",
1, codec1978.GenVersion, file)
panic(err)
}
if false { // reference the types, but skip this branch at build/run time
= }
}
The stray = on the penultimate line is incorrect and stops compilation. Removing this seems to result in correct code.
Hi,
I has this snippet of code:
type Encoder interface {
Encode(v interface{}) error
}
type Decoder interface {
Decode(v interface{}) error
}
type (
EncoderFactory func(w io.Writer) Encoder
DecoderFactory func(r io.Reader) Decoder
)
type Codec struct {
EncodingName string
NewEncoder EncoderFactory
NewDecoder DecoderFactory
}
var JsonCodec = Codec{
"json",
func(w io.Writer) Encoder {
return json.NewEncoder(w)
},
func(r io.Reader) Decoder {
return json.NewDecoder(r)
},
}
var MsgpackCodec = Codec{
"msgpack",
func(w io.Writer) Encoder {
var mh codec.MsgpackHandle
return codec.NewEncoder(w, &mh)
},
func(r io.Reader) Decoder {
var mh codec.MsgpackHandle
return codec.NewDecoder(r, &mh)
},
}
so that I can use either json or msgpack interchangeably. Now to my surprise, when I pass interface{}
or map[string]interface{}
to Decode, I get different results from encoding/json
Decoder and your codec
Decoder, namely the former one returns values as strings while the latter one as []uint8
. Is that desired? Is it for some kind of performance reasons? I think that it would be nice to be somehow compatible...
Thanks!
Regards,
Ondrej Kupka
The below snippet fails to decode with the error:
2013/11/15 09:16:22 Could not decode a: codec.decoder: readContainerLen:
Unrecognized descriptor byte: hex: 90, dec: 144
Replacing the []byte field with with an array of pointers to structs, or even []int makes it decode without problems.
package main
import (
"github.com/ugorji/go/codec"
"log"
"bytes"
)
var test1 = []byte{0x91, 0x90}
type Astruct struct {
Anarray []byte
}
func main() {
var mph codec.MsgpackHandle
var a = Astruct{}
da := codec.NewDecoder(bytes.NewBuffer(test1), &mph)
if err := da.Decode(&a); err != nil {
log.Fatalln("Could not decode a:", err)
}
log.Println(a)
}
I did some benchmarks of RPC encoding speeds.
https://gist.github.com/pcdummy/6168792
Do you know why Binc and MsgPack perform that bad?
Please, document is it safe to declare a global MsgpackHandle and/or Encoder/Decoder objects and use them concurrently from different goroutines.
Could it support something like this:
rpcServer := rpc.NewServer()
rpcServer.Register(mything)
rpcServer.ServeCodec(rpcCodec)
rpcServer.Accept(listener)
I can't see a way to get the rpcCodec with already accepting a connection though.
It would be nice if the import path mapped directly to a URL to the source code.
Currently, it doesn't, and for no good reason. Initially, we used a repository name of go, with the
hope of putting multiple packages into the same repository. However, in retrospect, that is not a
good idea; one repository should only hold one package, with possible sub-packages which relate
directly to that package.
The plan of action is:
The new codec/fail.go file will contain
We hope to do this soon, likely by the end of january.
At f8f25d7 with go1.2rc3 (amd64/linux), I can encode a slice of interface R pointers. But I cannot decode them. Is there a trick to this? Example:
package main
import (
"bytes"
"fmt"
"github.com/ugorji/go/codec"
)
type R interface {
String() string
}
type Block struct {
Name string
Rs []*R
}
type Registration struct {
Reg string
}
func (r Registration) String() string {
return fmt.Sprintf("Registration: Reg: '%s'", r.Reg)
}
func NewBlock(name string) *Block {
blk := &Block{
Name: name,
Rs: make([]*R, 0),
}
return blk
}
func main() {
name := "hello"
blk := NewBlock(name)
var r1, r2 R
r1 = Registration{Reg: "and-a-one"}
r2 = Registration{Reg: "and-a-two"}
blk.Rs = append(blk.Rs, &r1)
blk.Rs = append(blk.Rs, &r2)
bincEncodeDecode(blk)
}
func bincEncodeDecode(blk *Block) {
var bh codec.BincHandle
b := make([]byte, 0)
//var b []byte
enc := codec.NewEncoderBytes(&b, &bh)
err := enc.Encode(blk)
if err != nil {
panic(err)
}
fmt.Printf("encoded blk in bytes is: %v\n", b)
fmt.Printf("pre-encoding blk is: %v\n", blk)
fmt.Printf("pre-encoding blk.name is: '%v'\n", blk.Name)
//var blk2 interface{}
blk2 := NewBlock("")
dec := codec.NewDecoder(bytes.NewBuffer(b), &bh)
err = dec.Decode(blk2)
if err != nil {
panic(err) // panic: codec.decoder: decodeValue: Cannot decode non-nil codec value into nil main.R (1 methods)
}
fmt.Printf("decoded blk2 is: %v\n", blk2)
fmt.Printf("blk2.name is: '%v'\n", blk2.Name)
/* output:
go run bincdecode.go
encoded blk in bytes is: [118 180 1 4 78 97 109 101 73 104 101 108 108 111 180 2 2 82 115 102 117 180 3 3 82 101 103 77 97 110 100 45 97 45 111 110 101 117 176 3 77 97 110 100 45 97 45 116 119 111]
pre-encoding blk is: &{hello [0xc21000aa20 0xc21000aa30]}
pre-encoding blk.name is: 'hello'
panic: codec.decoder: decodeValue: Cannot decode non-nil codec value into nil main.R (1 methods)
goroutine 1 [running]:
runtime.panic(0x669080, 0xc21000ab90)
/usr/local/go1.2rc3/go/src/pkg/runtime/panic.c:266 +0xb6
main.bincEncodeDecode(0xc210066810)
/home/me/dev/bincdecode.go:71 +0x4f2
main.main()
/home/me/dev/bincdecode.go:45 +0x327
exit status 2
Compilation exited abnormally with code 1 at Mon Dec 16 20:54:28
*/
}
This records the issue found by @mattrichards in PR #54 .
errors could occur although some bytes may have been successfully read.
Hello,
I'm trying to use your MessagePack implementation to encode an decode a map[string]interface{} but the behavior I'm getting isn't intuitive and I can't find any doc about it.
Basically here's what happen:
It looks like the string was encoded or decoded as a slice of bytes or something similar.
Ideally I would like the map value to be encoded and decoded as a UTF-8 string, is there something special I'm supposed to do to get the Encoder/Decoder to behave this way?
I'm new to go so maybe I'm not understanding the type system very well though, sorry if that problem isn't related to the MessagePack implementation.
Thanks for your time.
package main
import (
"github.com/ugorji/go/codec"
"bytes"
"log"
)
type A struct {
Aint int
}
type B struct {
*A
Bint int
}
type C struct {
*A
Cint int
}
type D struct {
Structs map[string]*B
Array []*C
}
func main() {
obj := &D{
Structs: map[string]*B{
"test": &B{&A{5}, 6},
},
}
var m codec.MsgpackHandle
m.StructToArray = true
var buf bytes.Buffer
encoder := codec.NewEncoder(&buf, &m)
if err := encoder.Encode(obj); err != nil {
log.Fatal("Failed to encode:", err)
}
decoder := codec.NewDecoder(&buf, &m)
d := D{}
if err := decoder.Decode(&d); err != nil {
log.Fatal("Failed to decode:", err)
}
log.Println(d)
}
This code receives the error:
reflect: call of reflect.Value.Field on ptr Value
It runs correctly when the struct pointers are named and not embedded OR the StructToArray option is turned off.
Hi,
I am trying to encode a struct that contains a map[string]<comparable struct>
(which is still making the whole struct incomparable). I am getting
panic: runtime error: comparing uncomparable type data.App
goroutine 13 [running]:
runtime.panic(0x40b540, 0xc21007aa50)
/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/ugorji/go/codec.isEmptyValue(0x4596a0, 0xc210051750, 0x192, 0x0)
/Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/helper_internal.go:60 +0x34d
github.com/ugorji/go/codec.(*encFnInfo).kStruct(0xc2100b8450, 0x4133a0, 0xc210051750, 0x192)
/Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:472 +0x27c
github.com/ugorji/go/codec.(*Encoder).encodeValue(0xc2100abcb0, 0x4133a0, 0xc210051750, 0x192)
/Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:893 +0x45a
github.com/ugorji/go/codec.(*Encoder).encode(0xc2100abcb0, 0x4133a0, 0xc210051750)
/Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:808 +0x1f1
github.com/ugorji/go/codec.(*Encoder).Encode(0xc2100abcb0, 0x4133a0, 0xc210051750, 0x0, 0x0)
/Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:692 +0x69
where
type App struct {
Id bson.ObjectId `bson:"_id" codec:"-"`
Alias string `bson:"alias" codec:"alias"`
Name string `bson:"name" codec:"name,omitempty"`
Version string `bson:"version" codec:"version,omitempty"`
Description string `bson:"description" codec:"description,omitempty"`
Repository string `bson:"repository" codec:"repository,omitempty"`
Vars Variables `bson:"vars,omitempty" codec:"vars,omitempty" json:"variables"`
Enabled bool `bson:"enabled" codec:"enabled,omitempty"`
Status string `bson:"-" codec:"status,omitempty"`
}
type Variables map[string]Variable
type Variable struct {
Usage string `bson:"usage" codec:"usage"`
Type string `bson:"type" codec:"type"`
Secret bool `bson:"secret,omitempty" codec:"secret,omitempty"`
Optional bool `bson:"optional,omitempty" codec:"optional,omitempty"`
Value string `bson:"value" codec:"value"`
}
Hi, I often use your great library, thanks.
I encountered a failure in decoding long []byte with BincHandle.
The following code is an example to reproduce the failure.
I expect that the both lengths should be 65536, but the length of decoded one is 256.
package main
import (
"fmt"
"github.com/ugorji/go/codec"
"os"
)
func main() {
b := make([]byte, 0)
var h codec.BincHandle
enc := codec.NewEncoderBytes(&b, &h)
sample := make([]byte, 65536)
err := enc.Encode(sample)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
dec := codec.NewDecoderBytes(b, &h)
sampleDecoded := make([]byte, 0)
err = dec.Decode(&sample_test)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Printf("len=%d\n", len(sample))
fmt.Printf("len=%d\n", len(sampleDecoded)
}
Hi there--thanks for providing this library!
I'm in the process of upgrading from your now-deprecated go-msgpack library, where I was using the convenience methods Marshal and Unmarshal. Using this new library, Encode works as a perfect drop in replacement for Marshal, but when I try to decode I get the following runtime error:
"interface is []uint8, not string"
where casting to a string (as it was encoding as a string) worked before. Is this a bug, or has the usage changed?
My Marshal/Unmarshal replacement are:
var mh codec.MsgpackHandle
//Convenience methods for simple encoding/decoding
func MsgpackEncode(val interface{}) (packed []byte, err error) {
enc := codec.NewEncoderBytes(&packed, &mh)
err = enc.Encode(val)
return
}
func MsgpackDecode(packed []byte, val interface{}) (err error) {
dec := codec.NewDecoderBytes(packed, &mh)
err = dec.Decode(&val)
return
}
Appreciate the help.
Notice the type definition of binary to a []byte. This causes the []byte to not be serialized as a regular []uint8, but as a needlessly verbose array of bytes.
package main
import (
"github.com/ugorji/go/codec"
"bytes"
"log"
)
type binary []byte
type bstruct struct {
B binary
}
func main() {
b := bstruct{
[]byte{5, 7, 8},
}
log.Println(b)
var m codec.MsgpackHandle
m.StructToArray = true
var buf bytes.Buffer
encoder := codec.NewEncoder(&buf, &m)
if err := encoder.Encode(b); err != nil {
log.Fatal("Failed to encode:", err)
}
log.Printf("%#v", buf.Bytes())
decoder := codec.NewDecoder(&buf, &m)
b = bstruct{}
if err := decoder.Decode(&b); err != nil {
log.Fatal("Failed to decode:", err)
}
log.Println(b)
}
The output is as follows:
2013/11/14 09:05:01 {[5 7 8]}
2013/11/14 09:05:01 []byte{0x91, 0x93, 0xcc, 0x5, 0xcc, 0x7, 0xcc, 0x8}
2013/11/14 09:05:01 {[5 7 8]}
Notice the 0xcc in there, describing the single byte each time for all bytes in the array.
Not strictly an issue, but is there anywhere a working example of how to encode/ deconde from start to end? Really does not matter into the details, but I can't figure out how one it's even supposed to start.
It's most probably me being idiot but I can't figure out from the example in the readme, which to my newbie eyes has some obscure things.
:D
Currently pointer types are not valid for declaring extended types. This can create problems when the desired output is to set an interface value with a pointer value. To solve this problem for my case I simply made all decoding into interfaces as pointers, but I think a more general solution is needed.
My solution (still break unit tests) dmcgowan@5d26f5f
Needs to separate map k:v pairs with commas - ',' - use case below w/o codecgen output. To complete "go test" case run "codecgen -o codecdemo_codec.go codecdemo.go" first.
codecdemo.go:
package codecdemo
import (
"github.com/ugorji/go/codec"
"strconv"
"time"
)
type MapVal struct {
Name string
Time time.Time
}
type Entry struct {
Vals map[string]*MapVal
}
type Wrapper struct {
Entries []*Entry
}
func Codecdemo() ([]byte, error) {
w := new(Wrapper)
w.Entries = make([]_Entry, 1)
e := new(Entry)
w.Entries[0] = e
w.Entries[0].Vals = make(map[string]_MapVal, 0)
for i := 0; i < 4; i++ {
m := new(MapVal)
m.Name = strconv.Itoa(i + 1)
m.Time = time.Now()
w.Entries[0].Vals[m.Name] = m
}
b := make([]byte, 0)
h := new(codec.JsonHandle)
enc := codec.NewEncoderBytes(&b, h)
if err := enc.Encode(w); err != nil {
return nil, err
}
return b, nil
}
codecdemo_test.go:
import (
"encoding/json"
"testing"
)
func TestCodecDemo(t *testing.T) {
val, err := Codecdemo()
if err != nil {
t.Fatal(err)
}
m := new(Wrapper)
if err = json.Unmarshal(val, m); err != nil {
println("val:", string(val))
t.Fatal(err)
}
println("val:", string(val))
}
Latest version is causing issues for us, as we now get the following:
/opt/gopath/src/github.com/ugorji/go/codec/helper.go:504 (0x51c024)
com/ugorji/go/codec.panicToErr: debug.PrintStack()
/opt/go/src/pkg/runtime/panic.c:229 (0x413f61)
panic: reflect·call(d->fn, (byte*)d->args, d->siz);
/opt/go/src/pkg/runtime/iface.c:564 (0x40ae58)
ifacehash1: runtime·panic(err);
/opt/go/src/pkg/runtime/iface.c:584 (0x40af59)
efacehash: return ifacehash1(a.data, a.type, h);
/opt/go/src/pkg/runtime/alg.c:387 (0x4032eb)
nilinterhash: *h = runtime·efacehash(*(Eface*)a, *h ^ M0) * M1;
/opt/go/src/pkg/runtime/hashmap.c:597 (0x40834d)
hash_insert: t->key->alg->hash(&hash, t->key->size, key);
/opt/go/src/pkg/runtime/hashmap.c:1272 (0x4099d7)
mapassign: hash_insert(t, h, ak, av);
/opt/go/src/pkg/runtime/hashmap.c:1301 (0x409a5d)
mapassign1: runtime·mapassign(t, h, ak, av);
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:904 (0x51563f)
com/ugorji/go/codec.(*Decoder).decMapIntfIntf: m[mk] = mv
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:682 (0x5141ff)
com/ugorji/go/codec.(*Decoder).decode: d.decMapIntfIntf(v)
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:341 (0x511b2b)
com/ugorji/go/codec.(*decFnInfo).kInterface: f.d.decode(v)
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:817 (0x51496a)
com/ugorji/go/codec.(*Decoder).decodeValue: fn.f(fn.i, rv)
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:687 (0x51402f)
com/ugorji/go/codec.(*Decoder).decode: d.decodeValue(reflect.ValueOf(iv).Elem())
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:626 (0x5135e7)
com/ugorji/go/codec.(*Decoder).Decode: d.decode(v)
/opt/gopath/src/github.com/ugorji/go/codec/rpc.go:79 (0x51fbc1)
com/ugorji/go/codec.(*rpcCodec).read: return c.dec.Decode(obj)
/opt/gopath/src/github.com/ugorji/go/codec/rpc.go:87 (0x51fc69)
com/ugorji/go/codec.(*rpcCodec).ReadResponseBody: return c.read(body)
/opt/go/src/pkg/net/rpc/client.go:138 (0x4ac74e)
(*Client).input: err = client.codec.ReadResponseBody(call.Reply)
/opt/go/src/pkg/runtime/proc.c:1223 (0x417ae0)
goexit: runtime·goexit(void)
2013/10/17 00:52:38 rpc: client protocol error: runtime error: hash of unhashable type []uint8
Still trying to get a minimal test case, but I can reproduce this in our project test suite.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.