cloudwego / dynamicgo

Dynamically and efficiently operate RPC data for Go

Dynamic-Go

Dynamically operating data for Go. Aiming at reducing serialization/deserializtion process thus it can be fast as much as possible.

Introduction

Dynamic-Go for Thrift protocol: introduction.md.

Dynamic-Go for Protobuf protocol: introduction.md

Usage

thrift

Thrift IDL parser and message operators. It can parse thrift IDL in runtime and handle thrift data in generic way.

thrift/generic

Reflection APIs to search, modify, deserialize, serialize thrift value with or without runtime type descriptor.

thrift/base

The meta data about message transportation, including caller, address, log-id, etc. It is mainly used for conv (protocol convertion) modules.

thrift/annotation

Built-in implementation of thrid-party annotations, see thrift_idl_annotation_standards. They are mainly used for conv (protocol convertion) modules.

proto

Protobuf IDL parser and message operators. It can parse protobuf IDL in runtime and handle protobuf data in generic way.

proto/generic

Reflection APIs to search, modify, deserialize, serialize protobuf value with or without runtime descriptor.

proto/protowire

Protobuf data encode and decode APIs. It parses and formats the low-level raw wire encoding. It is modified from Protobuf official code encoding/protowire.

proto/binary

BinaryProtocol tool for Protobuf Protocol. It can read, wirte and skip fields directly on binary data of protobuf message.

http

Http request/response wrapper interfaces. They are mainly used to pass http values on http<>thrift conversion.

conv

Protocol convertors. Based on reflecting ability of thrift, json and protobuf modules, it can convert message from one protocol into another.

conv/j2t

Convert JSON value or JSON-body HTTP request into thrift message.

conv/t2j

Convert thrift message to JSON value or JSON-body HTTP response.

conv/j2p

Convert JSON value into protobuf message.

conv/p2j

Convert protobuf message into JSON value.

dynamicgo's Issues

对thrift map类型，支持添加新的key-value对

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

UnwrapBinaryMessage return EOF error when biz resp is nil

Describe the bug

thrift.UnwrapBinaryMessage could not unwrap empty biz buf

To Reproduce

func TestBinaryUnwrap(t *testing.T) {
	b := BinaryProtocol{
		Buf:  nil,
		Read: 0,
	}
	var i32 int32 = -2147418110
	b.WriteI32(i32)
	b.WriteString("DummyNew")
	b.WriteI32(1)
	b.WriteByte(byte(STOP))
	_, _, _, _, _, err := UnwrapBinaryMessage(b.Buf)

	t.Errorf("UnwrapBinaryMessage() error = %v", err)

}

Steps to reproduce the behavior:

run the test case

Expected behavior

err is nil and body is empty slice bytes

Screenshots

If applicable, add screenshots to help explain your problem.

Version:

v0.3.0

Environment:

Additional context

Add any other context about the problem here.

Proposal: support high-performance generic-call SDK for `protobuf` protocol

Background

Like Thrift, protobuf also needs a easy-to-use and high-performance generic-calling system in practice.

Goals

• Implement a generic calling SDK based on protobuf protocol, supporting both server and client
• Meet primary needs for generic calling, including routing (optional)、type-describing、protocol-conversion、dynamic-reflection (searching and serialization/deserialization)
• High performance and wide compatiblity
• (optional) Be compatible with dynamicgo's thrift/generic API in the aspect of dynamic reflection.

Alternatives

• Kitex's protobuf generic-calling SDK is applicable only on HTTP(json)-conversion mode, and performance is not well enough.
• Offcial SDK support dynamic reflection and json<>protobuf conversion both, but alse has performance problem and not compatible with DSL tools like GraphGL well enough.

Desgin Advices

• For type-describing, we could use official implementation, considering the compatibility. But this implementation will have some impact on performance (not sure) ?
• For protocol-conversion, we can implement it using the way dynamicgo/conv does —— converting in place and sequentially, without any intermediator
• For dynamic-reflection, we can still split it into two APIs -- searching (based on skipping) and serializing/deserializing (based on DOM). For searching, one important thing is the speed of skipping algorithm —— I'm afraid that varint encoding has big impact on it. For full-load serializing/deserializing, I want to keep the same memory model with Thrift's DOM, for future conversion of these two —— in application on DSL like GraphGL.

incorrect spell of Descriptor

NewDescritorXXX in thrift/idl.go everywhere should be NewDescriptorXXX :)

protobuf 增加从byte数组中读取指定数量field的函数

protobuf的编解码属于属于位置不可移动类型. 如果能确定编码的时候，某几个字段在生成的字节数组，那么，对于转发或者代理之类的网关，就可以不需要解码整个消息，之需要解码前面几个字段。

生成器生成protobuf代码的时候, 固定写入字段的顺序(编码)，这是可以固定的。另外一个是, 在需要代理转发的地方，根据二进制直接读取前面需要的字段. 目前，dynamicgo中有comsumefixed64这样的方法可以读取用一个字段，是否可以增加，连续读取几个字段的方法呢

[conv/j2p] not support unknown-fields of JSON

dynamicgo/conv/j2p/decode.go

Line 420 in 497e7b3

curDesc = *rootDesc.ByJSONName(key)

this code may panic if parsed JSON key is not defined in IDL. need repair it.

编译报错

编译报错

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior:

1. 在kitex rpc项目里使用这个0.4.0的库
2. go build -o output/bin/${RUN_NAME} '...'

报错信息
go: downloading github.com/flyerxp/webrpclib/v2 v2.0.5
go: downloading github.com/cloudwego/thriftgo v0.3.17
go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.63.13

github.com/cloudwego/kitex/pkg/generic/thrift

/root/go/pkg/mod/github.com/cloudwego/[email protected]/pkg/generic/thrift/http_go116plus_amd64.go:69:67: field.Type().Type().ToThriftTType undefined (type "github.com/cloudwego/dynamicgo/thrift".Type has no field or method ToThriftTType)
/root/go/pkg/mod/github.com/cloudwego/[email protected]/pkg/generic/thrift/json_go116plus_amd64.go:105:67: field.Type().Type().ToThriftTType undefined (type "github.com/cloudwego/dynamicgo/thrift".Type has no field or method ToThriftTType)

Version:
go.mod 配置
github.com/cloudwego/kitex v0.10.3
github.com/cloudwego/dynamicgo v0.4.0 // indirect

Environment:

The output of go env.

**
GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY='github.com/flyerxp/shopping.shop,github.com/flyerxp/shopping.shop,github.com/flyerxp/web.www,github.com/flyerxp/globalStruct,github.com/flyerxp/content.news.rpc,github.com/flyerxp/shopping.shop.rpc,github.com/flyerxp/shopping.report,github.com/flyerxp/shopping.pay.rpc'
GONOSUMDB='github.com/flyerxp/shopping.shop,github.com/flyerxp/shopping.shop,github.com/flyerxp/web.www,github.com/flyerxp/globalStruct,github.com/flyerxp/content.news.rpc,github.com/flyerxp/shopping.shop.rpc,github.com/flyerxp/shopping.report,github.com/flyerxp/shopping.pay.rpc'
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE='github.com/flyerxp/shopping.shop,github.com/flyerxp/shopping.shop,github.com/flyerxp/web.www,github.com/flyerxp/globalStruct,github.com/flyerxp/content.news.rpc,github.com/flyerxp/shopping.shop.rpc,github.com/flyerxp/shopping.report,github.com/flyerxp/shopping.pay.rpc'
GOPROXY='https://goproxy.cn,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/www/site/code/shop/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3744700916=/tmp/go-build -gno-record-gcc-switches'
**

A faster function to replace node by path

Describe the solution you'd like

Add a new function to replace node field based on the old one, and use only one search by path.

The function may be like:

// File: `thrift/generic/node.go`
func (self *Node) ReplaceByPath(f func(Node) Node, path ...Path) (exist bool, err error) {
	if len(path) == 0 {
		*self = f(*self)
		return true, nil
	}
	if err := self.Check(); err != nil {
		return false, err
	}
	// search source node by path
	var v = self.GetByPath(path...)
	if v.IsError() {
		return false, v
	} else {
		exist = true
	}
	sub := f(v)
	if sub.IsError() {
		return true, sub
	}
	err = self.replace(v, sub)
	return
}

proto/generic Interface 方法增加对 ENUM类型的支持

在使用过程中发现，尝试对二进制数据解码后并转化为 map 类型数据的时候，不支持 ENUM 类型

当初是因为设计上的考虑没有支持 ENUM 么？

是否能够增加如下支持：

R_X86_64_PC32 against undefined symbol `github.com/cloudwego/dynamicgo/internal/native/avx2.__quote'

编译出错，
2023/10/24 22:11:24 go: downloading github.com/cloudwego/dynamicgo v0.1.3

2023/10/24 22:11:31 + export GO111MODULE=off
2023/10/24 22:11:31 + GO111MODULE=off
2023/10/24 22:11:31 + myfile=./vendor/golang.org/x/net/trace/trace.go
2023/10/24 22:11:31 + startLine=120
2023/10/24 22:11:31 + endLine=132
2023/10/24 22:11:31 + [[ linux-gnu =~ ^darwin ]]
2023/10/24 22:11:31 + '[' -f ./vendor/golang.org/x/net/trace/trace.go ']'
2023/10/24 22:11:31 + sed -i '120,132s/^////' ./vendor/golang.org/x/net/trace/trace.go
2023/10/24 22:11:31 + startLine=74
2023/10/24 22:11:31 + endLine=74
2023/10/24 22:11:31 + [[ linux-gnu =~ ^darwin ]]
2023/10/24 22:11:31 + '[' -f ./vendor/golang.org/x/net/trace/trace.go ']'
2023/10/24 22:11:31 + sed -i '74,74s/^////' ./vendor/golang.org/x/net/trace/trace.go
2023/10/24 22:11:31 + mkdir -p output
2023/10/24 22:11:31 + go build -buildmode=plugin -o ./output/stress.so main.go
2023/10/24 22:13:34 # command-line-arguments
2023/10/24 22:13:34 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
2023/10/24 22:13:34 /usr/bin/ld: /tmp/go-link-419334024/go.o: relocation R_X86_64_PC32 against undefined symbol `github.com/cloudwego/dynamicgo/internal/native/avx2.__quote' can not be used when making a shared object; recompile with -fPIC
2023/10/24 22:13:34 /usr/bin/ld: final link failed: Bad value
2023/10/24 22:13:34 collect2: error: ld returned 1 exit status

Empty list in json string causes err in j2p

Describe the bug

When there is a value of [] in json string for a repeated field in protobuf definition, j2p will return err.

To Reproduce

func main() { content := syntax = "proto3";

package pb3;
option go_package = "pb/example";

message ExampleScalarsReq {
optional string Msg = 1;
optional double Cookie = 2;
string Path = 3;
repeated string Query = 4;
bool Header = 5;
int64 Code = 6;
}

message ExampleScalarsResp {
string Msg = 1;
optional double Cookie = 2;
int32 Status = 3;
optional bool Header = 4;
int64 Code = 5;
double Subfix = 6;
}

service TestService {
rpc ScalarsMethodTest(ExampleScalarsReq) returns (ExampleScalarsResp);
}`
opts := proto.Options{}
descriptor, err := opts.NewDesccriptorFromContent(context.Background(), "bugreport.proto", content, map[string]string{})
if err != nil {
panic(err)
}
typeDescriptor := descriptor.Methods()["ScalarsMethodTest"].Input()

input := []byte(`{"Msg": "msg", "Query": [], "Header": false, "Code": 0}`)
cv := j2p.NewBinaryConv(conv.Options{})
out, err := cv.Do(context.Background(), typeDescriptor, input)
if err != nil {
	panic(err)
}
fmt.Println(string(out))

}
`

Expected behavior

Empty list in json can be ignored or well handled when converting to protobuf.

Screenshots

Version:

v0.4.4

Environment:

The output of go env.

Additional context

Remove the "Query": [] or put a value in the list, then it will return without error