cloudwego / frugal Goto Github PK
View Code? Open in Web Editor NEWA very fast dynamic Thrift serializer & deserializer.
License: Apache License 2.0
A very fast dynamic Thrift serializer & deserializer.
License: Apache License 2.0
Describe the bug
Your dependency github.com/chenzhuoyu/iasm
is unlicensed. As a result cloudwego/frugal
cannot be used in MIT / Apache 2 licensed projects.
Please update github.com/chenzhuoyu/iasm
to v0.9.0
which added a license. Then create a new release.
Please also update cloudwego/kitex
and other projects to use the new cloudwego/frugal
release with the updated dependency.
谢谢
Bernd, Maintainer of Dapr (dapr.io) -- github.com/dapr
无需生成代码的意思是什么
我理解直接通过idl就可以调用接口,可示例让我感觉很不清楚
可以增加一些示例么
无需生成 kitex_gen 么 我看示例也有引用呀
还是 将 kitex_gen换成 thriftgo生成的代码呢
可以给一些清楚的示例 仅转换了一个结构体 我没看懂 如何 无需生成代码
🚩GLCC 高校编程夏令营赛题
👏欢迎大家参与~
GLCC 是由**计算机学会举办的高校开源编程夏令营活动,活动为项目准备了开源奖学金。请在以下链接中报名参与⬇️
报名链接: https://www.gitlink.org.cn/glcc/subjects/detail/161
关于issue相关问题、解题思路的答疑将在群内进行
1、题目描述
Frugal 是字节自研的一款高性能 Thrift 库,旨在使用 JIT 技术来加速 Thrift 的编解码过程。由于 Thrift 采用了大端序,而 x86_64 等常见的处理器架构均为小端序,因此对于 list 这一类的数据类型需要一个循环来实现小端到大端的转换。而这个过程如果能够利用 SIMD 指令的话,可以极大的提高性能。
2、编码任务
3、技能要求和编程语言
4、预期完成结果
Describe the bug
I cannot compile my project with go 1.16.
Error message:
15# github.com/cloudwego/frugal/internal/rt
16../compile_path/pkg/mod/github.com/cloudwego/[email protected]/internal/rt/runtime.go:120:14: undefined: reflect.Pointer
The reflect.Pointer
is added to go in 1.18 golang/go@17910ed . But this package's mod version is 1.15, which has caused some misunderstandings.
Please use reflect.Ptr
to ensure compatibility with lower versions.
To Reproduce
Build this project with a version of go lower than 1.18.
Describe the bug
Can't use frugal to encode/decode on my MacBook Pro (M1)
To Reproduce
Steps to reproduce the behavior:
A local Docker is need for 'testcontainers'.
package main
import (
"context"
"fmt"
"time"
"github.com/cloudwego/frugal"
"github.com/google/uuid"
"github.com/nats-io/nats.go"
"github.com/testcontainers/testcontainers-go"
natscontainer "github.com/testcontainers/testcontainers-go/modules/nats"
)
const (
SubjectEventUserRegister = "event.user.register"
)
type EventUserRegister struct {
UserID string `frugal:"1,required"`
RegisterTime int64 `frugal:"2,required"`
}
func Encode[T any](data *T) []byte {
bytes := make([]byte, frugal.EncodedSize(data))
_ = must(frugal.EncodeObject(bytes, nil, data))
return bytes
}
func Decode[T any](bytes []byte) *T {
var data T
_ = must(frugal.DecodeObject(bytes, &data))
return &data
}
func main() {
ctx := context.Background()
natsContainer := must(natscontainer.RunContainer(ctx,
testcontainers.WithImage("nats:2.9"),
))
should(natsContainer.IsRunning())
nc := must(nats.Connect(must(natsContainer.ConnectionString(ctx))))
should(nc.IsConnected())
_ = must(nc.Subscribe(SubjectEventUserRegister, func(msg *nats.Msg) {
event := Decode[EventUserRegister](msg.Data)
fmt.Printf("%#v\n", event)
}))
must0(nc.Publish(SubjectEventUserRegister, Encode[EventUserRegister](&EventUserRegister{
UserID: uuid.New().String(),
RegisterTime: time.Now().UnixMilli(),
})))
select {}
}
func must[T any](t T, err error) T {
if err != nil {
panic(err)
}
return t
}
func must0(err error) {
if err != nil {
panic(err)
}
}
func should(b bool) {
if !b {
panic(b)
}
}
I got this when I go run main.go
:
# github.com/cloudwego/frugal/internal/atm/abi
../../../../go/go1.20/pkg/mod/github.com/cloudwego/[email protected]/internal/atm/abi/abi.go:31:11: undefined: ArchCreateABI
Expected behavior
print the value of EventUserRegister
Kitex version:
no kitex
Environment:
GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/tzq0301/Library/Caches/go-build'
GOENV='/Users/tzq0301/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/tzq0301/go/go1.20/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/tzq0301/go/go1.20:/Users/tzq0301/go/go1.20/go'
GOPRIVATE=''
GOPROXY='https://goproxy.cn,direct'
GOROOT='/Users/tzq0301/go/go1.21.4'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/tzq0301/go/go1.21.4/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/tzq0301/Code/Go/nats-demo/frugal/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/sk/nrp4br617kscldkf4jsr0dsh0000gn/T/go-build3032409702=/tmp/go-build -gno-record-gcc-switches -fno-common'
Frugal is only available on AMD64 arch machines, while some user have both AMD64 & ARM64 machines, which makes it difficult to keep the same codebase.
Is your feature request related to a problem? Please describe.
Compiling using go1.19 will failed with the following messages:
../../../../go/pkg/mod/github.com/cloudwego/[email protected]/internal/loader/funcdata_invalid.go:27:9: panic("Unsupported Go version. Supported versions are: 1.15, 1.16, 1.17, 1.18") (no value) used as value
Describe the solution you'd like
support compile using go1.19
Describe alternatives you've considered
Additional context
If I simply run the examples in the Serialization and deserialization on a customized Go struct
section in README.md, it will get an error message:
panic: frugal: cannot measure encoded size: invalid tag for field main.MyStruct.Msg
goroutine 1 [running]:
github.com/cloudwego/frugal/internal/binary/encoder.EncodedSize({0xf714e0, 0xc00010f620})
/home/yunwei/coding/frugal/internal/binary/encoder/encoder.go:69 +0x8e
github.com/cloudwego/frugal.EncodedSize(...)
/home/yunwei/coding/frugal/frugal.go:27
main.main()
/home/yunwei/coding/frugal/cmd/main.go:20 +0xa9
exit status 2
the test code is like this, which is copied from README:
package main
import (
"github.com/cloudwego/frugal"
)
type MyStruct struct {
Msg string `frugal:"1,default"`
Code int64 `frugal:"2,default"`
Numbers []int64 `frugal:"3,default,list<i64>"`
}
func main() {
ms := &MyStruct{
Msg: "my message",
Code: 1024,
Numbers: []int64{0, 1, 2, 3, 4},
}
buf := make([]byte, frugal.EncodedSize(ms))
frugal.EncodeObject(buf, nil, ms)
got := &MyStruct{}
frugal.DecodeObject(buf, got)
}
However, if I change the struct to:
type MyStruct struct {
Msg string `frugal:"1,default,string"`
Code int64 `frugal:"2,default,i64"`
Numbers []int64 `frugal:"3,default,list<i64>"`
}
Then the example can be run successfully.
The README said:
Frugal tag is like
frugal:"1,default,string"
,1
is field ID,default
is field requiredness,string
is field type. Field ID and requiredness is always required, but field type is only required forlist
,set
andenum
.
while the code in the internal/binary/defs/resolver.go
gives that:
/* must have 3 fields: ID, Requiredness, Type */
if ft = strings.Split(tv, ","); len(ft) != 3 {
return nil, fmt.Errorf("invalid tag for field %s.%s", vt, sf.Name)
}
Is the field type
optional tag not support currently, or the tag should be provided each time in Frugal?
By the way, another minor fix for README example is in PR #7 .
When use the kitex with go1.20 I encountered the following problems , I hope there will be support for go 1.20 soon
# github.com/cloudwego/frugal/internal/loader
../../../../go/pkg/mod/github.com/cloudwego/[email protected]/internal/loader/funcdata.go:32:22: undefined: _ModuleData
../../../../go/pkg/mod/github.com/cloudwego/[email protected]/internal/loader/funcdata.go:35:27: undefined: _ModuleData
../../../../go/pkg/mod/github.com/cloudwego/[email protected]/internal/loader/funcdata.go:39:16: undefined: _ModuleData
../../../../go/pkg/mod/github.com/cloudwego/[email protected]/internal/loader/funcdata.go:75:26: undefined: _ModuleData
../../../../go/pkg/mod/github.com/cloudwego/[email protected]/internal/loader/funcdata_invalid.go:27:9: panic("Unsupported Go version. Supported versions are: 1.15, 1.16, 1.17, 1.18, 1.19") (no value) used as value
frugal/internal/atm/ssa/pass_fusion_amd64.go
Line 111 in 167cbd2
在现代处理器上三操作数LEA比两操作数LEA慢。
Intel® 64 and IA-32 Architectures Optimization Reference Manual
3.5.1.3 Using LEA
For LEA instructions with three source operands and some specific situations, instruction latency has increased to 3 cycles, and must dispatch via port 1:
— LEA that has all three source operands: base, index, and offset.
— LEA that uses base and index registers where the base is EBP, RBP, or R13.
...
see https://reviews.llvm.org/D32277
frugal/internal/atm/ssa/pass_fusion_amd64.go
Lines 160 to 161 in 167cbd2
bts指令对于内存目标很慢.
see https://go-review.googlesource.com/c/go/+/318149
frugal/internal/atm/ssa/pass_fusion_amd64.go
Lines 138 to 144 in 167cbd2
why not support it in rc cycle? or add a default slow support for outter version
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.