seefan / gossdb Goto Github PK
View Code? Open in Web Editor NEWssdb client for go with pool surport
License: MIT License
ssdb client for go with pool surport
License: MIT License
更新到你的最新版本之后直接崩溃了, 请问当前可以直接使用的是什么版本呢?TagV1.0吗?
| // ReadSlice reads until the first occurrence of delim in the input,
| // returning a slice pointing at the bytes in the buffer.
| // The bytes stop being valid at the next read.
| // If ReadSlice encounters an error before finding a delimiter,
| // it returns all the data in the buffer and the error itself (often io.EOF).
| // ReadSlice fails with error ErrBufferFull if the buffer fills without a delim.
| // Because the data returned from ReadSlice will be overwritten
| // by the next I/O operation, most clients should use
| // ReadBytes or ReadString instead.
| // ReadSlice returns err != nil if and only if line does not end in delim.
goroutine 996 [running]:
panic: runtime error: slice bounds out of range
bufio.(*Reader).ReadSlice(0xc42006ed20, 0xc42004550a, 0x453567, 0xc4200458f0, 0x42ca84, 0xc420045588, 0x452bb0)
/usr/local/go/src/bufio/bufio.go:316 +0x258
bufio.(*Reader).ReadBytes(0xc42006ed20, 0xc420468b0a, 0xc420468b70, 0x0, 0x0, 0xc4200459c8, 0x55f8af)
/usr/local/go/src/bufio/bufio.go:416 +0x66
github.com/seefan/gossdb.(*SSDBClient).Recv(0xc4200f16b0, 0xc420045950, 0x3, 0x3, 0x0, 0x0)
/root/go/src/github.com/seefan/gossdb/ssdbclient.go:196 +0x8a
github.com/seefan/gossdb.(*SSDBClient).do(0xc4200f16b0, 0xc420045950, 0x3, 0x3, 0x0, 0xc4200457d8, 0x410c32, 0xc42021e1b0, 0x10)
/root/go/src/github.com/seefan/gossdb/ssdbclient.go:60 +0x9c
github.com/seefan/gossdb.(*SSDBClient).Do(0xc4200f16b0, 0xc420045950, 0x3, 0x3, 0x40eb9f, 0x683e60, 0xc42021e1b0, 0xc420045900, 0xc42021e1b0)
/root/go/src/github.com/seefan/gossdb/ssdbclient.go:80 +0x74
github.com/seefan/gossdb.(*Client).Do(0xc42000d580, 0xc420045950, 0x3, 0x3, 0x0, 0xc4200458b0, 0x4b0088, 0xc4200b0040, 0x54003d)
/root/go/src/github.com/seefan/gossdb/client.go:46 +0x50
github.com/seefan/gossdb.(*Client).Hget(0xc42000d580, 0xc42021e184, 0xc, 0x70c9e5, 0x1, 0xc, 0xc420045a18, 0xc4200f3a20, 0xc42021e184)
/root/go/src/github.com/seefan/gossdb/hset.go:30 +0x184
sock.SetReadBuffer(8192),8192还是太小,测试大数据读的时候性能太差
建议改成配置文件,默认1024*1024
Hope you can add validation method.
例如
cli.MultiGetSlice("a", "b")
会报“the trace error is bad arguments type”。
经查看实现,内部没有对args进行解构,导致do的时候传入的是
[multi_get [ ]]
导致类型判断出错。
望修复。
现在tag 是v2.0,go mod不会认。。请使用v2.0.0格式
使用Qpush_back等操作的时候,会出现
the trace error is
client send error
the trace error is
bad arguments type
设置了 gossdb.Encoding = true后, 出现
access ssdb error, code is [client_error wrong number of arguments], parameter is [list1]
使用之前的版本代码库, 就没有问题
能否告知是 测试中发现什么问题吗?
不好意思。自己配置的问题。没有错误。
使用连接池
上层方法手误连续写了两个defer c.Close()
第一次请求正常,之后的请求报错:
2019-03-06 16:59:52.367 error runtime error: index out of range
2019-03-06 16:59:52.367 error goroutine 49 [running]:
runtime/debug.Stack(0xc000545168, 0x1, 0x1)
/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
jqdataserver2/routers.post.func1.1(0xc00026e000)
/home/server/go/src/jqdataserver2/routers/router.go:84 +0x7f
panic(0xb07fc0, 0x1315900)
/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Slice).Get(0xc0000a4c30, 0x0, 0x0, 0x0)
/home/server/go/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/slice.go:101 +0x133
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Pool).Get(0xc000115080, 0xc0005453e0, 0x0, 0xc0001c4180)
/home/server/go/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/pool.go:150 +0x5b
github.com/seefan/gossdb.(*Connectors).NewClient(0xc00017afa0, 0xc0001bf080, 0xbe1f23, 0xc)
/home/server/go/src/github.com/seefan/gossdb/connectors.go:86 +0x3b
打印信息发现:
pool size:50 actived client:-1 wait create:0 config max pool size:5000
actived client是-1
感觉在Close()方法可以优化一下
明明取用户自己key的数据,偶尔会取到其他用户key的数据
gossdb.Client().HRScanArray(key, "", "", int64(limit))
我们正式环境的服务器,之前存储的消息量小余13GByte 时,没有任何问题,但是最近两天频繁出现Hinc失败,没有返回错误,因为有响应,然后响应的长度大于2,所以Hinc之后的返回值为-1, 但是没有任何错误。以下为两次的结果,好像串包了。貌似info命令和get命令了。请问您之前是否有碰到呢?
附加信息:我用的是你master 分支的0394129 这个版本的driver,这个版本使用的仍然是ideawu官方的go语言driver。
Hinc hashName=8019915#gmsg key=0 num=1 err=access ssdb error, code is [ok ssdb-server version 1.9.4 links 120 total_calls 1223109 dbsize 13026765809 binlogs capacity : 100000000
ssdbOperator.Hinc failed err=access ssdb error, code is [ok 68856_1507898802415_7624782ΰ���� *�x�lP͊�0~�a.{�l˖�[)K��-�՚$"��Jʺ!���������tO��|3����YY�
Zrange Zrrange 接口有bug:
val 没有分配内存,val=make(map[string]int64)
麻烦修复下
现有的连接池要服务自己去获取连接和归还连接,这一步对服务而言有点多余,能不能将获取与归还的操作转移到具体命令函数中,让连接的使用对用户无感知?
➜ ~ go get github.com/seefan/gossdb
gopath/src/github.com/seefan/gossdb/connectors.go:68: syntax error: unexpected range, expecting {
gopath/src/github.com/seefan/gossdb/connectors.go:72: non-declaration statement outside function body
gopath/src/github.com/seefan/gossdb/connectors.go:73: syntax error: unexpected }
pool, err := gossdb.NewPool(&gossdb.Config{
Host: "127.0.0.1",
Port: 6380,
MinPoolSize: 5,
MaxPoolSize: 50,
AcquireIncrement: 5,
})
使用上面代码时,出现了undefined: gossdb.Config的问题
根据 文档 bitcount 的范围值是 [0, 1073741824]
应该是需要 uint32/64类型
setbit 方法的 bit 需要时int类型,否者出错
func (s *Client) Setbit(key string, offset int, bit int)
该语句在connectors.go 中的healthWorker方法.
觉得打印信息的功能由使用者调用比较好吧?
在从连接池中取client时候是不是没有判断该连接的可用状态?比如高并发情况下,重启ssdb服务器,会出现大量pool is busy,can not get new client in x seconds的错误,这是什么情况呢?
作者,您好,我存了一个 8M 左右的 value, 但是我在程序客户端 Get 不到这个 value,每次取出来的都是空字符串,请指教
我安装了您的gossdb :然后,队尾插入数据,队首取数据并删除。但是不知道什么原因,导致我队列数据堆积过多,然后发现只能写数据,不能读数据了。而且,qclear 执行之后,也是只能写入,不能读取。不知道您遇到过这种情况吗?
client.SSDBClient需要添加Ping函数,现在有两个IClient (client 和 gopoil目录中)
balance/balance.go:109: cannot use client.SSDBClient literal (type *client.SSDBClient) as type gopool.IClient in return argument:
*client.SSDBClient does not implement gopool.IClient (missing Ping method)
connectors line 68
github.com/seefan/gossdb/connectors.go:68: syntax error: unexpected range, expecting {
gossdb/ssdbclient/ssdb_client.go
Line 445 in 9789440
delim的值代表啥意思,感觉在更新完offset的值以后这个值应该也要重置吧,总体感觉这个parseBuffer函数有点问题,我这边遇到读取一个超大value后后续的client的值读取都不正确了.
如题,见下面, IO wait, 83 minutes, 有很多这样的,重启下 ssdb server 就好了
goroutine 27972 [IO wait, 83 minutes]: internal/poll.runtime_pollWait(0x7fb6a7865eb0, 0x72, 0x0) /home/soft/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420012598, 0x72, 0xffffffffffffff00, 0xda2360, 0xd9d708) /home/soft/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc420012598, 0xc42017f000, 0x1000, 0x1000) /home/soft/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc420012580, 0xc42017f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/soft/go/src/internal/poll/fd_unix.go:125 +0x18a net.(*netFD).Read(0xc420012580, 0xc42017f000, 0x1000, 0x1000, 0x7, 0x0, 0x0) /home/soft/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc42000e6a8, 0xc42017f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/soft/go/src/net/net.go:176 +0x6d bufio.(*Reader).fill(0xc420052fc0) /home/soft/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc420052fc0, 0xa002017f00a, 0x38, 0x40, 0xc420ffca80, 0x7fb6a7897e18, 0x0) /home/soft/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadBytes(0xc420052fc0, 0xa, 0x1, 0x0, 0x0, 0xc420ffca80, 0x2) /home/soft/go/src/bufio/bufio.go:416 +0x6b github.com/seefan/gossdb.(*SSDBClient).Recv(0xc42008c840, 0xc420597458, 0x2, 0x2, 0x0, 0x0) /home/soft/gopath/src/github.com/seefan/gossdb/ssdbclient.go:197 +0x72 github.com/seefan/gossdb.(*SSDBClient).do(0xc42008c840, 0xc420597458, 0x2, 0x2, 0xc4205972a8, 0xc4205972e8, 0x877011, 0xc420140158, 0xc420597358) /home/soft/gopath/src/github.com/seefan/gossdb/ssdbclient.go:59 +0x9c github.com/seefan/gossdb.(*SSDBClient).Do(0xc42008c840, 0xc420597458, 0x2, 0x2, 0x10, 0x9b3d00, 0xc420df2001, 0xc4216679b0, 0xc420272478) /home/soft/gopath/src/github.com/seefan/gossdb/ssdbclient.go:79 +0x74 github.com/seefan/gossdb.(*Client).Do(0xc4200c77c0, 0xc420597458, 0x2, 0x2, 0x5c, 0x412397, 0x7fb6a78692a8, 0x465992, 0x0) /home/soft/gopath/src/github.com/seefan/gossdb/client.go:46 +0x50 github.com/seefan/gossdb.(*Client).Get(0xc4200c77c0, 0xc4202724b0, 0xd, 0x4d2934, 0xc4205974f0, 0x42be79, 0xc400000008) /home/soft/gopath/src/github.com/seefan/gossdb/set.go:51 +0xf7 main/app/model.(*ssdbCache).Get(0xc4205975d0, 0x9982a0, 0xc420a85ce0, 0x1, 0x1)
初始化链接池失败时会一直开协程导致服务的内存和cpu暴涨
NewPool() -> Start() -> go watch()-> Start() ->go watch() 。。。。。。
会一直重复 start()函数。
以下我个人修改方法,
func (p *Pool) Start() error {
p.defaultConfig()
p.poolWait = make(chan *PooledClient, p.MaxWaitSize)
p.waitCount = 0
p.pooled.Init(p.AcquireIncrement, p.MinPoolSize, p.MaxPoolSize, p)
p.pooled.Append(p.MinPoolSize)
if p.Status == PoolInit {//只是首次执行 否则会内存泄露
p.Status = PoolStart
p.watcher = time.NewTicker(time.Second * time.Duration(p.HealthSecond))
go p.watch()
}
return nil
}
错误信息 : Get 1:123 error
配置中只加了这两条
maxPoolSize = 500
GetClientTimeout = 200
代码如下
err := gossdb.Start(&conf.Config{
Host: 127.0.01,
Port: 8888,
MaxPoolSize: 500,
GetClientTimeout: 200,
})
if err != nil {
fmt.Println("gossdb start err:", err)
r.Response.WriteJsonExit(g.Map{
"ERROR": "SSDB_CONNECT_ERROR_1",
})
}
defer gossdb.Shutdown()
c, err := gossdb.NewClient()
if err != nil {
fmt.Println("gossdb new client err:", err)
r.Response.WriteJsonExit(g.Map{
"ERROR": "SSDB_CONNECT_ERROR_2",
})
}
defer c.Close()
val, err := c.Get(key)
if err != nil {
fmt.Println("get config", err)
r.Response.WriteJsonExit(g.Map{
"ERROR": "SSDB_CONNECT_ERROR_3",
})
}
10 : 863199410872
9 : 433702681372
11 : 433702628072
3 result(s) (0.001 sec)
(0.001 sec)
panic: runtime error: slice bounds out of range
goroutine 1819 [running]:
panic(0x4b46c0, 0xc820010080)
/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:464 +0x3e6
github.com/ssdb/gossdb/ssdb.(_Client).parse(0xc82009c4d0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/ssdb/gossdb/ssdb/ssdb.go:146 +0x52f
github.com/ssdb/gossdb/ssdb.(_Client).recv(0xc82009c4d0, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/ssdb/gossdb/ssdb/ssdb.go:132 +0x86
github.com/ssdb/gossdb/ssdb.(_Client).Do(0xc82009c4d0, 0xc8209b9d30, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/ssdb/gossdb/ssdb/ssdb.go:34 +0xb4
github.com/seefan/gossdb.(_Client).Do(0xc82009c4d0, 0xc8209b9d30, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/seefan/gossdb/client.go:160 +0x407
github.com/seefan/gossdb.(_Client).qpush(0xc82009c4d0, 0xc820edf2f0, 0x2f, 0x1, 0xc8209b9e60, 0x1, 0x1, 0xc820d10af0, 0x0, 0x0)
/Users/left2right/go/src/github.com/seefan/gossdb/queue.go:76 +0x3b1
github.com/seefan/gossdb.(_Client).Qpush(0xc82009c4d0, 0xc820edf2f0, 0x2f, 0xc8209b9e60, 0x1, 0x1, 0x1, 0x0, 0x0)
/Users/left2right/go/src/github.com/seefan/gossdb/queue.go:93 +0x73
main.SsdbStoreMessageLog(0xc820616ff0)
2个独立的程序, 调用同一个库的话
如何操作连接池?
我目前使用是 2个 gossdb.Start(), 但是第二个启动的程序会卡住, 不能做任何数据库操作
数据库是同一个
在macOS平台, 执行 HScan(大约1000多个数据,)只需要100ms
但是在Linux(Centos7/8, Debian 8/10, ubuntu 20.04/18.04/16.04)都需要 15s左右
ssdbPool, err = gossdb.NewPool(&gossdb.Config{
Host: ssdb_addr[0],
Port: port,
MinPoolSize: 5,
MaxPoolSize: 200,
AcquireIncrement: 5,
})
//导入代码部份
conn, err := ssdbPool.NewClient()
if err != nil {
log.Println("myg ssdb:", err.Error())
return
}
defer conn.Close()
_, err = conn.Qpush_front("visit", testvisit)
是 不是写的有问题,不是的话,就有问题了,导致最后程序无法链接。
问题描述,
初始化连接池,暂时不从连接池里面获取链接,指定错误的ip或者端口,程序崩溃
panic: runtime error: index out of range
goroutine 3 [running]:
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Slice).checkMinPoolClient(0xc420098550, 0x3c)
/home/wangchangqing/work/tuser/tuser-center/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/watch.go:56 +0x2d5
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Pool).watch(0xc420126380)
/home/wangchangqing/work/tuser/tuser-center/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/watch.go:16 +0x149
created by github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Pool).Start
/home/wangchangqing/work/tuser/tuser-center/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/pool.go:95 +0x17b
exit status 2
查看代码
if checkIndex < 0 || checkIndex < s.current {
checkIndex = s.length - 1
}
//同一个连接检查要间隔HealthSecond秒
if s.pooled[checkIndex] != nil && !s.pooled[checkIndex].isUsed && s.pooled[checkIndex].lastTime+hs < now {
s.pooled[checkIndex].lastTime = now
if !s.pooled[checkIndex].Client.Ping() {
s.pooled[checkIndex].Client.Close()
if err := s.pooled[checkIndex].Client.Start(); err != nil {
s.length -= 1
}
}
}
checkIndex=-1,导致的崩溃,
这块是不是对checkIndex在加一层判断
[root@localhost gossdb]# grep -r 'goerr.New(' ./*
./client.go: return goerr.New("ssdb respone error")
./client.go: return goerr.New("access ssdb error, code is %v, parameter is %v", resp, errKey)
./client.go: return goerr.New("access ssdb error, code is %v", resp)
./connectors.go: return goerr.New("Please call the init function first")
./connectors.go: return nil, goerr.New("Please call the init function first")
./connectors.go: return nil, goerr.New("get client error")
./ssdbclient.go: return nil, goerr.New("gossdb client is closed.")
./ssdbclient.go: return nil, goerr.New("Authentication failed,password is wrong")
// gossdb 1.1.3 + ssdb-server-1.9.0.exe
for i := 0; i < 10; i++ {
p, err := gossdb.NewPool(&conf.Config{
Password: cfg.SSDB.Password,
Host: cfg.SSDB.Ip,
Port: cfg.SSDB.Port,
//RetryEnabled: true,
})
c, err := p.NewClient()
checkErr(err)
if !c.Ping() { // 第三次时,就死在这里
checkErr(errors.New("aaa"))
}
}
Sorry I cant seem to find how to run hset / multi / batch / exec / qclear etc
错误: Trace:pool is busy,can not get new client in 5 seconds,wait count is 1
我应该如何来解决这个问题?
谢谢
超出MaxPoolSize的连接会进入等待连接状态, 由于占用着锁资源, 导致了其他client无法正常关闭连接,
它就无法获得Client, 意味着只有超出的连接请求全部超时时, 程序才会恢复.
NewPool()这个函数的超时如何设置? 我尝试设置GetClientTimeout 这个变量,好像丝毫没有效果
github.com/seefan/gossdb/client.go:4:2: cannot find package "github.com/seefan/goerr" in any of:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.