Coder Social home page Coder Social logo

gossdb's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gossdb's Issues

更新到最新版本 执行crash

更新到你的最新版本之后直接崩溃了, 请问当前可以直接使用的是什么版本呢?TagV1.0吗?
| // ReadSlice reads until the first occurrence of delim in the input,
  | // returning a slice pointing at the bytes in the buffer.
  | // The bytes stop being valid at the next read.
  | // If ReadSlice encounters an error before finding a delimiter,
  | // it returns all the data in the buffer and the error itself (often io.EOF).
  | // ReadSlice fails with error ErrBufferFull if the buffer fills without a delim.
  | // Because the data returned from ReadSlice will be overwritten
  | // by the next I/O operation, most clients should use
  | // ReadBytes or ReadString instead.
  | // ReadSlice returns err != nil if and only if line does not end in delim.

image

goroutine 996 [running]:
panic: runtime error: slice bounds out of range

bufio.(*Reader).ReadSlice(0xc42006ed20, 0xc42004550a, 0x453567, 0xc4200458f0, 0x42ca84, 0xc420045588, 0x452bb0)
/usr/local/go/src/bufio/bufio.go:316 +0x258
bufio.(*Reader).ReadBytes(0xc42006ed20, 0xc420468b0a, 0xc420468b70, 0x0, 0x0, 0xc4200459c8, 0x55f8af)
/usr/local/go/src/bufio/bufio.go:416 +0x66
github.com/seefan/gossdb.(*SSDBClient).Recv(0xc4200f16b0, 0xc420045950, 0x3, 0x3, 0x0, 0x0)
/root/go/src/github.com/seefan/gossdb/ssdbclient.go:196 +0x8a
github.com/seefan/gossdb.(*SSDBClient).do(0xc4200f16b0, 0xc420045950, 0x3, 0x3, 0x0, 0xc4200457d8, 0x410c32, 0xc42021e1b0, 0x10)
/root/go/src/github.com/seefan/gossdb/ssdbclient.go:60 +0x9c
github.com/seefan/gossdb.(*SSDBClient).Do(0xc4200f16b0, 0xc420045950, 0x3, 0x3, 0x40eb9f, 0x683e60, 0xc42021e1b0, 0xc420045900, 0xc42021e1b0)
/root/go/src/github.com/seefan/gossdb/ssdbclient.go:80 +0x74
github.com/seefan/gossdb.(*Client).Do(0xc42000d580, 0xc420045950, 0x3, 0x3, 0x0, 0xc4200458b0, 0x4b0088, 0xc4200b0040, 0x54003d)
/root/go/src/github.com/seefan/gossdb/client.go:46 +0x50
github.com/seefan/gossdb.(*Client).Hget(0xc42000d580, 0xc42021e184, 0xc, 0x70c9e5, 0x1, 0xc, 0xc420045a18, 0xc4200f3a20, 0xc42021e184)
/root/go/src/github.com/seefan/gossdb/hset.go:30 +0x184

ssdbclient.go读性能问题

sock.SetReadBuffer(8192),8192还是太小,测试大数据读的时候性能太差
建议改成配置文件,默认1024*1024

no auth

Hope you can add validation method.

ARMv7环境下,会发生连接池泄露

netstat -anp | grep 8888 | wc -l有3166个连接,最多时有2w多个。
使用方式如下

ssdbConn, err := store.ssdbPool.NewClient()
if err != nil {
    //log
    return err
}
defer ssdbConn.Close()

x86环境下未出现该问题,初步怀疑ARMv7下defer的实现有bug,正在验证。

a7c20f26-ab28-4097-ba98-92ef3388d740

最新的代码 , QPUSH相关操作会有问题

使用Qpush_back等操作的时候,会出现
the trace error is
client send error
the trace error is
bad arguments type

设置了 gossdb.Encoding = true后, 出现
access ssdb error, code is [client_error wrong number of arguments], parameter is [list1]

使用之前的版本代码库, 就没有问题

连续c.Close()导致报错

使用连接池
上层方法手误连续写了两个defer c.Close()
第一次请求正常,之后的请求报错:
2019-03-06 16:59:52.367 error runtime error: index out of range
2019-03-06 16:59:52.367 error goroutine 49 [running]:
runtime/debug.Stack(0xc000545168, 0x1, 0x1)
/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
jqdataserver2/routers.post.func1.1(0xc00026e000)
/home/server/go/src/jqdataserver2/routers/router.go:84 +0x7f
panic(0xb07fc0, 0x1315900)
/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Slice).Get(0xc0000a4c30, 0x0, 0x0, 0x0)
/home/server/go/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/slice.go:101 +0x133
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Pool).Get(0xc000115080, 0xc0005453e0, 0x0, 0xc0001c4180)
/home/server/go/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/pool.go:150 +0x5b
github.com/seefan/gossdb.(*Connectors).NewClient(0xc00017afa0, 0xc0001bf080, 0xbe1f23, 0xc)
/home/server/go/src/github.com/seefan/gossdb/connectors.go:86 +0x3b

打印信息发现:
pool size:50 actived client:-1 wait create:0 config max pool size:5000
actived client是-1

感觉在Close()方法可以优化一下

Hincr 连接好像串包了

我们正式环境的服务器,之前存储的消息量小余13GByte 时,没有任何问题,但是最近两天频繁出现Hinc失败,没有返回错误,因为有响应,然后响应的长度大于2,所以Hinc之后的返回值为-1, 但是没有任何错误。以下为两次的结果,好像串包了。貌似info命令和get命令了。请问您之前是否有碰到呢?
附加信息:我用的是你master 分支的0394129 这个版本的driver,这个版本使用的仍然是ideawu官方的go语言driver。

Hinc hashName=8019915#gmsg key=0 num=1 err=access ssdb error, code is [ok ssdb-server version 1.9.4 links 120 total_calls 1223109 dbsize 13026765809 binlogs capacity : 100000000

ssdbOperator.Hinc failed err=access ssdb error, code is [ok 68856_1507898802415_7624782ΰ���� *�x�lP͊�0~�a.{�l˖�[)K��-�՚$"��Jʺ!���������tO��|3����YY�

连接数不够用

并发量大的情况下, 连接数会不够用
image

我已经设置了MaxPoolSize=500
感觉像是连接没有释放, 但是 我有在每个操作之后, 都去调用Close方法
image

隐藏连接池相关操作

现有的连接池要服务自己去获取连接和归还连接,这一步对服务而言有点多余,能不能将获取与归还的操作转移到具体命令函数中,让连接的使用对用户无感知?

~ go get github.com/seefan/gossdb

➜ ~ go get github.com/seefan/gossdb

github.com/seefan/gossdb

gopath/src/github.com/seefan/gossdb/connectors.go:68: syntax error: unexpected range, expecting {
gopath/src/github.com/seefan/gossdb/connectors.go:72: non-declaration statement outside function body
gopath/src/github.com/seefan/gossdb/connectors.go:73: syntax error: unexpected }

undefined: gossdb.Config

pool, err := gossdb.NewPool(&gossdb.Config{
	Host:             "127.0.0.1",
	Port:             6380,
	MinPoolSize:      5,
	MaxPoolSize:      50,
	AcquireIncrement: 5,
})

使用上面代码时,出现了undefined: gossdb.Config的问题

Bitcount 返回值 byte 位数不够

Q1

根据 文档 bitcount 的范围值是 [0, 1073741824]

应该是需要 uint32/64类型

Q2

setbit 方法的 bit 需要时int类型,否者出错

func (s *Client) Setbit(key string, offset int, bit int)

获取大的 value,获取不到

作者,您好,我存了一个 8M 左右的 value, 但是我在程序客户端 Get 不到这个 value,每次取出来的都是空字符串,请指教

关于ssdb 队列缓存数据上限的问题

我安装了您的gossdb :然后,队尾插入数据,队首取数据并删除。但是不知道什么原因,导致我队列数据堆积过多,然后发现只能写数据,不能读数据了。而且,qclear 执行之后,也是只能写入,不能读取。不知道您遇到过这种情况吗?

最新版问题

client.SSDBClient需要添加Ping函数,现在有两个IClient (client 和 gopoil目录中)

balance/balance.go:109: cannot use client.SSDBClient literal (type *client.SSDBClient) as type gopool.IClient in return argument:
*client.SSDBClient does not implement gopool.IClient (missing Ping method)

connectors.go line 68

connectors line 68

github.com/seefan/gossdb/connectors.go:68: syntax error: unexpected range, expecting {

ssdb set 永远不超时

如题,见下面, IO wait, 83 minutes, 有很多这样的,重启下 ssdb server 就好了
goroutine 27972 [IO wait, 83 minutes]: internal/poll.runtime_pollWait(0x7fb6a7865eb0, 0x72, 0x0) /home/soft/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420012598, 0x72, 0xffffffffffffff00, 0xda2360, 0xd9d708) /home/soft/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc420012598, 0xc42017f000, 0x1000, 0x1000) /home/soft/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc420012580, 0xc42017f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/soft/go/src/internal/poll/fd_unix.go:125 +0x18a net.(*netFD).Read(0xc420012580, 0xc42017f000, 0x1000, 0x1000, 0x7, 0x0, 0x0) /home/soft/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc42000e6a8, 0xc42017f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/soft/go/src/net/net.go:176 +0x6d bufio.(*Reader).fill(0xc420052fc0) /home/soft/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc420052fc0, 0xa002017f00a, 0x38, 0x40, 0xc420ffca80, 0x7fb6a7897e18, 0x0) /home/soft/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadBytes(0xc420052fc0, 0xa, 0x1, 0x0, 0x0, 0xc420ffca80, 0x2) /home/soft/go/src/bufio/bufio.go:416 +0x6b github.com/seefan/gossdb.(*SSDBClient).Recv(0xc42008c840, 0xc420597458, 0x2, 0x2, 0x0, 0x0) /home/soft/gopath/src/github.com/seefan/gossdb/ssdbclient.go:197 +0x72 github.com/seefan/gossdb.(*SSDBClient).do(0xc42008c840, 0xc420597458, 0x2, 0x2, 0xc4205972a8, 0xc4205972e8, 0x877011, 0xc420140158, 0xc420597358) /home/soft/gopath/src/github.com/seefan/gossdb/ssdbclient.go:59 +0x9c github.com/seefan/gossdb.(*SSDBClient).Do(0xc42008c840, 0xc420597458, 0x2, 0x2, 0x10, 0x9b3d00, 0xc420df2001, 0xc4216679b0, 0xc420272478) /home/soft/gopath/src/github.com/seefan/gossdb/ssdbclient.go:79 +0x74 github.com/seefan/gossdb.(*Client).Do(0xc4200c77c0, 0xc420597458, 0x2, 0x2, 0x5c, 0x412397, 0x7fb6a78692a8, 0x465992, 0x0) /home/soft/gopath/src/github.com/seefan/gossdb/client.go:46 +0x50 github.com/seefan/gossdb.(*Client).Get(0xc4200c77c0, 0xc4202724b0, 0xd, 0x4d2934, 0xc4205974f0, 0x42be79, 0xc400000008) /home/soft/gopath/src/github.com/seefan/gossdb/set.go:51 +0xf7 main/app/model.(*ssdbCache).Get(0xc4205975d0, 0x9982a0, 0xc420a85ce0, 0x1, 0x1)

协程过多导致内存cpu过高

初始化链接池失败时会一直开协程导致服务的内存和cpu暴涨
NewPool() -> Start() -> go watch()-> Start() ->go watch() 。。。。。。
会一直重复 start()函数。
以下我个人修改方法,
func (p *Pool) Start() error {
p.defaultConfig()
p.poolWait = make(chan *PooledClient, p.MaxWaitSize)
p.waitCount = 0
p.pooled.Init(p.AcquireIncrement, p.MinPoolSize, p.MaxPoolSize, p)
p.pooled.Append(p.MinPoolSize)
if p.Status == PoolInit {//只是首次执行 否则会内存泄露
p.Status = PoolStart
p.watcher = time.NewTicker(time.Second * time.Duration(p.HealthSecond))
go p.watch()
}
return nil
}

开启300个协程去读取操作, 会有一半的读取操作报错

错误信息 : Get 1:123 error
配置中只加了这两条
maxPoolSize = 500
GetClientTimeout = 200
代码如下
err := gossdb.Start(&conf.Config{
Host: 127.0.01,
Port: 8888,
MaxPoolSize: 500,
GetClientTimeout: 200,
})
if err != nil {
fmt.Println("gossdb start err:", err)
r.Response.WriteJsonExit(g.Map{
"ERROR": "SSDB_CONNECT_ERROR_1",
})
}
defer gossdb.Shutdown()
c, err := gossdb.NewClient()
if err != nil {
fmt.Println("gossdb new client err:", err)
r.Response.WriteJsonExit(g.Map{
"ERROR": "SSDB_CONNECT_ERROR_2",
})
}
defer c.Close()
val, err := c.Get(key)
if err != nil {
fmt.Println("get config", err)
r.Response.WriteJsonExit(g.Map{
"ERROR": "SSDB_CONNECT_ERROR_3",
})
}

用zrrange取出的map有时不按顺序排列

map[10:863199410872 9:433702681372 11:433702628072]
map[10:863199410872 9:433702681372 11:433702628072]
map[9:433702681372 11:433702628072 10:863199410872]
map[10:863199410872 9:433702681372 11:433702628072]
map[9:433702681372 11:433702628072 10:863199410872]
map[10:863199410872 9:433702681372 11:433702628072]
用zrrange取出的map。上面的数据取自同一个zset, zset没有数据插入和更新。
下面是直接用ssdb命令取的结果是保持每次都是有序的。
ssdb 127.0.0.1:18888> zrrange gamescore:15 0 3
key value

10 : 863199410872
9 : 433702681372
11 : 433702628072
3 result(s) (0.001 sec)
(0.001 sec)

runtime error: slice bounds out of rang

panic: runtime error: slice bounds out of range

goroutine 1819 [running]:
panic(0x4b46c0, 0xc820010080)
/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:464 +0x3e6
github.com/ssdb/gossdb/ssdb.(_Client).parse(0xc82009c4d0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/ssdb/gossdb/ssdb/ssdb.go:146 +0x52f
github.com/ssdb/gossdb/ssdb.(_Client).recv(0xc82009c4d0, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/ssdb/gossdb/ssdb/ssdb.go:132 +0x86
github.com/ssdb/gossdb/ssdb.(_Client).Do(0xc82009c4d0, 0xc8209b9d30, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/ssdb/gossdb/ssdb/ssdb.go:34 +0xb4
github.com/seefan/gossdb.(_Client).Do(0xc82009c4d0, 0xc8209b9d30, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/left2right/go/src/github.com/seefan/gossdb/client.go:160 +0x407
github.com/seefan/gossdb.(_Client).qpush(0xc82009c4d0, 0xc820edf2f0, 0x2f, 0x1, 0xc8209b9e60, 0x1, 0x1, 0xc820d10af0, 0x0, 0x0)
/Users/left2right/go/src/github.com/seefan/gossdb/queue.go:76 +0x3b1
github.com/seefan/gossdb.(_Client).Qpush(0xc82009c4d0, 0xc820edf2f0, 0x2f, 0xc8209b9e60, 0x1, 0x1, 0x1, 0x0, 0x0)
/Users/left2right/go/src/github.com/seefan/gossdb/queue.go:93 +0x73
main.SsdbStoreMessageLog(0xc820616ff0)

HScan 执行时间超长 [Linux平台]

数据库是同一个

在macOS平台, 执行 HScan(大约1000多个数据,)只需要100ms
但是在Linux(Centos7/8, Debian 8/10, ubuntu 20.04/18.04/16.04)都需要 15s左右

问题,最后导致链接占满,超时,必须重启程序

ssdbPool, err = gossdb.NewPool(&gossdb.Config{
Host: ssdb_addr[0],
Port: port,
MinPoolSize: 5,
MaxPoolSize: 200,
AcquireIncrement: 5,

})

//导入代码部份
conn, err := ssdbPool.NewClient()
if err != nil {
log.Println("myg ssdb:", err.Error())
return
}
defer conn.Close()
_, err = conn.Qpush_front("visit", testvisit)

是 不是写的有问题,不是的话,就有问题了,导致最后程序无法链接。

连接池问题

问题描述,
初始化连接池,暂时不从连接池里面获取链接,指定错误的ip或者端口,程序崩溃

panic: runtime error: index out of range

goroutine 3 [running]:
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Slice).checkMinPoolClient(0xc420098550, 0x3c)
/home/wangchangqing/work/tuser/tuser-center/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/watch.go:56 +0x2d5
github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Pool).watch(0xc420126380)
/home/wangchangqing/work/tuser/tuser-center/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/watch.go:16 +0x149
created by github.com/seefan/gossdb/vendor/github.com/seefan/gopool.(*Pool).Start
/home/wangchangqing/work/tuser/tuser-center/src/github.com/seefan/gossdb/vendor/github.com/seefan/gopool/pool.go:95 +0x17b
exit status 2

查看代码

if checkIndex < 0 || checkIndex < s.current {
	checkIndex = s.length - 1
}
//同一个连接检查要间隔HealthSecond秒
if s.pooled[checkIndex] != nil && !s.pooled[checkIndex].isUsed && s.pooled[checkIndex].lastTime+hs < now {
	s.pooled[checkIndex].lastTime = now
	if !s.pooled[checkIndex].Client.Ping() {
		s.pooled[checkIndex].Client.Close()
		if err := s.pooled[checkIndex].Client.Start(); err != nil {
			s.length -= 1
		}
	}
}

checkIndex=-1,导致的崩溃,
这块是不是对checkIndex在加一层判断

引用包goerr没有New方法,go get 出错

[root@localhost gossdb]# grep -r 'goerr.New(' ./*
./client.go: return goerr.New("ssdb respone error")
./client.go: return goerr.New("access ssdb error, code is %v, parameter is %v", resp, errKey)
./client.go: return goerr.New("access ssdb error, code is %v", resp)
./connectors.go: return goerr.New("Please call the init function first")
./connectors.go: return nil, goerr.New("Please call the init function first")
./connectors.go: return nil, goerr.New("get client error")
./ssdbclient.go: return nil, goerr.New("gossdb client is closed.")
./ssdbclient.go: return nil, goerr.New("Authentication failed,password is wrong")

最多创建3个pool就无法使用了

// gossdb 1.1.3  + ssdb-server-1.9.0.exe

for i := 0; i < 10; i++ {
		p, err := gossdb.NewPool(&conf.Config{
			Password: cfg.SSDB.Password,
			Host:     cfg.SSDB.Ip,
			Port:     cfg.SSDB.Port,
			//RetryEnabled: true,
		})
		c, err := p.NewClient()   
		checkErr(err)
		if !c.Ping() {   // 第三次时,就死在这里
			checkErr(errors.New("aaa"))
		}
	}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.