yongman / tidis Goto Github PK

View Code? Open in Web Editor NEW

1.4K 55.0 139.0 6.83 MB

Distributed transactional NoSQL database, Redis protocol compatible using tikv as backend

Home Page: https://github.com/tidb-incubator/tidis

License: MIT License

Makefile 0.10% Go 83.05% Shell 0.02% Python 16.68% Dockerfile 0.15%

redis nosql database tikv 2pc raft distributed-nosql-database transaction-storage transaction rocksdb

tidis's People

Contributors

Stargazers

Watchers

Forkers

zhanglei shizy818 hadoop835 sensecollective younisshah keevol mythz yut148 wuxiaobo hoverbear koalacxr crazyguitar storv kooksee zac-garby rcholic db4u dikang123 priestd09 xiaoxiugege carlvine500 lephuocmy668 strogo xushiwei b-xiang alimy rohitjoshi liukun4515 yanghongkjxy wowuwowuwo nxsre fsycg jacklli kulasama threadshare qhsong hellojimmywang feng-y liuzexin apple006 harry775 libragen yeshm jinares mj520 heyker fredchenbj logonod stormltf cisoft8341 forget6 mailmahee 532848980 drag0nglass betashepherd yangwu neil88 bigbigwei xifenmin isgasho danceiny yuanfeng0905 awesomegolang negashev jklofs awesomedatatool wbmcloud tomyang1898 xjump hbprotoss skyformat99 searise ming-fork chivasto 2014214128 openmiddleware cnsky2016 junftnt benguang yxjsnj2018 coreycheung zhoffice shenlijungg csyofchina wangzhibin kana112233 didi20gh antewu best199101 lijianwei123 nkzxw ci123chain startime-h yutiansut hoangpq razertory hubolin slow-zhang kaisatec hzliangbin

tidis's Issues

del command bug

删除string 类型的key值时，不存在key的情况下也会返回1

can keys be stored in memory ?

I wander if keys can be stored in memory, and have 2 or 3 replicas across nodes.
i don't need data persistence, i don't have expensive ssd disk. i just need speed and survive one node crash.
if could, how ?

Build Error: l.SetLevel undefined (type *logrus.Logger has no field or method SetLevel)

I am getting below error

make
go build -o bin/tidis-server cmd/server/*
# github.com/yongman/go/log
../../go/src/github.com/yongman/go/log/log.go:36:4: l.SetLevel undefined (type *logrus.Logger has no field or method SetLevel)
../../go/src/github.com/yongman/go/log/log.go:39:4: l.SetLevel undefined (type *logrus.Logger has no field or method SetLevel)
../../go/src/github.com/yongman/go/log/log.go:42:4: l.SetLevel undefined (type *logrus.Logger has no field or method SetLevel)
../../go/src/github.com/yongman/go/log/log.go:45:4: l.SetLevel undefined (type *logrus.Logger has no field or method SetLevel)
../../go/src/github.com/yongman/go/log/log.go:48:4: l.SetLevel undefined (type *logrus.Logger has no field or method SetLevel)
../../go/src/github.com/yongman/go/log/log.go:51:4: l.SetLevel undefined (type *logrus.Logger has no field or method SetLevel)
make: *** [build] Error 2

add transactions operation api

multi/exec/discard should be implemented

do you have a plan to support more complex feature for command SET just like redis

建议测试换成ssd

测试性能数据太难看了，建议用换成ssd，真实环境肯定会用ssd，这样能够提高产品认同度。

do u hv the benchmark for redis server next to tidis benchmark?

support redis eval script (lua)

Can we do support the EVAL command?

https://redis.io/commands/eval

discard bug

discard命令的返回值是异常的

sort set withscores bug

当执行时

zrangebyscore dss 0 0 withscores limit 0 2999

返回空值，可是空值的返回码和redis的不一致
redis的为：

而tidis中为

Lazy delete ttl=0

懒删除没有生效，ttl=0时,在轮询删除没有轮询到时，还是可以查到值

172.16.1.32:0>set dssvalue1 dsada EX 1
"OK"

172.16.1.32:0>set dssvalue1 dsada EX 1
"OK"

172.16.1.32:0>ttl dssvalue1
"0"

172.16.1.32:0>get dssvalue1
"dsada"

remove expire key when delete user key meta

remove expire key meta when clear or delete a user key which already assigned expiration

add ttl support to set and zset type

add ttl support to set and zset

sort set bug

sort set 在redis中的score值为浮点型，而在tidis中该值被限定为长整型

suggestion

to author :
Big God， this project is very niubility, hope you bring more test and enhancement, keep it stable and niubihonghong.
I will deploy your niubi program for production use in my future projects.
Hope you happy, wealthy and healthy.
thanks.

Is tidis production ready?

Is tidis production ready?
Do you use tidis in your compony ?

Question: Multi/exec transaction behavior

See the below example where I initiated a transaction with setting two hash values. First one successful and 2nd failed. I thought a transaction would roll back 1st operation because 2nd failed.
but that's not the case.

127.0.0.1:5379> multi
OK
127.0.0.1:5379> hsetnx key1 field3 val3
QUEUED
127.0.0.1:5379> hsetnx key1 field1 val2
QUEUED
127.0.0.1:5379> exec
1) (integer) 1
2) (integer) 0
127.0.0.1:5379> hget key1 field3
"val3"
127.0.0.1:5379>

大神求助

测试发现为什么 tidis 服务运行一段时间会自动结束呢

sclear/zclear/ldel bug

sclear/zclear/ldel 在key对应的value为空时命令会阻塞

level=error msg="[pd] failed to get cluster id: rpc error: code = Unavailable desc = grpc: the connection is unavailable"

Any documentation on transaction support

I have a use-case where two key-val should be inserted atomically (if not exist)and either insert fails should roll back. The 2nd key is a secondary index.

Key1= Hash1
Value1= Token1

Key2= Token1
Value2= json payload

How can I insert both atomically? Now if Key1 or Key2 (Token1) already exists, it should roll back the operation.

I see HmsetWithTxn but would like to see support for HmsetnxWithTxn which allows insertion of multiple keys if don't exist in a transaction.

tidis optimize to delete big key later ?

I thank tidis delete big key, it is slow . example, delete a 100w items hash data.
thank you .

When production ready?

[Proposal] Any plan to use RawKV apis?

TiKV provides atomic Get/Put/Scan/Delete APIs at RawAPI layer with a higher performance than Transactional API.
Maybe some scenarios people just want a high performance KV database with scan API (just like dynamoDB & HBase), using RawAPI in these scenarios seems a good idea.

Hello from TiKV

Hi @yongman

I am very glad to see this project, do you meet any problem? If yes, please tell me.
It is very appreciated that you can give me your Wechat or email, and we can have a deep discussion later.

Thank you!

does tidis support zset data type?

does tidis support zset data type?
Since there's no zset commands listed in README.

Why do not floating-point scores be supported in zset?

In redis command, the score value of Zset is floating point number, while in Tidis it is int64.

ERRO[0005] [pd] create tso stream error: rpc error: code = Unavailable desc = grpc: the connection is unavailable

I created a TiDB cluster using docker-compose file as below. When I use tidis-server using one or more pd addresses, it gives below error. I tried replacing 127.0.0.1 with 0.0.0.0. I am able to connect using MySQL connect.

./bin/tidis-server -backend "127.0.0.1:32768,127.0.0.1:32769,127.0.0.1:32770"
\INFO[0000] server started
INFO[0000] [pd] create pd client with endpoints [127.0.0.1:32768 127.0.0.1:32769 127.0.0.1:32770]
INFO[0000] [pd] leader switches to: http://pd2:2379, previous:
INFO[0000] [pd] init cluster id 6574888069347456351
ERRO[0005] [pd] create tso stream error: rpc error: code = Unavailable desc = grpc: the connection is unavailable
FATA[0005] rpc error: code = Unavailable desc = grpc: the connection is unavailable
ERRO[0005] updateTS error: rpc error: code = Unavailable desc = grpc: the connection is unavailable

docket ps :

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                               NAMES
3f118e56f9ec        pingcap/tispark:latest       "/opt/spark/sbin/sta…"   About an hour ago   Up About an hour    0.0.0.0:38081->38081/tcp                            tidb-docker-compose_tispark-slave0_1
dea5c36bcafe        pingcap/tispark:latest       "/opt/spark/sbin/sta…"   About an hour ago   Up About an hour    0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp      tidb-docker-compose_tispark-master_1
e544f04fe431        pingcap/tidb:latest          "/tidb-server --stor…"   About an hour ago   Up About an hour    0.0.0.0:4000->4000/tcp, 0.0.0.0:10080->10080/tcp    tidb-docker-compose_tidb_1
2e7c32ebf66f        pingcap/tikv:latest          "/tikv-server --addr…"   About an hour ago   Up About an hour    20160/tcp                                           tidb-docker-compose_tikv2_1
0a5852beec59        pingcap/tikv:latest          "/tikv-server --addr…"   About an hour ago   Up About an hour    20160/tcp                                           tidb-docker-compose_tikv1_1
fe457ad77c3c        pingcap/tikv:latest          "/tikv-server --addr…"   About an hour ago   Up About an hour    20160/tcp                                           tidb-docker-compose_tikv0_1
f3f88f3b9ba3        pingcap/pd:latest            "/pd-server --name=p…"   About an hour ago   Up About an hour    2380/tcp, 0.0.0.0:32770->2379/tcp                   tidb-docker-compose_pd0_1
f766338f652c        pingcap/pd:latest            "/pd-server --name=p…"   About an hour ago   Up About an hour    2380/tcp, 0.0.0.0:32769->2379/tcp                   tidb-docker-compose_pd2_1
1f5203fedbf1        pingcap/pd:latest            "/pd-server --name=p…"   About an hour ago   Up About an hour    2380/tcp, 0.0.0.0:32768->2379/tcp                   tidb-docker-compose_pd1_1
572eb8f2ef2e        pingcap/tidb-vision:latest   "/bin/sh -c 'sed -i …"   About an hour ago   Up About an hour    80/tcp, 443/tcp, 2015/tcp, 0.0.0.0:8010->8010/tcp   tidb-docker-compose_tidb-vision_1

I am able to connect using MySQL client as below

mysql -h 127.0.0.1 -P 4000 -u root -D test
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7.10-TiDB-v2.1.0-beta-21-g73dc86c MySQL Community Server (Apache License 2.0)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

please grant me to pull request

I add setbit function (just for setbit command) for tidis, but cannot pull request to you.
please grant me to pull request.

runtest failed 4 test cases

`

FAIL: test_expire (test_string.StringTest)

Traceback (most recent call last):
File "/disk/ssd/tidis/tests/test_string.py", line 165, in test_expire
self.assertIsNone(self.r.get(self.k1))
AssertionError: 'value1' is not None

======================================================================
FAIL: test_expireat (test_string.StringTest)

Traceback (most recent call last):
File "/disk/ssd/tidis/tests/test_string.py", line 175, in test_expireat
self.assertIsNone(self.r.get(self.k1))
AssertionError: 'value1' is not None

======================================================================
FAIL: test_pexpire (test_string.StringTest)

Traceback (most recent call last):
File "/disk/ssd/tidis/tests/test_string.py", line 146, in test_pexpire
self.assertIsNone(self.r.get(self.k1))
AssertionError: 'value1' is not None

======================================================================
FAIL: test_pexpireat (test_string.StringTest)

Traceback (most recent call last):
File "/disk/ssd/tidis/tests/test_string.py", line 156, in test_pexpireat
self.assertIsNone(self.r.get(self.k1))
AssertionError: 'value1' is not None

Ran 82 tests in 137.529s

FAILED (failures=4)
`

Hi, I just built tidis from latest source, and ran the runtest script, however, got 4 test cases failed as above, what might be the reason? Could be due to pd/tikv? Thanks.

Keys and scans are needed

缺少keys和scans在日常运维中将是个灾难

Implement BLPOP functionality

Excellent project!

I'm currently using LedisDB [1] in my Golang project, but would prefer to have a 100% Golang option.

What are your thoughts on implementing BLPOP functionality in tidis?

[1] https://github.com/siddontang/ledisdb

Edit: Added link to LedisDB

can multi keys which belong to different region execute successfully in one set command

can multi keys which belong to different region execute successfully in one set command？
for example:
mset(key1,key2);
key1 belongs to region1, and key2 belongs to region2

discard bug

discard命令在redis协议中，发送完成后并不会关闭连接，而在tidis中会关闭连接。这样的话在pipeline中无法做到取消事务操作。

Add docker support

Add docker support and binary release downloads.

Can you make tidis compatible with other storage backend?

something like blevesearch's backend storage where we can change to other storage solution other than tikv

Big key deletion failed if items changed all the time

Big list\set\zset\hash keys deletion will be last for a few second, if some changed to this big key during the deletion, delete operation will commit failed because of transaction conflict.

So some meta control mechanism must be added to avoid this scenario.

Build Error: cannot find package

Hi,
I am trying to build tidis with the make command. I get the following error. Can you give me some help?

CGO_ENABLED=0 go build -o bin/tidis-server cmd/server/*
cmd/server/main.go:16:2: cannot find package "github.com/yongman/go/log" in any of:
/usr/local/Cellar/go/1.11.2/libexec/src/github.com/yongman/go/log (from $GOROOT)
/Users/lei.gu/go/src/github.com/yongman/go/log (from $GOPATH)
cmd/server/main.go:17:2: cannot find package "github.com/yongman/tidis/config" in any of:
/usr/local/Cellar/go/1.11.2/libexec/src/github.com/yongman/tidis/config (from $GOROOT)
/Users/lei.gu/go/src/github.com/yongman/tidis/config (from $GOPATH)
cmd/server/main.go:18:2: cannot find package "github.com/yongman/tidis/server" in any of:
/usr/local/Cellar/go/1.11.2/libexec/src/github.com/yongman/tidis/server (from $GOROOT)
/Users/lei.gu/go/src/github.com/yongman/tidis/server (from $GOPATH)
make: *** [build] Error 1

I saw three imports in cmd/server/main.go
"github.com/yongman/go/log"
"github.com/yongman/tidis/config"
"github.com/yongman/tidis/server"

where can I find these three dependencies?

panic: interface conversion: interface {} is nil, not int

WARN[0810] ttl checker hashkey failed, [tikv:9001]PD server timeout[try again later]
WARN[0810] BatchInTxn execute failed
panic: interface conversion: interface {} is nil, not int

goroutine 135 [running]:
github.com/yongman/tidis/tidis.(*ttlChecker).Run(0xc42015c4c0)
	/Users/xxxx/projects/go/src/github.com/yongman/tidis/tidis/ttl.go:169 +0x1350
created by github.com/yongman/tidis/server.(*App).Run
	/Users/xxxx/projects/go/src/github.com/yongman/tidis/server/app.go:83 +0x197

sinter/sdiff/sunion/sunionstore/sdiffstore/sinterstore bug

sinter/sdiff/sunion/sunionstore/sdiffstore/sinterstore当传入的set的key出现空值时会抛异常

sinter dssSet3 dssSet2

cat redis_command_zset_310100.log | redis-cli -h 10.125.233.215 -p 5379 --pipe
Error reading from the server: Connection reset by peer

hi, what's problem of this sence

$ ./tidis-server -backend :32768
INFO[0000] server started                               
INFO[0000] [pd] create pd client with endpoints [:32768] 
INFO[0000] [pd] leader switches to: http://pd1:2379, previous:  
INFO[0000] [pd] init cluster id 6537956209576529350     
ERRO[0000] [pd] create tso stream error: rpc error: code = Unavailable desc = grpc: the connection is unavailable 
FATA[0000] rpc error: code = Unavailable desc = grpc: the connection is unavailable

add unit test for each api

Unit tests should be added make sure every api works fine after some changed commited.

HdelWithTxn: panic: interface conversion: interface {} is nil, not uint64

127.0.0.1:5379> hdel hash2 token3
Error: Connection reset by peer

panic: interface conversion: interface {} is nil, not uint64

goroutine 259 [running]:
github.com/yongman/tidis/tidis.(*Tidis).HdelWithTxn(0xc4200a2a80, 0x1880160, 0xc420466b00, 0xc4200c0b60, 0x5, 0x5, 0xc4208bdb60, 0x1, 0x1, 0x1689134, ...)
	/Users/user/projects/go/src/github.com/yongman/tidis/tidis/t_hash.go:205 +0x2e8
github.com/yongman/tidis/tidis.(*Tidis).Hdel.func1(0x1880160, 0xc420466b00, 0xc420466b00, 0x0, 0x0, 0x1819d20)
	/Users/user/projects/go/src/github.com/yongman/tidis/tidis/t_hash.go:124 +0x81
github.com/yongman/tidis/store/tikv.(*Tikv).BatchInTxn(0xc4201a03a0, 0xc420401580, 0x18f3eb0, 0xc420991cb0, 0xc4207e4000, 0xc420991ca0)
	/Users/user/projects/go/src/github.com/yongman/tidis/store/tikv/tikv.go:536 +0xcf
github.com/yongman/tidis/tidis.(*Tidis).Hdel(0xc4200a2a80, 0xc4200c0b60, 0x5, 0x5, 0xc4208bdb60, 0x1, 0x1, 0x10, 0x1, 0xc420344768)
	/Users/user/projects/go/src/github.com/yongman/tidis/tidis/t_hash.go:128 +0xcc
github.com/yongman/tidis/server.hdelCommand(0xc4201a2c60, 0xc42028fb00, 0xc4200c0b6c)
	/Users/user/projects/go/src/github.com/yongman/tidis/server/command_hash.go:119 +0x193
github.com/yongman/tidis/server.(*Client).execute(0xc4201a2c60, 0x4, 0x18a712f)
	/Users/user/projects/go/src/github.com/yongman/tidis/server/client.go:350 +0x1fd
github.com/yongman/tidis/server.(*Client).handleRequest(0xc4201a2c60, 0xc4208bdb30, 0x3, 0x3, 0x0, 0x0)
	/Users/user/projects/go/src/github.com/yongman/tidis/server/client.go:335 +0x5a0
github.com/yongman/tidis/server.(*Client).connHandler(0xc4201a2c60)
	/Users/user/projects/go/src/github.com/yongman/tidis/server/client.go:211 +0x190
created by github.com/yongman/tidis/server.ClientHandler
	/Users/user/projects/go/src/github.com/yongman/tidis/server/client.go:86 +0x254

zrangebyscore bug

当limit值为-1时会抛错

172.16.1.32:0>zrangebyscore dss 0 100 LIMIT 0 -1
Connection error:Execution timeout

yongman / tidis Goto Github PK

tidis's People

Contributors

Stargazers

Watchers

Forkers

tidis's Issues

`

FAIL: test_expire (test_string.StringTest)

====================================================================== FAIL: test_expireat (test_string.StringTest)

====================================================================== FAIL: test_pexpire (test_string.StringTest)

====================================================================== FAIL: test_pexpireat (test_string.StringTest)

Recommend Projects

Recommend Topics

Recommend Org

======================================================================
FAIL: test_expireat (test_string.StringTest)

======================================================================
FAIL: test_pexpire (test_string.StringTest)

======================================================================
FAIL: test_pexpireat (test_string.StringTest)