ledisdb / ledisdb

A high performance NoSQL Database Server powered by Go

LedisDB

Ledisdb is a high-performance NoSQL database library and server written in Go. It's similar to Redis but store data in disk. It supports many data structures including kv, list, hash, zset, set.

LedisDB now supports multiple different databases as backends.

Features

• Rich data structure: KV, List, Hash, ZSet, Set.
• Data storage is not limited by RAM.
• Various backends supported: LevelDB, goleveldb, RocksDB, RAM.
• Supports Lua scripting.
• Supports expiration and TTL.
• Can be managed via redis-cli.
• Easy to embed in your own Go application.
• HTTP API support, JSON/BSON/msgpack output.
• Replication to guarantee data safety.
• Supplies tools to load, dump, and repair database.
• Supports cluster, use xcodis.
• Authentication (though, not via http).
• Repair integrated: You can use ledis repair to repair broken databases and ledis repair-ttl to repair a very serious bug for key expiration and TTL if you upgraded from v0.4.

Build from source

Create a workspace and checkout ledisdb source

git clone [email protected]:ledisdb/ledisdb.git
cd ledisdb

#set build and run environment 
source dev.sh

make
make test

Then you will find all the binary build on ./bin directory.

LevelDB support

• Install leveldb and snappy.

  LedisDB supplies a simple script to install leveldb and snappy:

    sudo sh tools/build_leveldb.sh
  

  It will install leveldb at /usr/local/leveldb and snappy at /usr/local/snappy by default.

  LedisDB uses the modified LevelDB for better performance. Details.

  You can easily use other LevelDB versions (like Hyper LevelDB or Basho LevelDB) instead, as long as the header files are in include/leveldb, not include/hyperleveldb or any other location.

• Set LEVELDB_DIR and SNAPPY_DIR to the actual install path in dev.sh.

• make clean && make

RocksDB support

• Install rocksdb(5.1+)(make shared_lib) and snappy first.

  LedisDB has not yet supplied a simple script to install.

• Set ROCKSDB_DIR and SNAPPY_DIR to the actual install path in dev.sh.

• make clean && make

If the RocksDB API changes, LedisDB may not build successfully. LedisDB currently supports RocksDB version 5.1 or later.

Choose store database

LedisDB now supports goleveldb, leveldb, rocksdb, and RAM. It will use goleveldb by default.

Choosing a store database to use is very simple.

• Set in server config file

  db_name = "leveldb"

• Set in command flag

  ledis -config=/etc/ledis.conf -db_name=leveldb

Flag command set will overwrite config setting.

Lua support

Lua is supported using gopher-lua, a Lua VM, completely written in Go.

Configuration

LedisDB uses toml as the configuration format. The basic configuration ./etc/ledis.conf in LedisDB source may help you.

If you don't use a configuration, LedisDB will use the default for you.

Server Example

//set run environment if not
source dev.sh

./bin/ledis -config=/etc/ledis.conf

//another shell
./bin/ledis cli -p 6380

ledis 127.0.0.1:6380> set a 1
OK
ledis 127.0.0.1:6380> get a
"1"

//use curl
curl http://127.0.0.1:11181/SET/hello/world
→ {"SET":[true,"OK"]}

curl http://127.0.0.1:11181/0/GET/hello?type=json
→ {"GET":"world"}

Package Example

import (
    lediscfg "github.com/ledisdb/ledisdb/config"
    "github.com/ledisdb/ledisdb/ledis"
)

# Use Ledis's default config
cfg := lediscfg.NewConfigDefault()
l, _ := ledis.Open(cfg)
db, _ := l.Select(0)

db.Set(key, value)

db.Get(key)

Replication Example

Set slaveof in config or dynamiclly

ledis cli -p 6381 

ledis 127.0.0.1:6381> slaveof 127.0.0.1 6380
OK

Cluster support

LedisDB uses a proxy named xcodis to support cluster.

CONTRIBUTING

See [CONTRIBUTING.md] .

Benchmark

See benchmark for more.

Todo

See Issues todo

Client

See Clients to find or contribute LedisDB client.

Links

• Official Website
• GoDoc
• Server Commands

Caveat

• Changing the backend database at runtime is very dangerous. Data validation is not guaranteed if this is done.

Requirement

• Go version >= 1.11

Related Repos

• pika

Donate

If you like the project and want to buy me a cola, you can through:

PayPal	微信
	[

Feedback

• Gmail: [email protected]

ledisdb's Issues

error input params test

zunionstore zinterstore support

http://redis.io/commands/zunionstore

http://redis.io/commands/zinterstore

boltdb and kv support

boltdb
kv

LedisDB on Node.js

Hi guys!

Is there a way to use LedisDB on Node.js?

Thanks

ledisdb cluster support?

@silentsai @wenyekui @holys @tiaotiaoyly

I think we should begin to support cluster, qpush now uses 2 ledis-server, maybe more later.

I think it is very complicated for redis3.0 cluster solution, (if someone want to learn it and can promise that has the power to do it, very welcome!!!) .

I want to use a proxy solution, like https://github.com/siddontang/mixer or https://github.com/twitter/twemproxy

Some features:

• customize rule to shard with db + key
• handle failover using backup server, replication may be very importance
• double write?
• cluster management, monitor
• auto resharding， maybe very hard, but easier than MySQL
• more......

any suggestion?

replication enhance

refer mysql, maybe refactor

Go dependencies

Perhaps instead of bootstrap.sh where you go get a bunch of libraries, instead of Goop (https://github.com/nitrous-io/goop) or Godeps (https://github.com/tools/godep) .. these are two popular ways of managing dependencies in a Go project. 👍

ttl iterator limit range cpu 100%

Only appears once and not reappears anymore.

Record here to pay attention to.

tokuft support

tokuft https://github.com/Tokutek/ft-index

go-tokuft https://github.com/siddontang/go-tokuft

refer mdb or boltdb, tokuft supports transaction

Ledis-server cli, config file required

My first time playing with ledis-server, just wanted to offer some feedback. The "-config" file must exist or the server can't be started. Rather it makes more sense to either require config or db_name to start since the config file doesnt have anything in it besides the db_name.

Also, perhaps instead of a json config file you could use https://github.com/BurntSushi/toml ? .. doesnt really matter I guess, but even if you call it /etc/ledis.conf using a toml format, it will feel good and unixy, and just a better format for this kind of thing IMHO.

undefined: strdup

我仿照首页的指示写了个小程序，但遇到一个错误：
github.com/szferi/gomdb(.text): undefined: strdup

var cache *ledis.DB
var err error
if cache, err = func() (*ledis.DB, error) {
    cfg := config.NewConfigDefault()
    l, err := ledis.Open(cfg)
    if err != nil {
        return nil, err
    }
    return l.Select(0)
}(); err != nil {
    log.Fatal(err)
}

这个可能不是ledisdb的问题，但我没搜到什么办法跳过这个步骤，只查到说strdup本身不是一个标准的C方法。另外我用默认的goleveldb，能不依赖gomdb这个库么？

我的环境是windows8，git shell，装了mingw-w64，Go1.3

lmdb sync configuration

• sync every commit
• sync N seconds
• async(no sync)

multi db instance

support multi disks ?

nodejs support

flush refactor

using scan, hscan ... to delete safely

benchmark

redis-benchmark -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q

Redis: only cache, no file save
SSDB
LedisDB + LevelDB
LedisDB + goleveldb
LedisDB + RocksDB
LedisDB + LMDB

Before every bench, remove all old data first

Set result in wiki

Add transaction support

It would be nice to have at least redis-like transactions.

redis-cli complete hook

replication enhance v2

i thought. if all servers have [ip1, ip2, ip3] synced list from master[ip1]. and if i request eg. "set" operation on ip2 and if ip1 alive -> just proxy it. (maybe keep set operation on local "tmp" table)
any new "slave-of" registers to master and all slaves can became master. becouse any slave knows where too route "write" request.
if master timeout - write to second ip :)
when server "wake ups" -> check several "masters" to get servers list.
i think you got idea ;)

a little bit from google groupcache - if client requests me ID and i dont have/or have in tmp table ant user request me and i check master i can commit "write" ? :) and "cache local" :)

what we can do with "tmp uncommited writes" then master goes down? commit them on second "master" ?

HyperLevelDB support?

https://github.com/rescrv/HyperLevelDB

why need clone to src/github.com/siddontang/ledisdb

I want to clone the repository to a folder not contain path src/github.com/siddontang/ledisdb

but when I follow the README do source dev.sh the shell crashed.

It seems that I must clone the project into src/github.com/siddontang/ledisdb, I think it is not necessary.

Add support for boltdb?

Hi all!
Is it possible to add support for boltdb to ledis?
BoltDB is quite similar to goleveldb and attempts to gain some momentum (690 stars).

Many thanks for ledisdb, it's a great project!

Support "scan"?

It seems scan is unsupported after reading the docs and redis-cli test. I think scan is a unreplacable and useful feature.

support data struct hyperloglog

ledis-binlog enhance

patch binlog to database

restful api test

libleveldb.so.1: cannot open shared object file: No such file or directory

遇到这个问题请问怎么解决?

[root@ledisdb]#  ledis-server ./etc/ledis.json 
ledis-server: error while loading shared libraries: libleveldb.so.1: cannot open shared object file: No such file or directory

leveldb已经编译,并且库文件已经ln -s 到/usr/local/lib

lrwxrwxrwx   1 root root       35 7月  18 19:09 libleveldb.a -> /usr/local/leveldb/lib/libleveldb.a
lrwxrwxrwx   1 root root       36 7月  18 19:09 libleveldb.so -> /usr/local/leveldb/lib/libleveldb.so
lrwxrwxrwx   1 root root       38 7月  18 19:09 libleveldb.so.1 -> /usr/local/leveldb/lib/libleveldb.so.1
lrwxrwxrwx   1 root root       41 7月  18 19:09 libleveldb.so.1.17 -> /usr/local/leveldb/lib/libleveldb.so.1.17

embed lua?

use https://github.com/stevedonovan/luar or other?

but must support transaction first

rocksdb tunning

https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

add correspond option function to set

integrate goleveldb + leveldb + rocksdb

store snapshort support

add snapshot interface, all backend dbs must support

config and info command

dump and restore command support

see redis dump and restore

restful api

can I use it in production environment?

Now I am using Redis in a game server. But it may use too much memory, can I use this project in the game server? Thanks!

transaction support

db supports begin commit rollback

migrate command support

see redis migrate command migrate

support data structure sets

move command support

move
hmove
lmove
zmove
bmove

store Compact support

leveldb* support compact function to reduce db size

I don't know whether boltdb and lmdb support, google

tool:migrate from redis to ledis

scan refactor

scan kv
hscan hash

master branch: fail to build leveldb support

just issued cannot find lib snappy.

in dev.sh, fix as follow:

# check snappy
if [ -f $SNAPPY_DIR/include/snappy.h ]; then
    CGO_CFLAGS="$CGO_CFLAGS -I$SNAPPY_DIR/include"
    CGO_CXXFLAGS="$CGO_CXXFLAGS -I$SNAPPY_DIR/include"
    CGO_LDFLAGS="$CGO_LDFLAGS -L$SNAPPY_DIR/lib -lsnappy"
    LD_LIBRARY_PATH=$(add_path $LD_LIBRARY_PATH $SNAPPY_DIR/lib)
    DYLD_LIBRARY_PATH=$(add_path $DYLD_LIBRARY_PATH $SNAPPY_DIR/lib)
fi

binlog reduce performance

Maybe I will implement another better replication merchanism

c/c++ client api

use twemproxy

https://github.com/twitter/twemproxy

refactor configuration

use toml instead of json

official website refactor

use bootstrap?

Have you considered to use pure Go leveldb to replace cgo leveldb?

I like pure Go project.

Add scripting

To achieve atomicity with more sophisticated record manipulation, redis has a scripting support. http://redis.io/commands/eval
Lua scripting is very easy to embed and is super fast (especially luajit)
It would be great to add have something similar in ledisdb.