focinfi / oncekv

Easy scaling distributed Key/Value dababase

oncekv

A key/value database, each pair of key/value can be set only one time.

Why limit the set time?

Cause we design oncekv for the scenarios like once a key set, it will never be changed, for example, a versioned cache file database.

At the same time, limitation gives us chances to improve performance but still easily to scale.

Design

1. Performance, using groupcache for read cache.
2. Reliable, using raft for consensus and boltdb for underlying database.
3. Easy horizontal scaling, simple HTTP API, inspired by hraftf.

Have a try

1.Change working directory into example

cd $GOPATH/src/github.com/Focinfi/oncekv/example

2.Start the admin server

go run ./admin/admin.go

This server provide API of known server list for the front end porject and starts the database master server.

3.Start the admin front end server

cd $GOPATH/src/github.com/Focinfi/oncekv/admin/oncekv
npm run dev

It shows only title, now, let's add a database node.

4.Start one database node

bash ./databases/node/node1.sh

Cause it's the only node of this raft cluster, so it becomes the Leader. And the admin page will show its stats.

Then add another node by bash ./databases/node/node2.sh

5.Start the cache cluster

Start the master:

go run ./cache/master/master.go;

Start the cache first node:

bash ./cache/node/node1.sh

Start the cache second node:

bash ./cache/node/node2.sh

Start the cache third node:

bash ./cache/node/node3.sh

6.Set/Get some key/vlaue pairs:

# set foo/bar into database leader
curl -XPOST http://127.0.0.1:11000/key -d '{"key":"foo","value":"bar"}'

# get for cache node-1
curl -XGET http://127.0.0.1:5551/key/foo

# get for cache node-2
curl -XGET http://127.0.0.1:5552/key/foo

# get for cache node-3
curl -XGET http://127.0.0.1:5553/key/foo

#... try more key/value

And then your admin page will look like this:

Powered by vuejs and element

oncekv's Issues

Plan for v0.3.0

1. replace http with net/rpc for master/node communication
2. database shard

[BUG] accept: too many open files in system

When a cache or database node run locally about 20 minutes, it will raise a error: accept: too many open files in system.

Potential problem:

1. websocket, close connection correctly
2. mac max open file limitaion

Plan for v0.2.0

1. Complete test, add test CI script. (failed to integerate into coveralls.io)
2. Complete document
3. Out of box play script. (write tutorial in README.md)