datakungfu / leveldb-server

leveldb-server

• Async leveldb server and client based on zeromq
• Storage engine leveldb
• Networking library zeromq
• We use leveldb-server at Safebox

License

New BSD license. Please see license.txt for more details.

Features

• Very simple key-value storage
• Data is sorted by key – allows range queries
• Data is automatically compressed
• Can act as persistent cache
• For our use at Safebox it replaced memcached+mysql
• Simple backups cp -rf level.db backup.db
• Networking/wiring from zeromq messaging library – allows many topologies
• Async server for scalability and capacity
• Sync client for easy coding
• Easy polyglot client bindings. See zmq bindings

>>> db.put("k3", "v3")
'True'
>>> db.get("k3")
'v3'
>>> db.range()
'[{"k1": "v1"}, {"k2": "v2"}, {"k3": "v3"}]'
>>> db.range("k1", "k2")
'[{"k1": "v1"}, {"k2": "v2"}]'
>>> db.delete('k1')
>>>

Will be adding high availability, replication and autosharding using the same zeromq framework.

Dependencies

python 2.6+ (older versions with simplejson)
zmq
pyzmq
leveldb
pyleveldb 

Getting Started

Instructions for an EC2 Ubuntu box.

Installing zeromq

wget http://download.zeromq.org/zeromq-2.1.10.tar.gz
tar xvfz zeromq-2.1.10.tar.gz
cd zeromq-2.1.10
sudo ./configure
sudo make
sudo make install

Installing pyzmq

wget https://github.com/zeromq/pyzmq/downloads/pyzmq-2.1.10.tar.gz
tar xvfz pyzmq-2.1.10.tar.gz
cd pyzmq-2.1.10/
sudo python setup.py configure --zmq=/usr/local/lib/
sudo python setup.py install

Installing leveldb and pyleveldb

svn checkout http://py-leveldb.googlecode.com/svn/trunk/ py-leveldb-read-only
cd py-leveldb-read-only
sudo compile_leveldb.sh
sudo python setup.py install

Starting the leveldb-server

> python leveldb-server.py -h
Usage: leveldb-server.py 
	-p [port and host settings] Default: tcp://127.0.0.1:5147
	-f [database file name] Default: level.db

leveldb-server

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -p HOST, --host=HOST  
  -d DBFILE, --dbfile=DBFILE
> python leveldb-server.py

Using the leveldb-client-py

> cd clients/py/
> sudo python setup.py install
> python 
>>> from leveldbClient import database
>>> db = database.leveldb()
>>> db.get("Key")
>>> db.put("K", "V")
>>> db.range()
>>> db.range(start, end)
>>> db.delete("K")

Backups

cp -rpf level.db backup.db

Known issues and work in progress

Would love your pull requests on

• Benchmarking and performance analysis
• client libraries for other languages
• [issue] zeromq performance issues with 1M+ inserts at a time
• [feature] timeouts in client library
• [feature] support for counters
• [feature] limit support in range queries
• Serializing and seperate threads for get/put/range in leveldb-server
• HA/replication/autosharding and possibly pub-sub for replication

Thanks

Thanks to all the folks who have contributed to all the dependencies. Special thanks to pyzmq/examples/mongo* author for inspiration.