afthill / siberite

Siberite is a simple, lightweight, leveldb backed message queue written in Go.

Siberite

Siberite is a simple leveldb backed message queue server
(twitter/kestrel, wavii/darner rewritten in Go).

Siberite is a very simple message queue server. Unlike in-memory servers such as redis, Siberite is designed to handle queues much larger than what can be held in RAM. And unlike enterprise queue servers such as RabbitMQ, Siberite keeps all messages out of process, using goleveldb as a persistent storage.

The result is a durable queue server that uses a small amount of in-resident memory regardless of queue size.

Siberite is based on Robey Pointer's Kestrel - simple, distributed message queue. Like Kestrel, Siberite follows the "No talking! Shhh!" approach to distributed queues: A single Siberite server has a set of queues identified by name. Each queue is a strictly-ordered FIFO, and querying from a fleet of Siberite servers provides a loosely-ordered queue. Siberite also supports Kestrel's two-phase reliable fetch: if a client disconnects before confirming it handled a message, the message will be handed to the next client.

Compared to Kestrel and Darner, Siberite is easier to build, maintain and distribute. It uses an order of magnitude less memory compared to Kestrel, but has less configuration far fewer features.

Siberite is used at Spyonweb.com.
We used to use Darner before, but got 2 large production queues corrupted at some point and decided to rewrite it in Go.

##Benchmarks

Siberite performance benchmarks

Build

Make sure your GOROOT and GOPATH are correct

go get github.com/bogdanovich/siberite
cd $GOPATH/src/github.com/bogdanovich/siberite
go get ./...
cd siberite
go build siberite.go
mkdir ./data
./siberite -listen localhost:22133 -data ./data
2015/09/22 06:29:38 listening on 127.0.0.1:22133
2015/09/22 06:29:38 initializing...
2015/09/22 06:29:38 data directory:  ./data

or download darwin-x86_64 or linux-x86_64 builds

Protocol

Siberite follows the same protocol as Kestrel, which is the memcache TCP text protocol.

List of compatible clients

Telnet example

telnet localhost 22133
Connected to localhost.
Escape character is '^]'.

set work 0 0 10
1234567890
STORED

set work 0 0 2
12
STORED

get work
VALUE work 0 10
1234567890
END

get work/open
VALUE work 0 2
12
END

get work/close
END

stats
STAT uptime 47
STAT time 1443308758
STAT version siberite-0.4.1
STAT curr_connections 1
STAT total_connections 1
STAT cmd_get 2
STAT cmd_set 2
STAT queue_work_items 0
STAT queue_work_open_transactions 0
END

# other commands:
# get work/peek
# get work/open
# get work/close/open
# get work/abort
# flush work
# delete work
# flush_all

TODO

• Add multiple consumers get queue_name:consumer_name/open

Not supported

• Waiting a given time limit for a new item to arrive /t= (allowed by protocol but does nothing)