ldmtam / go-bitcask

A Log-Structured Hash Table for Fast Key/Value Data implemented in Golang

GO-BITCASK

What is Go-Bitcask

A Log-Structured Hash Table for Fast Key/Value Data implemented in Golang

Go-Bitcask storage engine features:

1. Low latency per item read or written
   • Write latency: ~80µs
   • Read latency: ~17µs
2. High throughput, especially when writing an incoming stream of random items
   • Write throughput (per second): ~330_000
   • Read throughput (per second): ~1_000_000
3. Crash friendliness, both in terms of fast recovery and not losing data
4. Ease of backup and restore
5. Relatively simple, understandable code structure and data format with high unit-test coverage (84%)

For more details, please reference to Bitcask whitepaper

Usage

Create new instance of go-bitcask

db, err := gobitcask.New(
    WithDirName(dirName),
    WithSegmentSize(1024 * 1024 * 1024),  // 1 GB per segment file
    WithMergeOpt(&MergeOption{
        Interval: 6 * time.Hour,          // run compaction every 6 hours
        MinFiles: 5,                      // at least 5 data files before merging
    })
)
if err != nil {
    log.Fatalf("create gobitcask instance failed: %v", err)
}
defer db.Close()

Store key/value pair to storage

err := db.Put([]byte("key1"), []byte("val1"))
if err != nil {
    log.Fatalf("store data to gobitcask failed: %v", err)
}

Get value from storage by particular key

val, err := db.Get([]byte("key1"))
if err != nil && err != gobitcask.ErrKeyNotFound {
    log.Fatalf("get data from gobitcask failed: %v", err)
}
fmt.Println(string(val))

Delete a key/value pair by key

err := db.Delete([]byte("key1"))
if err != nil {
    log.Fatalf("delete data from gobitcask failed: %v", err)
}

List all keys

keys, err := db.ListKeys()
if err != nil {
    log.Fatalf("list keys from gobitcask failed: %v", err)
}

Fold over all key/value pairs

err := db.Fold(func(key, val []byte) error {
    fmt.Printf("key: %v, val: %v\n", string(key), string(val))
    return nil
})
if err != nil {
    log.Fatalf("fold over all key/value pairs failed: %v", err)
}

Benchmark

Machine information: Macbook Pro 2021 (16 inch), M1 Pro, 16 GB RAM, 512 GB SSD

Action	Num of keys	Key size (byte)	Value size (byte)	Duration (second)	RAM usage (MB)
Put	1 000 000	8	128	2.809	636
Put	1 000 000	16	512	3.331	1514
Put	10 000 000	8	128	30.047	5697
Random Get	1 000 000	8	128	0.769	N/A
Random Get	1 000 000	16	512	0.867	N/A
Random Get	10 000 000	8	128	8.81	N/A