praveenmunagapati / pika

Pika is a nosql compatible with redis, it is developed by Qihoo's DBA and infrastructure team

Pika

Introduction中文

Pika is a persistent huge storage service , compatible with the vast majority of redis interfaces (details), including string, hash, list, zset, set and management interfaces. With the huge amount of data stored, redis may suffer for a capacity bottleneck, and pika was born for solving it. Except huge storage capacity, pika also support master-slave mode by slaveof command, including full and partial synchronization. You can also use pika together with twemproxy or codis(pika has supported data migration in codis，thanks left2right) for distributed Redis solution

UserList

More

Feature

• huge storage capacity
• compatible with redis interface, you can migrate to pika easily
• support master-slave mode (slaveof)
• various management interfaces

For developer

RoadMap

• optimize engine nemo to improve list performance

Dependencies

• snappy - a library for fast data compression
• glog - google log library

Upgrade your gcc to version at least 4.8 to get C++11 support.

Supported platforms

• linux - Centos 5&6

• linux - Ubuntu

If it comes to some missing libs, install them according to the prompts and retry it.

Compile

Upgrade your gcc to version at least 4.8 to get C++11 support.

Get the source code

git clone https://github.com/Qihoo360/pika.git

Then compile pika, all submodules will be updated automatically.

make

Usage

./output/bin/pika -c ./conf/pika.conf

Performance (provided by deep011)

test environment

CPU module：Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz

CPU threads：56

MEMORY：256G

DISK：3T flash

NETWORK：10GBase-T/Full * 2

OS：centos 6.6

benchmark tools

vire-benchmark

Test 1

Purpose

With different number of pika worker threads, we test pika's max QPS.

Condition

pika db_size : 800G

value : 128bytes

Threads are not bound to CPU cores.

Result

Description : Horizontal axis is the number of worker threads; Vertical axis is the QPS. The value size is 128bytes. set3/get7 means 30% set and 70% get.

Conclusion

The best pika worker threads number is 20-24.

Test 2

Purpose

With the optimal worker threads number, we test pika's round-trip time.

Condition

pika db_size：800G

value：128bytes

Result

====== GET ======
  10000000 requests completed in 23.10 seconds
  200 parallel clients
  3 bytes payload
  keep alive: 1
99.89% <= 1 milliseconds
100.00% <= 2 milliseconds
100.00% <= 3 milliseconds
100.00% <= 5 milliseconds
100.00% <= 6 milliseconds
100.00% <= 7 milliseconds
100.00% <= 7 milliseconds
432862.97 requests per second

====== SET ======
  10000000 requests completed in 36.15 seconds
  200 parallel clients
  3 bytes payload
  keep alive: 1
91.97% <= 1 milliseconds
99.98% <= 2 milliseconds
99.98% <= 3 milliseconds
99.98% <= 4 milliseconds
99.98% <= 5 milliseconds
99.98% <= 6 milliseconds
99.98% <= 7 milliseconds
99.98% <= 9 milliseconds
99.98% <= 10 milliseconds
99.98% <= 11 milliseconds
99.98% <= 12 milliseconds
99.98% <= 13 milliseconds
99.98% <= 16 milliseconds
99.98% <= 18 milliseconds
99.99% <= 19 milliseconds
99.99% <= 23 milliseconds
99.99% <= 24 milliseconds
99.99% <= 25 milliseconds
99.99% <= 27 milliseconds
99.99% <= 28 milliseconds
99.99% <= 34 milliseconds
99.99% <= 37 milliseconds
99.99% <= 39 milliseconds
99.99% <= 40 milliseconds
99.99% <= 46 milliseconds
99.99% <= 48 milliseconds
99.99% <= 49 milliseconds
99.99% <= 50 milliseconds
99.99% <= 51 milliseconds
99.99% <= 52 milliseconds
99.99% <= 61 milliseconds
99.99% <= 63 milliseconds
99.99% <= 72 milliseconds
99.99% <= 73 milliseconds
99.99% <= 74 milliseconds
99.99% <= 76 milliseconds
99.99% <= 83 milliseconds
99.99% <= 84 milliseconds
99.99% <= 88 milliseconds
99.99% <= 89 milliseconds
99.99% <= 133 milliseconds
99.99% <= 134 milliseconds
99.99% <= 146 milliseconds
99.99% <= 147 milliseconds
100.00% <= 203 milliseconds
100.00% <= 204 milliseconds
100.00% <= 208 milliseconds
100.00% <= 217 milliseconds
100.00% <= 218 milliseconds
100.00% <= 219 milliseconds
100.00% <= 220 milliseconds
100.00% <= 229 milliseconds
100.00% <= 229 milliseconds
276617.50 requests per second

Conclusion

Both the 99.9% get/set RTT are below 2ms.

Test 3

Purpose

With the optimal worker threads number, we test the max qps of different commands.

Condition

pika worker threads：20

key count：10000

field count：100（except list）

value：128bytes

commands execute times：10000000(except lrange)

Result

PING_INLINE: 548606.50 requests per second
PING_BULK: 544573.31 requests per second
SET: 231830.31 requests per second
GET: 512163.91 requests per second
INCR: 230861.56 requests per second
MSET (10 keys): 94991.12 requests per second
LPUSH: 196093.81 requests per second
RPUSH: 195186.69 requests per second
LPOP: 131156.14 requests per second
RPOP: 152292.77 requests per second
LPUSH (needed to benchmark LRANGE): 196734.20 requests per second
LRANGE_10 (first 10 elements): 334448.16 requests per second
LRANGE_100 (first 100 elements): 50705.12 requests per second
LRANGE_300 (first 300 elements): 16745.16 requests per second
LRANGE_450 (first 450 elements): 6787.94 requests per second
LRANGE_600 (first 600 elements): 3170.38 requests per second
SADD: 160885.52 requests per second
SPOP: 128920.80 requests per second
HSET: 180209.41 requests per second
HINCRBY: 153364.81 requests per second
HINCRBYFLOAT: 141095.47 requests per second
HGET: 506791.00 requests per second
HMSET (10 fields): 27777.31 requests per second
HMGET (10 fields): 38998.52 requests per second
HGETALL: 109059.58 requests per second
ZADD: 120583.62 requests per second
ZREM: 161689.33 requests per second
PFADD: 6153.47 requests per second
PFCOUNT: 28312.57 requests per second
PFADD (needed to benchmark PFMERGE): 6166.37 requests per second
PFMERGE: 6007.09 requests per second

Test 4

Purpose

Compare the pika max qps with the redis.

Condition

pika worker threads：20

key count：10000

field count：100（except list）

value：128bytes

commands execute times：10000000(except lrange)

Redis version：3.2.0

Result

Documents

1. Wiki

Contact Us

Mail: [email protected]

QQ group: 294254078 For more information about Pika, Atlas and some other technology please pay attention to our Hulk platform official account