sidecus / rkv Goto Github PK

Right now leader sends snapshot to follower using RPC stream. Follower receives it as RPC message stream and then converts it into a byte reader and copies it to a local file to ensure integrity and avoid partial snapshot apply. The file is then used to install the snapshot. When it's done client sends reply (reusing AE reply) back to leader.

The RPC message stream to byte reader conversion is done at kvstore package, and raft follower itself has no knowledge about individual chunk, and only sends one reply back.

When the snapshot is large, it takes time and leader might start sending another snapshot upon the 2nd heartbeat - it'll be ignored by the follower since it's a duplicate but it wastes bandwidth and compute on both leader and follower.

We'll need to create some "awareness" upon snapshot replication on the leader side to avoid this.

And btw it can also happen to normal log replication AE requests when follower is slow processing. Impact is much lower though.

The benchmark capability in rkvclient can be used to stress rkv. On a 3 node cluster using loopback, 1000 relatively small concurrent request takes about 150ms.

However, the system drops cmds when testing with 10000 concurrent requests, e.g. using below command:

./rkvclient benchmark -address localhost:27016 -times 10000

The reason is 10000 concurrent requests make the leader too busy. It has no bandwidth (or reaching grpc limit?) to send heartbeat in time. Eventually follower triggers new election - given we don't wait for commit (done asynchronously unlike the paper has), a lot of set requests were lost during new leader transition.

We should consider things like below:

1. Loosen election timeout upon certain conditions
2. Yield in leader to make sure heartbeat can be sent even when it's flooded with requests
3. Synchronous commit might be better for this kind of scenarios since it "forces" leader to do so - slower commit but guaranteed no data loss upon success