Distributed In-Memory Key Value Store
This project is based off of Amazon's Dynamo paper with certain design modifications as specified in the report.
The components present are Client, Proxy Server, Request Coordinators and Memory Nodes.
Client sends the get or put request to the Proxy server to retrieve the IP address of the request coordinator. Primary purpose of request proxy server is to balance the load of incoming requests to all the coordinators in a round-robin fashion. Upon receiving the IP of the current request coordinator, client now sends the request directly to the coordinator. Note that this process happens in the background and not visible to the client. This is done so that proxy server only does the work of load balancing.
Request coordinator now computes hash from the key to determine which memory node to choose from the ring structure (explained below) to store this key value pair. The request coordinator then sends the request to the primary memory node and to it's next N replicas in the ring at the same time. We follow quorum protocol to acknowledge successful read or write operation. After successful acknowledgement from all at least W or R memory nodes, request coordinator acknowledges the client about the success or failure of the request.
- put
- get
A client can interact with system via simple REST calls. A get request with the key would fetch the value for that key and a post request with a key value pair would set the key in the memory node. If a post request with an existing key is set, then it will overwrite the previously set value. These requests can be sent by simple curl commands or by using utilities like Postman. To make the task of sending these requests easier, we've integrated the client with Slack so that requests can now be sent by using Slack commands instead of typing commands on terminal
- Clone this repo
- Create a jar file containing code in the src directory
- Install Java v8 for proxy server, request coordinator and memory nodes. Install Java, Nodejs and npm for client. You can have a dedicated machine for proxy server, request coordinator and memory nodes can run on same machine. Since replication factor is 2, we need atleast one request coordinator and 2 memory nodes for successful write operation and 1 memory node for successful operation as per quorom protocol.
- Run the following files in sequence:
4.1 Start the proxy server on a machine:java -cp jarfile.jar kv_proxy.ProxyProc
4.2 Start n request coordinators as needed passing prox:java -cp jarfile.jar kv_requestCoordinator.Server <proxy server IP>
4.3 Start atleast 2 memory nodes:java -cp jarfile.jar kv_memNodes.MemNodeProc <proxy server IP>
4.4 Start client:java -cp jarfile.jar kv_client.ClientServer <proxy server IP>
4.5 Optionally you send put/get request via Slack slash commands. Follow this tutorial to set up slack slash command for your team and add SLACK_TOKEN_GET and SLACK_TOKEN_PUT to your environment variables. You will also need to use ngrok to establish a tunnel for your slack slash command inorder to run slash commands on local server.
Go to kv_slack folder, runnpm install
to install all dependencies, runnode client.js
to start the Slack server. - Perform put/get operations in the datastore using HTTP requests. You can use cURL, Postman or Slack to perform HTTP requests.
Postman put operation:
Slack put operation:
Slack get operation: