fabianishere / hugetable

Re-implementation of Google BigTable in Scala

HugeTable

Open-source implementation of Google's BigTable.

Getting Started

Obtaining the source code

Download the source code by running the following code in your command prompt:

git clone https://github.com/fabianishere/in4391.git

or simply grab a copy of the source code as a Zip file.

Building the project

For building the project, we use Gradle. To perform a build, enter the following in your command prompt in the root directory of the project:

./gradlew build

To build the binaries, make sure you also run the following command:

./gradlew installDist

Setup the environment

HugeTable requires the presence of a Hadoop HDFS cluster for distributed file storage and a ZooKeeper cluster for synchronization between the nodes. To simplify the deployment of HugeTable within a test environment, you may use the single node test environment we provide in the repository.

To start the test environment, please run the following command:

build/install/hugetable/bin/htable-test-env    

which will start a HDFS node on port 9000 and a ZooKeeper node on port 2181.

Start a HugeTable cluster

The next step is to start one or multiple HugeTable nodes, which together will form a cluster. You may start a node as follows:

build/install/hugetable/bin/htable-server-cli --zookeeper localhost:2181 --hadoop hdfs://localhost:9000 --port 8080

Make sure you specify the correct ports and don't use duplicate ports for different nodes.

Communicating with a HugeTable cluster

You may communicate with the HugeTable cluster by using the command-line client (htable-client-cli) or the Scala client library (htable-client). To use the command-line interface interactively, enter the following command in your command prompt:

build/install/hugetable/bin/htable-client-cli -i --zookeeper localhost:2181   

Make sure again that you specify the correct ZooKeeper port. Then, enter help to obtain the possible commands you can execute in the cluster.

License

HugeTable is available under the MIT license.

hugetable's Issues

Define API for accessing and mutating database

Use gRPC for defining the services.

Implement group membership

This issue is concerned with implementing group membership: keeping track of the active members in the HugeTable fleet and having the master reap group members that have lost connection.

Starting point: https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/nodes/GroupMember.java

Relevant literature:

Bigtable uses Chubby to keep track of tablet servers. When a tablet server starts, it creates, and acquires an exclusive lock on, a uniquely-named file in a specific Chubby directory. The master monitors this directory (the servers directory) to discover tablet servers. A tablet server stops serving its tablets if it loses its exclusive lock: e.g., due to a network partition that caused the server to lose its Chubby session. (Chubby provides an efficient mechanism that allows a tablet server to check whether it still holds its lock without incurring network traffic.)
The master is responsible for detecting when a tablet server is no longer serving its tablets, and for reassign- ing those tablets as soon as possible. To detect when a tablet server is no longer serving its tablets, the master periodically asks each tablet server for the status of its lock. If a tablet server reports that it has lost its lock, or if the master was unable to reach a server during its last several attempts, the master attempts to acquire an exclusive lock on the server’s file. If the master is able to acquire the lock, then Chubby is live and the tablet server is either dead or having trouble reaching Chubby, so the master ensures that the tablet server can never serve again by deleting its server file. Once a server’s file has been deleted, the master can move all the tablets that were previously assigned to that server into the set of unassigned tablets

Define interface for accessing storage layer