unshiftio / liferaft

Consensus protocol based on raft, but only for saving lifes and works in node as well as in browers.

liferaft

liferaft is an JavaScript implementation of the Raft consensus algorithm.

Installation

The liferaft module is distributed through npm and is compatible with browserify as well as node.js. You can install this module using:

npm install --save liferaft

Table Of Contents

• Installation
• Usage
  • Configuration
  • Events
  • LifeRaft.states
  • LifeRaft.{state}
  • LifeRaft#type()
  • LifeRaft#quorum()
  • LifeRaft#majority()
  • LifeRaft#indefinitely()
  • LifeRaft#packet()
  • LifeRaft#message()
  • LifeRaft#join()
  • LifeRaft#leave()
  • LifeRaft#promote()
  • LifeRaft#end()
  • LifeRaft#command()
• Log Replication
• Extending
• Transports
• License

Usage

In all examples we assume that you've imported the liferaft module as following:

'use strict';

var LifeRaft = require('liferaft')
  , raft = new Raft('address', { /* optional options */});

Please note that the instructions for Node.js and browser are exactly the same as we assume that your code will be compiled using a Browserify based system. The only major difference is that you probably don't need to configure a commit and append log (but this of course, fully optional).

The LifeRaft library is quite dumb by default. We try to be as lean and extendible as possible so you, as a developer, have complete freedom on how you want to implement Raft in your architecture. This also means that we ship this library without any build in transport. This allows you to use it with your existing technology stack and environment. If you want to use SharedWorkers as transport in the browser? Awesome, you can do that. Want to use it on node? There are literally thousands of different transport libraries that you can use.

Configuration

There are a couple of default options that you can configure in the constructor of your Raft:

• address A unique address of the node that we just created. If none is supplied we will generate a random UUID.
• heartbeat The heartbeat timeout. Make sure that this value is lower then your minimum election timeout and take message latency in consideration when specifying this and the minimum election value.
• election min Minimum election timeout.
• election max Maximum election timeout.
• threshold Threshold for when the heartbeat and latency is to close to the minimum election timeout.
• Log: An Log compatible constructor we which use for state and data replication.

The timeout values can be configured with either a number which represents the time milliseconds or a human readable time string such as 10 ms. The heartbeat timeouts are used to detect a disconnection from the LEADER process if no message has been received within the given timeout we assume its dead that we should be promoted to master. The election timeout is the time it may take to reach a consensus about the master election process. If this times out, we will start another re-election.

var raft = new Raft({
  'address': 'tcp://localhost:8089',
  'election min': '200 millisecond',
  'election max': '1 second'
});

As you might have noticed we're using two different styles of passing in the address to the raft instance, as address property in the options and as first argument in the constructor.

Events

The liferaft module is an EventEmitter at it's core and is quite chatty about the events it emits.

Event	Description
term change	The term has changed.
leader change	We're now following a newly elected leader.
state change	Our state/role changed.
heartbeat timeout	Heartbeat timeout, we're going to change to candidate.
data	Emitted by you, so we can work with the data.
vote	We've received a vote request.
leave	Node has been removed from the cluster.
join	Node has been added to the cluster.
end	This Raft instance has ended.
initialize	The node has been fully initialized.
error	An error happened while doing.. Things!
threshold	The heartbeat timeout is getting close to election timeout.
leader	Our state changed to leader.
follower	Our state changed to follower.
candidate	Our state changed to candidate.
stopped	Our state changed to stopped.
heartbeat	The leader is about to send a heartbeat message.
commit	A command has been saved to the majority of node's logs

---

Please note that the following properties are exposed on the constructor not on the prototype.

LifeRaft.states

This is an array that contains the names of the states. It can be used to create a human readable string from your current state.

console.log(LifeRaft.states[raft.state]); // FOLLOWER

LifeRaft.{FOLLOWER,LEADER,CANDIDATE,STOPPED,CHILD}

These are the values that we set as state. If you instance is a leader it's state will be set to LifeRaft.LEADER.

---

The rest of these properties are exposed on the LifeRaft prototype

LifeRaft#type(of)

Check the type of the given thing. This returns the correct type for arrays, objects, regexps and all the things. It's used internally in the library but might be useful for you as user as well. The function requires one argument which would be the thing who's type you need figure out.

raft.type([]); // array
raft.type({}); // object

LifeRaft#quorum(responses)

Check if we've reached our quorum (a.k.a. minimum amount of votes requires for a voting round to be considered valid) for the given amount of votes. This depends on the amount of joined nodes. It requires one argument which is the amount of responses that have been received.

raft.join('tcp://127.0.0.1');
raft.join('tcp://127.0.0.2');
raft.join('tcp://127.0.0.3');
raft.join('tcp://127.0.0.4');
raft.join('tcp://127.0.0.4');

raft.quorum(5); // true
raft.quorum(2); // false

LifeRaft#majority()

Returns the majority that needs to be reached for our quorum.

raft.majority(); // 4

LifeRaft#indefinitely(attempt, fn, timeout)

According to section 5.3 of the Raft paper it's required that we retry sending the RPC messages until they succeed. This function will run the given attempt function until the received callback has been called successfully and within our given timeout. If this is not the case we will call the attempt function again and again until it succeeds. The function requires 3 arguments:

1. attempt, The function that needs to be called over and over again until he calls the receiving callback successfully and without errors as we assume an error first callback pattern.
2. fn, Completion callback, we've successfully executed the attempt.
3. timeout, Time the attempt is allowed to take.

raft.indefinitely(function attemp(next) {
  dosomething(function (err, data) {
    //
    // if there is no error then we wil also pass the data to the completion
    // callback.
    //
    return next(err, data);
  });
}, function done(data) {
  // Successful execution.
}, 1000);

LifeRaft#packet(type, data)

Generate a new packet object that can be transfered to a client. The method accepts 2 arguments:

1. type, Type of packet that we want to transfer.
2. data, Data that should be transfered.

var packet = raft.packet('vote', { foo: 'bar' });

These packages will contain the following information:

• state If we are a LEADER, FOLLOWER or CANDIDATE
• term Our current term.
• address The address of this node.
• leader The address of our leader.
• last If logs are enabled we also include the last committed term and index.

And of course also the type which is the type you passed this function in and the data that you want to send.

LifeRaft#message(who, what, when)

The message method is somewhat private but it might also be useful for you as developer. It's a message interface between every connected node in your cluster. It allows you to send messages the current leader, or only the followers or everybody. This allows you easily build other scale and high availability patterns on top of this module and take advantage of all the features that this library is offering. This method accepts 2 arguments:

1. who, The messaging pattern/mode you want it use. It can either be:

• LifeRaft.LEADER: Send message to the current leader.
• LifeRaft.FOLLOWER: Send to everybody who is not a leader.
• LifeRaft.CHILD: Send to every child in the cluster (everybody).
• <node address>: Find the node based on the provided address.

2. what, The message body you want to use. We high suggest using the .packet method for constructing cluster messages so additional state can be send.
3. when, Optional completion callback for when all messages are send.

This message does have a side affect it also calculates the latency for sending the messages so we know if we are dangerously close to our threshold.

LifeRaft#join(address, write)

Add a new raft node to your cluster. All parameters are optional but normally you would pass in the name or address with the location of the server you want to add. The write method is only optional if you are using a custom instance that already has the write method defined.

var node = raft.join('127.0.0.1:8080', function write(packet) {
  // Write the message to the actual server that you just added.
});

As seen in the example above it returns the node that we created. This Node is also a Raft instance. When the node is added to the cluster it will emit the join event. The event will also receive a reference to the node that was added as argument:

raft.on('join', function join(node) {
  console.log(node.address); // 127.0.0.1:8080
});

LifeRaft#leave(address)

Now that you've added a new node to your raft cluster it's also good to know that you remove them again. This method either accepts the address of the node that you want to remove from the cluster or the returned node that was returned from the LifeRaft.join method.

raft.leave('127.0.0.1:8080');

Once the node has been removed from the cluster it will emit the leave event. The event will also receive a reference to the node that was removed as argument:

raft.on('leave', function leave(node) {
  console.log(node.address); // 127.0.0.1:8080
});

LifeRaft#promote()

Private method, use with caution

This promotes the Node from FOLLOWER to CANDIDATE and starts requesting votes from other connected nodes. When the majority has voted in favour of this node, it will become LEADER.

raft.promote();

LifeRaft#end()

This method is also aliased as .destroy.

This signals that the node wants to be removed from the cluster. Once it has successfully removed it self, it will emit the end event.

raft.on('end', function () {
  console.log('Node has shut down.');
});

raft.end();

LifeRaft#command(command)

Save a json command to the log. The command will be added to the log and then replicated to all the follower nodes. Once the majority of nodes have received and stored the command. A commit event will be triggered so that the command can be used.

raft.command({name: 'Jimi', surname: 'Hendrix'});

raft.on('commit', function (command) {
  console.log(command.name, command.surname);
});

Extending

LifeRaft uses the same pattern as Backbone.js to extend it's prototypes. It exposes an .extend method on the constructor. When you call this method it will return a fresh LifeRaft constructor with the newly applied prototypes and properties. So these extends will not affect the default instance. This extend method accepts 2 arguments.

1. Object with properties that should be merged with the prototype.
2. Object with properties that should be merged with the constructor.

var LifeBoat = LifeRaft.extend({
  foo: function foo() {
    return 'bar';
  }
});

Log Replication

LifeRaft uses Levelup for storing the log that is replicated to each node. Log replication is optional and so the log constructor needs to be included in the options when creating a raft instance. You can use any leveldown compatible database to store the log. LifeRaft will default to using leveldown. A unique path is required for each node's log.

const Log = require('liferaft/log');

const raft = new Raft({
  adapter: require('leveldown'),
  path: './db/log1'
  });

Transports

The library ships without transports by default. If we we're to implement this it would have made this library way to opinionated. You might want to leverage and existing infrastructure or library for messaging instead of going with our solution. There are only two methods you need to implement an initialize method and an write method. Both methods serve different use cases so we're going to take a closer look at both of them.

write

var LifeBoat = LifeRaft.extend({
  socket: null,
  write: function write(packet, callback) {
    if (!this.socket) this.socket = require('net').connect(this.address);
    this.socket.write(JSON.stringify(packet));

    // More code here ;-)
  }
});

There are a couple of things that we assume you implement in the write method:

• Message encoding The packet that you receive is an JSON object but you have to decide how you're going transfer that over the write in the most efficient way for you.
• message resending The Raft protocol states the messages that you write should be retried until indefinitely (Raft 5.3). There are already transports which do this automatically for you but if your's is missing this, the LifeRaft#indefinitely() is specifically written for this.

initialize

When you extend the LifeRaft instance you can assign a special initialize method. This method will be called when our LifeRaft code has been fully initialized and we're ready to initialize your code. The invocation type depends on the amount of arguments you specify in the function.

• synchronous: Your function specifies less then 2 arguments, it will receive one argument which is the options object that was provided in the constructor. If no options were provided it will be an empty object.
• asynchronous: Your function specifies 2 arguments, just like the synchronous execution it will receive the passed options as first argument but it will also receive a callback function as second argument. This callback should be executed once you're done with setting up your transport and you are ready to receive messages. The function follows an error first pattern so it receives an error as first argument it will emit the error event on the constructed instance.

var LifeBoat = LifeRaft.extend({
  socket: null,
  initialize: function initialize(options) {
    this.socket = new CustomTransport(this.address);
  }
});

//
// Or in async mode:
//
var LifeBoat = LifeRaft.extend({
  server: null,
  initialize: function initialize(options, fn) {
    this.server = require('net').createServer(function () {
      // Do stuff here to handle incoming connections etc.
    }.bind(this));

    var next = require('one-time')(fn);

    this.server.once('listening', next);
    this.server.once('error', next);

    this.server.listen(this.address);
  }
})

After your initialize method is called we will emit the initialize event. If your initialize method is asynchronous we will emit the event after the callback has been executed. Once the event is emitted we will start our timeout timers and hope that we will receive message in time.

License

MIT

liferaft's Issues

.message() to an undefined "who" fails silently.

Spend an hour trying to fix something because I had written FOLLOWERS instead of FOLLOWER. Perhaps if the nodes length is zero, throw an error or put one in the callback?

Old documentation (missing .extends) and Log Replication

Hi there, we are working using your liferaft lib in an university project. I found an update after few months and we updated from v.0.1.1 to 1.0.0.
First we tried using something like this (taking this example from your doc):

var LifeBoat = LifeRaft.extend({
  socket: null,
  write: function write(packet, callback) {
    // code etc...
  }
});

But it cannot startup and says "extend is not a function", because it seems disappeared from lib declarations.

We saw also your example in Log Replication and we saw you use now "extends LifeRaft", so maybe it's the documentation that is not updated.
We are asking ourselves also if our idea about transport layer is correct. Before your update we were implementing an "initialize" and "write" functions in your LifeRaft extension. We did a setup with expressjs in this way:

const MsgRaft = LifeRaft.extend({

	update: function(from) {
		this.stored = Object.assign(this.stored || {}, from);
	},

	initialize: function(options) {

		const raft = this; 
		const server = express();

		server.listen(port, () => console.log('initializing on :', raft.address));

		server.use(bodyParser.json())

		server.post('/message', (req, res) => {
			if (LifeRaft.LEADER == raft.state) {
				const packet = raft.packet('store', req.body)
				return raft.message(LifeRaft.CHILD, packet, () => res.json({ "success": true })) // Risposta al client       
			}
			res.json({ "success": false, "leader": raft.leader })
		})

		server.post('/', (req, res) => {
			const msg = req.body;
			if (msg.type == 'store') {
				if (msg.address = raft.leader) {
					raft.update(msg.data);
					raft.emit('new data', raft.stored);
				}
				return res.json({ 'store': true })
			}

			raft.emit('data', msg, function reply(data) {
				res.json(data)
			});

		});

		raft.once('end', () => server.close())

	},

	write: function(packet, fn) {
		console.log('sending packet to', this.address);
		request.post({ url: this.address, body: packet, json: true }, (err, res, body) => {
			fn(err, (body || {}).store ? null : body);
		});
	}

});

We are asking if it is correct the way they reply to each others. Because in this raft explanation seems that the Leader should wait the majority of responses before commit in memory its information, but in this way something seems missing and we do not find difference in your code.
Please can you give us some advises?

Unhandled rejection error on followers when log is not initialized

I ran example/index.js on my local machine without any modification on Node.js version 8.11.4. I think the error happens when the followers receive a packet with type "append" from the leader and log replication is not enabled.

Here's the logs:

(node:19891) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 101)
(node:19891) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'getLastInfo' of null
    at MsgRaft.raft.on (/home/andy/programs/liferaft/index.js:294:48)
    at MsgRaft.emit (/home/andy/programs/liferaft/node_modules/eventemitter3/index.js:130:35)
    at RepSocket.socket.on (/home/andy/programs/liferaft/example/index.js:29:12)
    at emitTwo (events.js:126:13)
    at RepSocket.emit (events.js:214:7)
    at Parser.<anonymous> (/home/andy/programs/liferaft/example/node_modules/axon/lib/sockets/rep.js:51:15)
    at emitOne (events.js:116:13)
    at Parser.emit (events.js:211:7)
    at Parser._write (/home/andy/programs/liferaft/example/node_modules/amp/lib/stream.js:91:16)
    at doWrite (_stream_writable.js:397:12)

Random not being random enough.

One of the timeout tests fails randomly. In this test we generate 100 timeouts and assert that we got at least more than 70% unique timeout ranges generated. This test fails randomly because we sometimes generate less unique/random timeouts for a given range. There are some possible causes for this:

• It could be that our test just makes an assumption that is just way to high, as we only generate timeouts within 150 and 300 but that would still leave 150 possibilities.
• Our timeout algorithm is flawed.
• The timeout is flawed.

With Math.random() analysis like this: https://github.com/nquinlan/better-random-numbers-for-javascript-mirror in the back of heads, it might be wise to come up something more random.

Persisting connections

From @jcrugzz

Related to the raft implementation, how should we go about managing connections to all the nodes? In the current use here, a new socket connection is opened up EACH time the write function is called. @3rd-Eden if Im missing something here then the interface is a bit confusing in how its communicating between all the servers. I know we keep the nodes that are in a specific cluster as child nodes of the main raft node but its not clear how the messages are being distributed to all of them. Is the write function called with the context of the child nodes in the cases where a message is being sent to them? (because this.name is the port/host signifier).

This is kind of a combo liferaft/sumo issue but made sense to put it here. @3rd-Eden tell me if im crazy or explain a bit of your thought process if you see where I'm coming from.

---

@3rd-Eden reply

It was somewhat intentional to open a connection for every single write as raft requires a request response pattern for communication. By just opening a new connection for every single write makes the code a bit easier to reason about as liferaft is still an early stage project and I didn't to complicate the logic to much by adding a persistent communications to sumo. It would need to have reconnection logic, message framing, some sort of protocol written on top the connection logic to figure out when something is a response to a message. We need to implement ping/heartbeats to ensure that the connection stays alive etc. All of this would add much more technical burden to the project so I decided to ignore it for now as connections are cheap anyways if we hit a limit with sumo where we're creating to much connections we can always easily switch out the connection logic.

As for the internal messaging interface I can understand it's a bit confusing. I think my poor decision to call a property name also contributes to the confusion but that can be resolved easily. As for the transport logic:

When you construct a new raft node, it will call the initialize method so it can construct a server/connection to receive messages. This raft node does use the write method because all communication will be done by writing directly to the incoming connections.

The write method is only used when you call the join method so it becomes aware of other nodes in the cluster and joins them. In order to message these child nodes, it will call the write method as these child nodes do not need to establish a server but only connect to the raft nodes and send messages.

But I can see this becoming an issue when you're starting to use persistent connections as the created Raft node will already have a persistent connection to each of the child nodes in the cluster and it would create another one every time you write. If this is the problem you're facing we could add a reference to the main raftnode when we create a child and store it as a parent property. So on the server you could store all persistent connections according to their names in an object on the parent and have your child nodes reference that connection when writing data. The only problem would be that receiving replies would be harder. But it's definitely something that we can solve.

But I think this issue more specific to liferaft than sumo so I'm going to move this discussion over there.

Explanatory link broken

liferaft is an JavaScript implementation of the Raft consensus algorithm.

forwarding client requests to follower -> leader

So we have no predefined way of how we are receiving data with the transport being agnostic but I was thinking it could be useful to have a hook in here to allow for proper forwarding of any CLIENT requests to the leader. Having the ability to have any client send a request to any node in a cluster and have it properly handled by raft seems relevant for the availability scenario as a client may not know which one is the leader.

@3rd-Eden this is an optimization but thought it was worth writing down since i had the thought.

Initialization should be async

We currently just bluntly add all initialize methods as initialize event and assume we're a follower & ready. We should actually async execute the initialize method so server/transport creation can also be done async and we can set our state to follower once servers are started up and emit the initialize method.

is liferaft currently safe to use in production?

The project looks somewhat forgotten; and while the API looks nice, is it readily usable? Or shall I expect to encounter issues very soon? :)

which version is ready for production ?

only see 0.1.1 in release list. how about master branch for production ? thanks.

Leader Memory Leak

There might be an issue with memory leaks. When trying out the example code, I monitored the memory usage of the leader, and it seems like it frees most memory periodically, but not all (see first screenshot). Some memory seem to be unreachable, and as those accumulates over time, we have a consistent positive slope (second screen shot), and it will eventually lead to memory exhaustion and thus the leader crushes. This is even when the leader is not doing anything besides sending/receiving packets to/from followers. Here I am running 5 nodes(1 leader, 4 followers). Could you help?

Thanks!

Emit follower event on first join

When a raft first joins the cluster as a follower, and event does not appear to be emitted. This is my ugly fix for that...

    raft.on('follower',      function () { self.emit('follower'); });
    raft.on('leader change', function () {
        if (!self._emittedFollower) {
            self.emit('follower');
            self._emittedFollower = true;
        }
    });

Liferaft status

This library looks really cool and the documentation is excellent. Do you know if this code is being used in production anywhere?

And would you be interested in a PR that updates the code to use es6 Classes and any other es6 modernizations that would be appropriate?