hyperkafka
A decentralized messaging platform.
npm i hyperkafka
Usage
// producer
var drive = hyperdrive(memdb())
var archive = drive.createArchive()
var producer = hyperkafka.Producer(archive)
producer.write('topic', 'hello')
// somewhere, another machine
var archive2 = drive.createArchive(archive.key)
var consumer = hyperkafka.Consumer(archive2)
consumer.get('topic', 0, function (err, msg) {
// msg === {offset: 0, ts: timestamp, payload: buffer('hello')}
})
// create a readable stream
var rs = consumer.createReadStream('topic')
rs.on('data', msg => { })
API
Producer
p = new Producer(archive, opts)
Create a new Producer
with a hyperdrive archive.
Option opts
includes:
- segmentSize: the file size limit of a single segment.
- linger: the delay before flushing buffer into hyperdrive
p.write(topic, message, [timestamp])
Append a message to a topic. message
can be Buffer
, String
, or object.
You can specify timestamp for the message. The timestamp should be a uint64be
buffer because there's no int64 support in javascript. By default it will be current time (Date.now()
).
p.topics(cb(err, topics))
Returns an array of topics
p.on('flush', cb(segmentSize, flushedTopci))
Producer will emit flush
event when a segment is flushed into hyperdrive.
Consumer
c = new Consumer(archive)
Create a new Consumer
with a hyperdrive archive.
c.get(topic, offset, cb(err, message))
Get a message on a topic with specified offset. Returns error if there's no message with given offset.
c.createReadStream(topic, start)
Get a readable stream with all messages in a topic after given offset.
c.topics(cb(err, topics))
Returns an array of topics
FAQ
Why?
Event-sourcing is a popular pattern for building large-scale high-performance system. It's also a simple and extensible software architecture.
By utilizing the well-designed Kafka's API, we can make developing P2P app simpler and easier to reason.
This is just append-only log! Why not just use hypercore?
Hyperdrive's indexing allows us to query and insert data efficiently.
Also, If we simply push every single message as a seperated block into hypercore. The overhead is big: 144 bytes per message for 100000 messages. Therefore we need batching.
This is basic messaging! How about MQTT/RabbitMQ/ZeroMQ...or any other message queue?
Message queues usually don't preserve message history. I want a messaging platform which can automatically archive all message in a decentralized environment.
Should I replace my apache kafka cluster with hyperkafka?
No. They have different goal and use cases. hyperkafka use the name because its internal storage is inspired by apache kafka, also it sounds cool.
If you want a private, highly-scalable messaging platform, use apache kafka.
If you want to store and share public realtime data, try hyperkafka. You might like it.
License
The MIT License