gelleson / jocko Goto Github PK

View Code? Open in Web Editor NEW

This project forked from travisjeffery/jocko

0.0 0.0 0.0 11 MB

Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

Home Page: https://twitter.com/travisjeffery

License: MIT License

Shell 0.07% Go 99.62% Makefile 0.17% Dockerfile 0.13%

jocko's Introduction

Jocko

Kafka/distributed commit log service in Go.

Goals of this project:

Implement Kafka in Go
Protocol compatible with Kafka so Kafka clients and services work with Jocko
Make operating simpler
Distribute a single binary
Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
Smarter configuration settings
- Able to use percentages of disk space for retention policies rather than only bytes and time kept
- Handling size configs when you change the number of partitions or add topics
Learn a lot and have fun

TODO

Hiatus Writing Book

I’m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the book’s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.

Reading

Project Layout

├── broker        broker subsystem
├── cmd           commands
│   └── jocko     command to run a Jocko broker and manage topics
├── commitlog     low-level commit log implementation
├── examples      examples running/using Jocko
│   ├── cluster   example booting up a 3-broker Jocko cluster
│   └── sarama    example producing/consuming with Sarama
├── protocol      golang implementation of Kafka's protocol
├── prometheus    wrapper around Prometheus' client lib to handle metrics
├── server        API subsystem
└── testutil      test utils
    └── mock      mocks of the various subsystems

Building

Local

Clone Jocko

$ go get github.com/travisjeffery/jocko

Build Jocko
```
$ cd $GOPATH/src/github.com/travisjeffery/jocko
$ make build
```
(If you see an error about dep not being found, ensure that $GOPATH/bin is in your PATH)