funkygao / gafka

A full ecosystem that is built around kafka powered by golang

gafka

                   __   _            
                  / _| | |           
   __ _    __ _  | |_  | | __   __ _ 
  / _` |  / _` | |  _| | |/ /  / _` |
 | (_| | | (_| | | |   |   <  | (_| |
  \__, |  \__,_| |_|   |_|\_\  \__,_|
   __/ |                             
  |___/                              

A full ecosystem for kafka/redis/PubSub/ElasticSearch/Zookeeper/haproxy.

Hope it can help you.

Components

• ehaproxy

  Elastic haproxy that sits in front of kateway.

• kateway

  A fully-managed real-time secure and reliable RESTful Cloud Pub/Sub streaming message/job service.

• actord

  kateway job scheduler and webhook dispatcher.

• gk

  Unified multi-datacenter multi-cluster kafka swiss-knife management console.

• zk

  A handy zookeeper CLI that supports recursive operation without any dependency.

• es

  ElasticSearch console.

• kguard

  Kafka clusters body guard that emits health info to InfluxDB and exports key warnings to zabbix for alerting.

Install

export PATH=$PATH:$GOPATH/bin

#========================================
# install go-bindata and go annotations
#========================================
go install github.com/jteeuwen/go-bindata/go-bindata
go install github.com/funkygao/goannotation

#========================================
# install gafka
#========================================
go get github.com/funkygao/gafka
cd $GOPATH/src/github.com/funkygao/gafka
./build.sh -a # build all components

#========================================
# try the gafka command 'gk'
#========================================
gk -h

Status

Currently gafka manages:

• 4 data centers
• 50+ kafka clusters
• 100+ kafka brokers
• 500+ kafka topics
• 2000+ kafka partitions
• 10Billion messages per day
• peak load
  • 1Million message per second
  • 8TB transfered per hour
• 5TB redis

gafka's Issues

cancellable delayed message feature

When I Pub a message, I want to specify when Sub client can receive that message. Before
that deadline, I am able to cancel that message.

hinted handoff when storage system gets into trouble

If backend(kafka) gets into trouble, e,g. hard disk suddenly gets very slow, network latency suddenly bursts, kateway can use hinted handoff mechanism for temporary storage and schedules all buffered messages for delivary when backend system is ok

server side message filter

message filter by tag

Pub

client can specify tags: a list of key/value pairs

Sub

client can specify which tag it is interested in

In access log, host entry found port number

Reference to missing github.com/funkygao/metrics library

compiling gk
/go/src/github.com/funkygao/gafka/cmd/gk/command/peek.go:17:2: cannot find package "github.com/funkygao/metrics" in any of:
/usr/lib/golang/src/github.com/funkygao/metrics (from $GOROOT)
/go/src/github.com/funkygao/metrics (from $GOPATH)

display real consumer client ip instead of the kateway instance ip

kateway as the message gateway, currently is displaying its ip for all consumer groups.
that is hard to tell the topology of the active connections.

kateway should show real consumer client ip besides kateway's ip

cannot find package error

eric@eric:gafka$ go get ./...
package github.com/funkygao/gafka/cmd/canal/command: cannot find package "github.com/funkygao/gafka/cmd/canal/command" in any of:
/home/eric/Downloads/go-go1.6/src/github.com/funkygao/gafka/cmd/canal/command (from $GOROOT)
/home/eric/Code/Go/GOPATH/src/github.com/funkygao/gafka/cmd/canal/command (from $GOPATH)
eric@eric:gafka$

message schema registry

I am using avro as message encoding/decoding.
Can kateway provide a avro schema registry feature?

• user can register the schema for a versioned topic
• user can get the schema definition of a versioned topic

multiplex for consumer groups

By default, a consumer group can not have more members than partitions of a topic.
However, client will have to implement their own election alg to make concurrent sub client=partitions, which is not easy to do.
kateway should be able to handle this for sub client with multiplex.

webhoook

When I create a topic, we want to be able setup its webhoook endpoints.
The use case is:
whenever a message is published to the webhook-enabled topic, kateway will POST that message to all the registered webhook endpoints.

• webhook can have several endpoints
• kateway will handle the problems when endpoints outage
• kateway can identify itself when POSTing to the endpoints