redaous / awesome-opensource-contribs-kafka

A list of all awesome open-source contributions for the Apache Kafka project

Awesome Open-Source Contribs for Apache Kafka

A curated list of awesome open-source frameworks, libraries, tools and examples for the Apache Kafka project.

Contributing to this project is easy! Just send a pull request and make the list growth!

Table of contents

• Client
• CLI/Tools
• CDC
• Examples
• Site Reliability Tools
• Security / ACL
• Infrastructure
• KafkaStreams
• Query/SQL
• REST API
• Replication
• Schema/Data
• Test
• UI/Cluster Management
• Other Awesome Lists about Apache Kafka or Data Engineering
• License

Clients

• librdkafka: The Apache Kafka C/C++ library. 2-clause BSD license

CLI / Tools

• Kafkacat: Generic command line non-JVM Apache Kafka producer and consumer. librdkafka - Apache Kafka C driver library

• kafkacli: CLI and Go Clients to manage Kafka components (Kafka Connect & SchemaRegistry). Apache License 2.0

Change Data Capture

• Debezium: Debezium is an open source project that provides a low latency data streaming platform for change data capture (CDC). Apache License 2.0

• Maxwell: Maxwell's daemon, a mysql-to-json kafka producer. Apache License 2.0

Examples

• Confluent Kafka Streams Examples: Demo applications and code examples for Apache Kafka's Streams API. Apache License 2.0

• Confluent Examples: Apache Kafka and Confluent Platform examples and demos. Apache License 2.0

• kafka-examples: Snippets and small examples demonstrating kafka features and configs. Apache License 2.0

Site Reliability Tools

• Datadog Kafka-kit: Kafka data mapping and recovery tools. Apache License 2.0

• kafka-monitoring-suite-demo-prometheus: Demonstration on how to monitor Kafka using Prometheus and Grafana. Apache License 2.0

• Linkedin - Burrow: Kafka Consumer Lag Checking. Apache License 2.0

• Linkedin - Cruise Control: Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters. BSD 2-Clause "Simplified" License

• Linkedin - Xinfra Monitor: Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service availability & message loss rate. It reassigns partition & trigger preferred leader election to ensure each broker acts as leader of at least 1 partition of monitor topic.Apache License 2.0

• Pinterest Doctor Kafka: DoctorKafka is a service for Kafka cluster auto healing and workload balancing. Apache License 2.0

• Uber - Chaperone : As Kafka audit system, Chaperone monitors the completeness and latency of data stream. The audit metrics are persisted in database for Kafka users to quantify the loss of their topics if any. Apache License 2.0

Security / ACL

• kafka-security-manager: Manage your Kafka ACL at scale. MIT License

• Kafka Specs: Tool to ease and automate Apache Kafka cluster configuration management. Apache License 2.0

Infrastructure

• Confluent Docker Images: Docker images for Confluent Platform. [Apache License 2.0](Apache License 2.0)

• Confluent Platform Ansible: Ansible playbooks for the Confluent Platform. Apache License 2.0

• Strimzi: Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. Apache License 2.0

KafkaStreams

• AzkarraStreams: Azkarra Streams is a lightweight Java framework which makes easy to develop and operate Kafka Streams applications. Apache License 2.0

• GoKa: Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go. BSD 3-Clause "New" or "Revised" License

• Kafka Streams CEP: Complex Event Processing on top of Kafka Streams . Apache License 2.0

• Kafka Streams Viz: A tool helps visualizing stream topologies by generating nice looking diagrams from a kafka stream topology descriptions.

Query/SQL

• ksqlDB: The event streaming database purpose-built for stream processing applications. Confluent Community License Agreement Version 1.0

REST API

• Kafka REST Proxy: Confluent REST Proxy for Kafka. Confluent Community License Agreement Version 1.0

Replication

• Mirror Maker 2: MM2 leverages the Connect framework to replicate topics between Kafka clusters. Apache License 2.0

• Uber uReplicator: Improvement of Apache Kafka Mirrormaker. Apache License 2.0

Schema / Data

• Confluent Schema Registry: Confluent Schema Registry for Kafka. Confluent Community License Agreement Version 1.0

Test

• Mocked Streams: Scala DSL for Unit-Testing Processing Topologies in Kafka Streams. Apache License 2.0

• Kafka Junit: This library wraps Kafka's embedded test cluster, allowing you to more easily create and run integration tests using JUnit against a "real" kafka server running within the context of your tests. No need to stand up an external kafka cluster! BSD 3-Clause "New" or "Revised" License

• kafka-streams-test-utils: Testing Kafka Streams. Apache License 2.0

• Test Container for Apache Kafka: Testcontainers can be used to automatically instantiate and manage Apache Kafka containers. More precisely Testcontainers uses the official Docker images for Confluent OSS Platform. MIT License

UI/Cluster Management

• CMAK: CMAK (previously known as Kafka Manager) is a tool for managing Apache Kafka clusters. See below for details about the name change. Apache License 2.0

• Kafka HQ: Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more. Apache License 2.0

• Kafdrop – Kafka Web UI: Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages. Apache License 2.0

• Kafka Eagle: A easy and high-performance monitoring system, as well as offsets or metadata and other kafka information. Apache License 2.0

• Kafkawize: Kafkawize : A Self service Apache Kafka Topic Management tool/portal. A Web application which automates the process of creating and browsing Kafka topics, acls, schemas by introducing roles/authorizations to users of various teams of an org. Apache License 2.0

• Remora: Kafka consumer lag-checking application for monitoring, written in Scala and Akka HTTP; a wrap around the Kafka consumer group command. Integrations with Cloudwatch and Datadog. Authentication recently added - MIT License

Other Awesome Lists about Apache Kafka & Data Engineering

• Awesome Open-Source Data Engineering
• Awesome Data Engineering
• infoslack/awesome-kafka: This list is for anyone wishing to learn about Apache Kafka, but do not have a starting point.
• dharmeshkakadia/awesome-kafka: Everything about Apache Kafka
• Apache Kafka Ecosystem

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.