coding-to-music / a-kafka-story-zookeeper-java-telegraf-jmx-grafana-ksql

Kafka ecosystem ... but step by step! Topics kafka zookeeper kafka-streams cdc kafka-broker debezium ksql

a-kafka-story-zookeeper-java-telegraf-jmx-grafana-ksql

🚀 Kafka ecosystem ... but step by step! kafka zookeeper kafka-streams cdc kafka-broker debezium ksql 🚀

https://github.com/coding-to-music/a-kafka-story-zookeeper-java-telegraf-jmx-grafana-ksql

From / By Florent Ramière https://github.com/framiere

https://github.com/framiere/a-kafka-story

Environment variables:

GitHub

git init
git add .
git remote remove origin
git commit -m "first commit"
git branch -M main
git remote add origin [e-mail]:coding-to-music/a-kafka-story-zookeeper-java-telegraf-jmx-grafana-ksql.git
git push -u origin main

Install Java JDK

java --version

java not found

sudo apt install default-jdk

long install output...

java --version

Output

openjdk 11.0.16 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

sudo apt install default-jre

Output

Reading package lists... Done
Building dependency tree       
Reading state information... Done
default-jre is already the newest version (2:1.11-72).
default-jre set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 35 not upgraded.

Please checkout these awesome references

1. http://developer.confluent.io/
2. https://kafka-tutorials.confluent.io/

And if you want to learn another way, just follow these steps.

Make docker and maven do their thing once for all by running ./fetch.sh

Then jump in the Kafka Story!

1. One zookeeper, one kafka broker
2. One zookeeper, many kafka brokers
3. Java consumer, java producer
4. Let's add data with telegraf
5. Let's setup better defaults
6. Enter kafka stream
7. Capture JMX metrics
8. Grafana
9. Kafka Connect
10. Kafka Connect and Schema Registry
11. Change Data Capture
12. Change Data Capture and Schema Registry
13. Change Data Capture and Schema Registry and export to S3
14. Ksql
15. Ksql server and UI
16. Change Data Capture, Schema Registry and Ksql
17. Change Data Capture, JSON, Ksql and join
18. Random producer and Complex joins
19. Sync random producer and mysql, capture CDC diff and push it to telegraf

Don't like Docker ? Please download Confluent platform here: https://www.confluent.io/download/

Also, please take a look at

1. https://github.com/confluentinc/cp-demo
2. https://github.com/confluentinc/demo-scene
3. https://github.com/confluentinc/examples
4. https://github.com/confluentinc/kafka-streams-examples
5. https://www.confluent.io/stream-processing-cookbook/